porntubeguru.com
robots.txt

Robots Exclusion Standard data for porntubeguru.com

Resource Scan

Scan Details

Site Domain porntubeguru.com
Base Domain porntubeguru.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-09-10T22:13:40+00:00
Next Scan 2024-12-09T22:13:40+00:00

Last Successful Scan

Scanned2024-04-21T14:41:30+00:00
URL https://porntubeguru.com/robots.txt
Domain IPs 213.174.130.195
Response IP 213.174.130.195
Found Yes
Hash 6834c770f56d435917f8cd12b8dc5ba0e6c4b7b1e56ab5bff5cc9fbae01b11d1
SimHash e37991470387

Groups

yandexbot

Rule Path
Disallow /cgi-bin/
Disallow /view/
Disallow /2257
Disallow /contact/
Disallow /embed/
Disallow /v/

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Disallow /cgi-bin/
Disallow /view/
Disallow /2257
Disallow /contact/
Disallow /embed/
Disallow /v/

Other Records

Field Value
crawl-delay 1

msnbot

Rule Path
Disallow /cgi-bin/
Disallow /view/
Disallow /2257
Disallow /contact/
Disallow /embed/
Disallow /v/

Other Records

Field Value
crawl-delay 1

abachobot
ahrefsbot
anarchie
antibot
appie
aspseek
asterias
attach
autoemailspider
b2w
backdoorbot
backweb
bandit
batchftp
black\ hole
blackwidow
blowfish
bot\ mailto
bot\ mailto:craftbot@yahoo.com
botalot
buddy
builtbottough
bullseye
bumblebee
bunnyslippers
cheesebot
cherrypicker
cherrypickerelite
cherrypickerse
chinaclaw
clariabot
clshttp
coast\ webmaster
coldfusion
collector
copier
copyrightcheck
cosmos
crescent
curl
custo
da
diamond
disco
disco\ pump
dittospyder
dloader
dotbot
download\ demon
download\ wonder
downloader
drip
dts\ agent
easydl
ecatch
eirgrabber
emailcollector
emailsiphon
emailwolf
erocrawler
exabot
express\ webpictures
extractorpro
extreme\ picture\ finder
eyenetie
fast\ webcrawler
fetch\ api\ request
filehound
flashget
flickbot
freefind.com
frontpage
generic
getright
getsmart
getweb!
gigabot
go!zilla
go-ahead-got-it
gotit
grabber
grabnet
grafula
gulliver
harvest
heretrix
hitboxdoctor
hloader
hmview
httpapp
httpfetcher
httplib
httpscraper
httptrack
httpviewer
httrack
humanlinks
ia_archiver
image\ stripper
cutestat
image\ sucker
indy\ library
infonavirobot
interget
internet\ ninja
internetseer.com
iria
irlbot
java
jennybot
jetcar
jobo
joc
joc\ web\ spider
jonzilla
justview
kenjin\ spider
keyword\ density
lachesis
larbin
leechftp
lexibot
lftp
libby_
libweb
libwww-perl
libwwwperl
likse
link
linkextractorpro
linkscan
linkwalker
lwp-trivial
lwp\ request
mag-net
magnet
mass\ downloader
mata\ hari
memo
mercator
metacarta
mewsoft\ search\ engine
mfc_tear_sample
microsoft\ url\ control
microsofturl
midown\ tool
miixpc
mirror
missigua
mister\ pix
mj12bot
moget
msfrontpage
msiecrawler
nationaldirectory\ webspider
navroad
nearsite
net\ probe
net\ vampire
netants
netmechanic
netresearchserver
netspider
netzip
netzip
nexuscache
nicerspro
nikto
ninja
npbot
obot
octopus
offline\ explorer
offline\ navigator
onestop
openfind
openfind\ data\ gatherer
orangebot
our\ agent
pagegrabber
papa\ foto
pavuk
pcbrowser
perl
php
php\ version
phpot
ping
pingalink\ monitoring\ services
pockey
pompos
propowerbot
prowebwalker
psbot
psycheclone
pump
python-urllib
bubing
ltx71 - (http://ltx71.com/)
python\ urllib
queryn
realdownload
reaper
recorder
reget
repomonkey
rico
rma
robozilla
rogerbot
scooter
scoutabout
semrushbot-sa
siphon
sitecheck.internetseer.com
sitesnagger
slysearch
smartdownload
snake
snapbot
snoopy
spacebison
spankbot
spanner
spinne
sqworm
stealer
stripper
sucker
superbot
superhttp
surfbot
suzuran
szukacz
takeout
teleport\ pro
telesoft
the\ intraformant
thenomad
tighttwatbot
titan
tocrawl
true_robot
turingos
turnitinbot
urldispatcher
urly\ warning
vacuum
vagabondo
vayala
vci
vintage
voideye
w3c_validator
web\ downloader
web\ image\ collector
web\ sucker
webauto
webcopier
webdownloader
webenhancer
webfetch
webgo\ is
webhook
webleacher
webminer
webmirror
webmole
webreaper
websauger
website
website\ extractor
website\ quester
websites
webster
webster\ pro
webstripper
websucker
webviewer
webwhacker
webzip
webzip
wells
wget
whacker
widow
wildsoft\ surfer
winhttp
winhttprequest
www-collector-e
wwwoffle
xaldon
xaldon\ webspider
xara
xenu
y!tunnelpro
yahooysmcm
zade
zbot
zeus
barkrowler
crazywebcrawler-spider
blexbot

Rule Path
Disallow /

*

Rule Path
Disallow /cgi-bin/
Disallow /view/
Disallow /2257
Disallow /contact/
Disallow /embed/
Disallow /v/
Disallow /push/

Warnings

  • 1 invalid line.
  • `host` is not a known field.