tpl.one
robots.txt
Robots Exclusion Standard data for tpl.one
Resource Scan
Scan Details
| Site Domain | tpl.one |
| Base Domain | tpl.one |
| Scan Status | Ok |
| Last Scan | 2024-10-30T06:33:20+00:00 |
| Next Scan | 2024-11-29T06:33:20+00:00 |
Last Scan
| Scanned | 2024-10-30T06:33:20+00:00 |
| URL | https://tpl.one/robots.txt |
| Domain IPs | 104.21.11.51, 172.67.165.43, 2606:4700:3034::ac43:a52b, 2606:4700:3037::6815:b33 |
| Response IP | 104.21.11.51 |
| Found | Yes |
| Hash | 3be81aa3579704f6ba7e3b1ad25dcc3c20c4be3dbb3040de6c3a203fa525541a |
| SimHash | 7f36c4e146fb |
Groups
*
| Rule | Path |
|---|---|
| Disallow | / |
googlebot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
googlebot-mobile
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
googlebot-image
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
googlebot-news
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
googlebot-video
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
mediapartners-google
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
bingbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
slurp
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
java
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
wget
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
curl
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
commons-httpclient
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
python-urllib
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
libwww
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
httpunit
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
nutch
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
go-http-client
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
phpcrawl
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
msnbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
jyxobot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
fast-webcrawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
fast enterprise crawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
biglotron
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
teoma
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
convera
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
seekbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
gigabot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
gigablast
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
exabot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
ngbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
ia_archiver
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
gingercrawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
webmon
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
httrack
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
webcrawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
grub.org
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
usinenouvellecrawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
antibot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
netresearchserver
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
speedy
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
fluffy
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
bibnum.bnf
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
findlink
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
msrbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
panscient
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
yacybot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
aisearchbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
ioi
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
ips-agent
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
tagoobot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
dotbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
woriobot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
yanga
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
buzzbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
mlbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
yandexbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
purebot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
linguee bot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
voyager
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
cyberpatrol
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
voilabot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
citeseerxbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
spbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
twengabot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
postrank
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
turnitinbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
scribdbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
page2rss
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
sitebot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
linkdex
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
adidxbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
blekkobot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
ezooms
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
dotbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
mail.ru_bot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
discobot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
heritrix
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
findthatfile
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
europarchive.org
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
nerdbynature.bot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
sistrix crawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
ahrefsbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
aboundex
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
domaincrawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
wbsearchbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
summify
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
ccbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
edisterbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
seznambot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
ec2linkfinder
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
gslfbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
aihitbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
intelium_bot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
facebookexternalhit
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
yeti
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
retrevopageanalyzer
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
lb-spider
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
sogou
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
lssbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
careerbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
wotbox
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
wocbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
ichiro
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
duckduckbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
lssrocketcrawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
drupact
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
webcompanycrawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
acoonbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
openindexspider
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
gnam gnam spider
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
web-archive-net.com.bot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
backlinkcrawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
coccoc
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
integromedb
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
content crawler spider
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
toplistbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
seokicks-robot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
it2media-domain-crawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
ip-web-crawler.com
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
siteexplorer.info
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
elisabot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
proximic
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
changedetection
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
blexbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
arabot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
wesee:search
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
niki-bot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
crystalsemanticsbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
rogerbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
psbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
interfaxscanbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
lipperhey seo service
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
cc metadata scaper
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
g00g1e.net
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
grapeshotcrawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
urlappendbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
brainobot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
fr-crawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
binlar
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
simplecrawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
livelapbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
twitterbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
cxensebot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
smtbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
bnf.fr_bot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
a6-indexer
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
admantx
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
facebot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
twitterbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
orangebot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
memorybot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
advbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
megaindex
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
semanticscholarbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
ltx71
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
nerdybot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
xovibot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
bubing
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
qwantify
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
archive.org_bot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
applebot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
tweetmemebot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
crawler4j
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
findxbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
semrushbot-sa
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
yoozbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
lipperhey
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
y!j-asr
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
domain re-animator bot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
addthis
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
screaming frog seo spider
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
metauri
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
scrapy
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
livelapbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
openhosebot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
capsulechecker
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
collection@infegy.com
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
istellabot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
deusu\/
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
betabot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
cliqzbot\/
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
mojeekbot\/
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
netestate ne crawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
safesearch microdata crawler
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
gluten free crawler\/
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
sonic
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
sysomos
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cgi-bin/ |
| Disallow | /1_loader/ |
| Disallow | /3_functions/ |
| Disallow | /8callback/ |
| Disallow | /ajax/ |
| Disallow | /ajax/cmd.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://tpl.one/sitemap.xml |
Warnings
- 9 invalid lines.