Log In Sign Up

pilotbrick.co.uk
robots.txt

Robots Exclusion Standard data for pilotbrick.co.uk

Resource Scan

Scan Details

Site Domain pilotbrick.co.uk
Base Domain pilotbrick.co.uk
Scan Status Ok
Last Scan2024-11-16T14:26:01+00:00
Next Scan 2024-11-23T14:26:01+00:00

Last Scan

Scanned2024-11-16T14:26:01+00:00
URL https://pilotbrick.co.uk/robots.txt
Domain IPs 5.9.21.39
Response IP 5.9.21.39
Found Yes
Hash f606377548f62a8ddce5f95a84d85b69626d98970136eb8f6ae24417536350e8
SimHash da3e58d0c288

Groups

*

Rule Path
Disallow /da/
Disallow /revi
Disallow /search?*
Disallow /login?*
Disallow /rgstr?*
Disallow /out?*

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

amazonbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

awariorssbot
awariosmartbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

blexbot

Rule Path
Disallow /

claudebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dataforseobot

Rule Path
Disallow /

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

imagesiftbot

Rule Path
Disallow /

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

geedobot

Rule Path
Disallow /

geedoproductsearch

Rule Path
Disallow /

petalbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

seznambot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

turnitinbot

Rule Path
Disallow /

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandexaccessibilitybot
yandexadditional
yandexadditionalbot
yandexadnet
yandexblogs
yandexbot
yandexcalendar
yandexdirect
yandexdirectdyn
yandexfavicons
yadirectfetcher
yandexfordomain
yandeximages
yandeximageresizer
yandexmobilebot
yandexmarket
yandexmedia
yandexmetrika
yandexmobilescreenshotbot
yandexnews
yandexontodb
yandexontodbapi
yandexpagechecker
yandexpartner
yandexrca
yandexrenderresourcesbot
yandexscreenshotbot
yandexsearchshop
yandexsitelinks
yandexspravbot
yandextracker
yandexturbo
yandexuserproxy
yandexvertis
yandexverticals
yandexvideo
yandexvideoparser
yandexwebmaster

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top