scarlettsparrotessentials.co.uk
robots.txt

Robots Exclusion Standard data for scarlettsparrotessentials.co.uk

Resource Scan

Scan Details

Site Domain scarlettsparrotessentials.co.uk
Base Domain scarlettsparrotessentials.co.uk
Scan Status Ok
Last Scan2025-10-12T07:49:22+00:00
Next Scan 2025-11-11T07:49:22+00:00

Last Scan

Scanned2025-10-12T07:49:22+00:00
URL https://scarlettsparrotessentials.co.uk/robots.txt
Redirect https://www.scarlettsparrotessentials.co.uk/robots.txt
Redirect Domain www.scarlettsparrotessentials.co.uk
Redirect Base scarlettsparrotessentials.co.uk
Domain IPs 192.124.249.138
Redirect IPs 192.124.249.138
Response IP 192.124.249.138
Found Yes
Hash e7efb3d731070febb74c5f561fa2fd0502d4feb0bd9f15046b00ae52183cc4b6
SimHash 2914d86af200

Groups

*

Rule Path
Disallow /checkout

adsbot-google

Rule Path
Disallow /checkout

nutch

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /checkout

Other Records

Field Value
crawl-delay 10

ahrefssiteaudit

Rule Path
Disallow /checkout

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://www.scarlettsparrotessentials.co.uk/pub/sitemap/sitemap.xml
sitemap https://www.scarlettsparrotessentials.co.uk/pub/sitemap/sitemap.xml
sitemap https://www.scarlettsparrotessentials.co.uk/pub/sitemap/sitemap.xml

Comments

  • Google adsbot ignores robots.txt unless specifically named!