thekidshouldseethis.com
robots.txt

Robots Exclusion Standard data for thekidshouldseethis.com

Resource Scan

Scan Details

Site Domain thekidshouldseethis.com
Base Domain thekidshouldseethis.com
Scan Status Ok
Last Scan2024-06-16T04:52:56+00:00
Next Scan 2024-06-23T04:52:56+00:00

Last Scan

Scanned2024-06-16T04:52:56+00:00
URL https://thekidshouldseethis.com/robots.txt
Domain IPs 172.66.40.65, 172.66.43.191, 2606:4700:3108::ac42:2841, 2606:4700:3108::ac42:2bbf
Response IP 172.66.40.65
Found Yes
Hash 7c50648a5038745d2ce8044a0ea9f8873641f7f68efb04f2d90ebe64e6413a6b
SimHash 4518c81047b0

Groups

scrapy

Rule Path
Allow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

googlebot

Rule Path
Disallow

bingbot

Rule Path
Disallow

*

Rule Path
Disallow /

Comments

  • Allow Googlebot
  • Allow Bingbot
  • Block all other bots