theworldwasherefirst.com
robots.txt

Robots Exclusion Standard data for theworldwasherefirst.com

Resource Scan

Scan Details

Site Domain theworldwasherefirst.com
Base Domain theworldwasherefirst.com
Scan Status Ok
Last Scan2026-01-20T12:00:53+00:00
Next Scan 2026-01-27T12:00:53+00:00

Last Scan

Scanned2026-01-20T12:00:53+00:00
URL https://theworldwasherefirst.com/robots.txt
Domain IPs 194.1.147.64, 194.1.147.68
Response IP 194.1.147.68
Found Yes
Hash 0265ddeeaae3f6752b64846dbc03c1a5d196a8760c3c47f01ab78c5c0c3c6629
SimHash 7110f280c502

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /*?add-to-cart=
Disallow /*%26add-to-cart%3D

storebot-google

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

bingbot

Rule Path
Disallow /?s=

slurp

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

pinterest

Rule Path
Disallow

facebookexternalhit

Rule Path
Disallow

twitterbot

Rule Path
Disallow

linkedinbot

Rule Path
Disallow

Comments

  • Google Merchant Center StoreBot
  • Google Ads crawler
  • Google Search
  • Google Images
  • Bing
  • Yahoo (Slurp)
  • DuckDuckGo
  • Pinterest scraper for pins
  • Facebook preview bot
  • Twitter link preview
  • LinkedIn preview bot