meanwhileinamerica.com
robots.txt

Robots Exclusion Standard data for meanwhileinamerica.com

Resource Scan

Scan Details

Site Domain meanwhileinamerica.com
Base Domain meanwhileinamerica.com
Scan Status Ok
Last Scan4/12/2025, 8:16:26 AM
Next Scan 5/12/2025, 8:16:26 AM

Last Scan

Scanned4/12/2025, 8:16:26 AM
URL https://meanwhileinamerica.com/robots.txt
Domain IPs 104.21.3.13, 172.67.130.5, 2606:4700:3031::6815:30d, 2606:4700:3033::ac43:8205
Response IP 172.67.130.5
Found Yes
Hash eb0f11b83c86e71d83d9191ad957928cea7a185437d54eba5417186c046bd883
SimHash 635df1d2c439

Groups

*

Rule Path
Disallow /x/
Disallow /LICENSE.txt
Disallow /error_log
Disallow /google430f9c71cc66dacd.html
Disallow /pinterest-ebbff.html
Disallow /ganalytics.js
Disallow /xmlrpc.php

Other Records

Field Value
crawl-delay 20

aboundexbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bot/0.1 (bot for jce)

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

discoverybot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou pic spider

Rule Path
Disallow /

sogou pic spider/3.0

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

y!j-asr/0.1 crawler

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

Comments

  • For syntax checking, see:
  • http://www.sxw.org.uk/computing/robots/check.html
  • Directories / Paths
  • Note: First x dir is to prevent spidering/following Yourls links (affs)
  • Files
  • Bots and Scrapers