crosswordhelper.com
robots.txt

Robots Exclusion Standard data for crosswordhelper.com

Resource Scan

Scan Details

Site Domain crosswordhelper.com
Base Domain crosswordhelper.com
Scan Status Ok
Last Scan2024-09-23T00:39:03+00:00
Next Scan 2024-09-30T00:39:03+00:00

Last Scan

Scanned2024-09-23T00:39:03+00:00
URL https://crosswordhelper.com/robots.txt
Domain IPs 13.33.88.114, 13.33.88.53, 13.33.88.57, 13.33.88.64
Response IP 13.33.88.53
Found Yes
Hash 57683c7e8dcec9756763d74157c673bb19872c17990a88a63f0043c01a8f7d27
SimHash 387b8d174d53

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
sitemap https://crosswordhelper.com/static/sitemaps/sitemap_index.xml
sitemap https://crosswordhelper.com/dynamic/sitemaps/sitemap_popular_index.xml
sitemap https://crosswordhelper.com/dynamic/sitemaps/daily-crosswords/sitemap_daily_crosswords_index.xml

Comments

  •  This will be the robots.txt to be used in production.
  •  TODO: Update the rules below when we are ready for Crawlers to visit this new domain