crosswordnexus.com
robots.txt

Robots Exclusion Standard data for crosswordnexus.com

Resource Scan

Scan Details

Site Domain crosswordnexus.com
Base Domain crosswordnexus.com
Scan Status Ok
Last Scan2024-10-08T11:58:39+00:00
Next Scan 2024-10-15T11:58:39+00:00

Last Scan

Scanned2024-10-08T11:58:39+00:00
URL https://crosswordnexus.com/robots.txt
Domain IPs 162.241.158.152
Response IP 162.241.158.152
Found Yes
Hash 6d0b2f892ef2cdafb5d8b3f689c694f620bd44a7855cd108dc6acc7390975b8d
SimHash 0a04da04c213

Groups

*

Rule Path
Disallow /new-site/
Disallow /webapp/

mediapartners-google

Rule Path
Disallow

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

netseer

Rule Path
Disallow /

yandex

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

maxpointcrawler
maxpointcrawler/nutch-1.1

Rule Path
Disallow /

proximic

Rule Path
Disallow /

Other Records

Field Value
sitemap http://cdn.attracta.com/sitemap/1042845.xml.gz

Comments

  • Google Adsense
  • Baiduspider
  • Netseer
  • Yandex
  • Sogou
  • MaxPointCrawler
  • Proximic