cfw.sarna.net
robots.txt

Robots Exclusion Standard data for cfw.sarna.net

Resource Scan

Scan Details

Site Domain cfw.sarna.net
Base Domain sarna.net
Scan Status Ok
Last Scan2024-10-22T13:06:15+00:00
Next Scan 2024-11-21T13:06:15+00:00

Last Scan

Scanned2024-10-22T13:06:15+00:00
URL https://cfw.sarna.net/robots.txt
Domain IPs 13.227.254.122, 13.227.254.50, 13.227.254.66, 13.227.254.83, 2600:9000:200a:1c00:1f:1272:1d80:93a1, 2600:9000:200a:4000:1f:1272:1d80:93a1, 2600:9000:200a:5000:1f:1272:1d80:93a1, 2600:9000:200a:5800:1f:1272:1d80:93a1, 2600:9000:200a:6800:1f:1272:1d80:93a1, 2600:9000:200a:8200:1f:1272:1d80:93a1, 2600:9000:200a:a200:1f:1272:1d80:93a1, 2600:9000:200a:c00:1f:1272:1d80:93a1
Response IP 13.227.254.50
Found Yes
Hash 8b52233cb3888dc73144a9c3f40c7d355591164324732a7d53cdf03d8ac447ac
SimHash bd5c3e04ccb2

Groups

*

Rule Path
Disallow /forums/showprofile.php
Disallow /forums/printthread.php
Disallow /forums/search.php
Disallow /forums/dosearch.php
Disallow /forums/showthreaded.php
Disallow /forums/newpost.php
Disallow /forums/editpost.php
Disallow /forums/newreply.php
Disallow /forums/online.php
Disallow /forums/gotothread.php
Disallow /forums/gotonpthread.php
Disallow /forums/addfavforum.php
Disallow /forums/markallread.php
Disallow /forums/domarkallreadanon.php
Disallow /forums/doreply.php
Disallow /forums/markthreadread.php
Disallow /forums/domarkthreadread.php
Disallow /forums/showmembers.php
Disallow /forums/litepostlist.php
Disallow /forums-neveron/

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.sarna.net/news/sitemap.xml.gz
sitemap https://www.sarna.net/sitemap-index.xml

Comments

  • Crawling too much
  • Crawling too much