Log In Sign Up

itest.2ch.net
robots.txt

Robots Exclusion Standard data for itest.2ch.net

Resource Scan

Scan Details

Site Domain itest.2ch.net
Base Domain 2ch.net
Scan Status Failed
Failure StageFetching resource.
Failure ReasonRequest timed out.
Last Scan2024-04-26T13:00:35+00:00
Next Scan 2024-07-25T13:00:35+00:00

Last Successful Scan

Scanned2023-03-11T04:12:20+00:00
URL https://itest.2ch.net/robots.txt
Redirect http://itest.5ch.net/robots.txt
Redirect Domain itest.5ch.net
Redirect Base 5ch.net
Domain IPs 104.20.49.205, 104.20.50.205, 172.67.0.175
Redirect IPs 172.64.96.3, 172.64.97.3
Response IP 172.64.97.3
Found Yes
Hash c6b2923127e0967f99f1a2ed0a3b8dc58ee70dcf2f10bf96d0abe75d952853f5
SimHash b2850d8d6550

Groups

*

No rules defined. All paths allowed.

Back to top

Other Records

Field Value
sitemap https://itest.5ch.net/sitemap.xml
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • User-agent: *
  • Disallow: /
Back to top