rotter.net
robots.txt

Robots Exclusion Standard data for rotter.net

Resource Scan

Scan Details

Site Domain rotter.net
Base Domain rotter.net
Scan Status Ok
Last Scan2024-11-13T05:08:54+00:00
Next Scan 2024-11-20T05:08:54+00:00

Last Scan

Scanned2024-11-13T05:08:54+00:00
URL https://rotter.net/robots.txt
Domain IPs 104.26.2.209, 104.26.3.209, 172.67.71.88, 2606:4700:20::681a:2d1, 2606:4700:20::681a:3d1, 2606:4700:20::ac43:4758
Response IP 172.67.71.88
Found Yes
Hash e863ff79e89feb2018c28c39a480fd5dcb9d9ac37d9c34b306af4fcec99fc0c9
SimHash f0c9da464395

Groups

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

*

Rule Path
Disallow /cgi-bin/forum/dcadmin.cgi
Disallow /cgi-bin/forum/dcboard.cgi?az=subscribe_to_topic
Disallow /cgi-bin/forum/dcboard.cgi?az=do_anchor
Disallow /cgi-bin/forum/dcboard.cgi?quote=not_empty
Disallow /cgi-bin/forum/dcboard.cgi?az=edit
Disallow /cgi-bin/forum/dcboard.cgi?az=post
Disallow /cgi-bin/forum/dcboard.cgi?az=add_bookmark
Disallow /cgi-bin/forum/dcboard.cgi?az=up
Disallow /cgi-bin/forum/dcboard.cgi?az=anchor_down
Disallow /cgi-bin/forum/dcboard.cgi?az=printer_format

Warnings

  • `request-rate` is not a known field.
  • `visit-time` is not a known field.