erudit.org
robots.txt

Robots Exclusion Standard data for erudit.org

Resource Scan

Scan Details

Site Domain erudit.org
Base Domain erudit.org
Scan Status Ok
Last Scan2025-11-10T05:56:21+00:00
Next Scan 2025-12-10T05:56:21+00:00

Last Scan

Scanned2025-11-10T05:56:21+00:00
URL https://erudit.org/robots.txt
Redirect https://www.erudit.org/robots.txt
Redirect Domain www.erudit.org
Redirect Base erudit.org
Domain IPs 198.168.187.86
Redirect IPs 198.168.187.86
Response IP 198.168.187.86
Found Yes
Hash 8c80c48732c31aa51aa74c508863da7f1b4cec30c09730c21c0fbc34aab58af2
SimHash 6d055c5567d1

Groups

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingpreview

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

core

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

petalbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

*

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://www.erudit.org/sitemap.xml