alphaxiv.org
robots.txt

Robots Exclusion Standard data for alphaxiv.org

Resource Scan

Scan Details

Site Domain alphaxiv.org
Base Domain alphaxiv.org
Scan Status Ok
Last Scan2025-10-22T07:33:43+00:00
Next Scan 2025-11-21T07:33:43+00:00

Last Scan

Scanned2025-10-22T07:33:43+00:00
URL https://alphaxiv.org/robots.txt
Redirect https://www.alphaxiv.org/robots.txt
Redirect Domain www.alphaxiv.org
Redirect Base alphaxiv.org
Domain IPs 76.76.21.21
Redirect IPs 76.76.21.22, 76.76.21.98
Response IP 76.76.21.61
Found Yes
Hash 62f17594ac492d174cbcfe9035de638735283943aaa86b1accf03ab06ae5810d
SimHash 0515796467db

Groups

*

Rule Path
Disallow /abs/
Disallow /pdf/

googlebot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

duckassistbot

Rule Path
Allow /

archive.org_bot

Rule Path
Allow /

adsbot-google

Rule Path
Allow /abs/
Allow /pdf/

googlebot-image

Rule Path
Allow /abs/
Allow /pdf/

bingbot

Rule Path
Allow /abs/
Allow /pdf/

yandexbot

Rule Path
Allow /abs/
Allow /pdf/

baiduspider

Rule Path
Allow /abs/
Allow /pdf/

Other Records

Field Value
sitemap https://www.alphaxiv.org/sitemaps/sitemap-index.xml