Log In Sign Up

semanticscholar.org
robots.txt

Robots Exclusion Standard data for semanticscholar.org

Resource Scan

Scan Details

Site Domain semanticscholar.org
Base Domain semanticscholar.org
Scan Status Ok
Last Scan2024-05-06T12:44:48+00:00
Next Scan 2024-05-20T12:44:48+00:00

Last Scan

Scanned2024-05-06T12:44:48+00:00
URL https://semanticscholar.org/robots.txt
Redirect https://www.semanticscholar.org/robots.txt
Redirect Domain www.semanticscholar.org
Redirect Base semanticscholar.org
Domain IPs 13.226.120.22, 13.226.120.27, 13.226.120.46, 13.226.120.57, 2600:9000:23d2:3400:6:4565:580:93a1, 2600:9000:23d2:5c00:6:4565:580:93a1, 2600:9000:23d2:5e00:6:4565:580:93a1, 2600:9000:23d2:8a00:6:4565:580:93a1, 2600:9000:23d2:b800:6:4565:580:93a1, 2600:9000:23d2:c00:6:4565:580:93a1, 2600:9000:23d2:e400:6:4565:580:93a1, 2600:9000:23d2:ea00:6:4565:580:93a1
Redirect IPs 13.226.120.22, 13.226.120.27, 13.226.120.46, 13.226.120.57, 2600:9000:2666:1000:6:4565:580:93a1, 2600:9000:2666:1400:6:4565:580:93a1, 2600:9000:2666:2600:6:4565:580:93a1, 2600:9000:2666:6800:6:4565:580:93a1, 2600:9000:2666:6c00:6:4565:580:93a1, 2600:9000:2666:7800:6:4565:580:93a1, 2600:9000:2666:7c00:6:4565:580:93a1, 2600:9000:2666:9e00:6:4565:580:93a1
Response IP 18.155.68.3
Found Yes
Hash c91d2f860bd8e57614c261217ae8071e7cbb10165a321504895e300606e60205
SimHash 7614e5147ec3

Groups

*

Rule Path
Disallow /search
Disallow /error
Disallow /me
Disallow /api
Disallow /author/*/claim
Disallow /author/*?
Disallow /paper/*?
Disallow /reader/
Allow /paper/*?p2df
Back to top

Other Records

Field Value
sitemap https://www.semanticscholar.org/sitemap_author_index.xml
sitemap https://www.semanticscholar.org/sitemap_paper_index.xml
sitemap https://www.semanticscholar.org/sitemap_topic_index.xml
Back to top

Comments

  • We are a non-profit research institute. If you would like to collaborate with us,
  • please contact us at: ai2-info@allenai.org
  • Or check out our public API http://api.semanticscholar.org/
Back to top