Log In Sign Up

cnr.ncsu.edu
robots.txt

Robots Exclusion Standard data for cnr.ncsu.edu

Resource Scan

Scan Details

Site Domain cnr.ncsu.edu
Base Domain ncsu.edu
Scan Status Ok
Last Scan2024-06-29T20:07:09+00:00
Next Scan 2024-07-29T20:07:09+00:00

Last Scan

Scanned2024-06-29T20:07:09+00:00
URL https://cnr.ncsu.edu/robots.txt
Domain IPs 152.7.106.51
Response IP 152.7.106.51
Found Yes
Hash a074d3d6f920ce83c4f6f21fda5f6c762d77d7f013bc15890e718c01080e1685
SimHash 0704d463af82

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

rogerbot

Rule Path
Disallow /event/
Disallow /events/
Disallow /jobs/
Disallow /news/tag/
Disallow /fer/event/

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://cnr.ncsu.edu/sitemapindex.xml
sitemap https://cnr.ncsu.edu/fb/sitemapindex.xml
sitemap https://cnr.ncsu.edu/fer/sitemapindex.xml
sitemap https://cnr.ncsu.edu/prtm/sitemapindex.xml
sitemap https://cnr.ncsu.edu/news/sitemapindex.xml
sitemap https://cnr.ncsu.edu/internalresources/sitemapindex.xml
sitemap https://cnr.ncsu.edu/geospatial/sitemapindex.xml
sitemap https://cnr.ncsu.edu/jobs/sitemapindex.xml
Back to top

Comments

  • moz rogerbot setup
Back to top

Warnings

  • `crawl-limit` is not a known field.
Back to top