edx.org
robots.txt
Robots Exclusion Standard data for edx.org
Resource Scan
Scan Details
Site Domain | edx.org |
Base Domain | edx.org |
Scan Status | Ok |
Last Scan | 2024-04-23T21:39:11+00:00 |
Next Scan | 2024-05-23T21:39:11+00:00 |
Last Scan
Scanned | 2024-04-23T21:39:11+00:00 |
URL | https://edx.org/robots.txt |
Redirect | https://www.edx.org/robots.txt |
Redirect Domain | www.edx.org |
Redirect Base | edx.org |
Domain IPs | 13.33.88.115, 13.33.88.124, 13.33.88.126, 13.33.88.81 |
Redirect IPs | 104.16.189.80, 104.16.190.80, 104.16.191.80, 104.16.192.80, 104.16.193.80, 2606:4700::6810:bd50, 2606:4700::6810:be50, 2606:4700::6810:bf50, 2606:4700::6810:c050, 2606:4700::6810:c150 |
Response IP | 104.16.191.80 |
Found | Yes |
Hash | a6a4f315c3e4e4235998517f56acc1dcb3ae264e8fcc85d89f7ada2acd58db6b |
SimHash | bc967d49d760 |
Groups
*
Rule | Path |
---|---|
Allow | /misc/*.css$ |
Allow | /misc/*.css? |
Allow | /misc/*.js$ |
Allow | /misc/*.js? |
Allow | /misc/*.gif |
Allow | /misc/*.jpg |
Allow | /misc/*.jpeg |
Allow | /misc/*.png |
Allow | /modules/*.css$ |
Allow | /modules/*.css? |
Allow | /modules/*.js$ |
Allow | /modules/*.js? |
Allow | /modules/*.gif |
Allow | /modules/*.jpg |
Allow | /modules/*.jpeg |
Allow | /modules/*.png |
Allow | /profiles/*.css$ |
Allow | /profiles/*.css? |
Allow | /profiles/*.js$ |
Allow | /profiles/*.js? |
Allow | /profiles/*.gif |
Allow | /profiles/*.jpg |
Allow | /profiles/*.jpeg |
Allow | /profiles/*.png |
Allow | /themes/*.css$ |
Allow | /themes/*.css? |
Allow | /themes/*.js$ |
Allow | /themes/*.js? |
Allow | /themes/*.gif |
Allow | /themes/*.jpg |
Allow | /themes/*.jpeg |
Allow | /themes/*.png |
Disallow | /includes/ |
Disallow | /misc/ |
Disallow | /modules/ |
Disallow | /profiles/ |
Disallow | /scripts/ |
Disallow | /themes/ |
Disallow | /preview/ |
Disallow | /es/preview/ |
Disallow | /secure-preview/ |
Disallow | /es/secure-preview/ |
Disallow | /new/ |
Disallow | /es/new/ |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
Comments