Log In Sign Up

shawu.edu
robots.txt

Robots Exclusion Standard data for shawu.edu

Resource Scan

Scan Details

Site Domain shawu.edu
Base Domain shawu.edu
Scan Status Ok
Last Scan2024-09-22T21:39:00+00:00
Next Scan 2024-10-22T21:39:00+00:00

Last Scan

Scanned2024-09-22T21:39:00+00:00
URL https://shawu.edu/robots.txt
Redirect https://www.shawu.edu/robots.txt
Redirect Domain www.shawu.edu
Redirect Base shawu.edu
Domain IPs 23.185.0.1, 2620:12a:8000::1, 2620:12a:8001::1
Redirect IPs 23.185.0.1, 2620:12a:8000::1
Response IP 23.185.0.1
Found Yes
Hash ad59c1cfef14665578d6f3474e4e6712a18893d6daf9e4b4c50109ac25720802
SimHash ea365558c8f9

Groups

*

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/

ahrefsbot

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/

siteimprovebot-crawler

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/

applebot

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/

bingbot

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/

duckduckbot

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/

googlebot

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/

adsbot-google

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/

mediapartners-google

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/

facebookexternalhit

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/

linkedinbot

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/

whatsapp

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/

twitterbot

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/

ia_archiver

Rule Path
Allow /
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search
Disallow /?s=
Allow /wp-includes/css/
Allow /wp-includes/js/
Back to top

Comments

  • This site is very specific about who it allows crawling from.
  • Our default is to not allow crawling:
  • Below are the crawlers that are allowed to crawl this site.
  • Below that list, you'll find paths that are blocked, even for them,
  • and then paths within those blocked paths that are allowed.
  • XML Sitemap:
  • Sitemap: https://www.shawu.edu/sitemap_index.xml
Back to top