Log In Sign Up

educationdoctoralprograms.com
robots.txt

Robots Exclusion Standard data for educationdoctoralprograms.com

Resource Scan

Scan Details

Site Domain educationdoctoralprograms.com
Base Domain educationdoctoralprograms.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2024-07-02T16:47:55+00:00
Next Scan 2024-09-30T16:47:55+00:00

Last Successful Scan

Scanned2024-03-05T16:46:21+00:00
URL https://educationdoctoralprograms.com/robots.txt
Redirect https://eddguide.org/robots.txt
Redirect Domain eddguide.org
Redirect Base eddguide.org
Domain IPs 173.231.192.44
Redirect IPs 140.82.4.225
Response IP 140.82.4.225
Found Yes
Hash 7d4b972da15cd8a71220827243d4520270514fc673843108e80e193565c6ab05
SimHash e6ed4e71f4b0

Groups

*
adsbot-google

Rule Path
Disallow /wp-json/
Disallow /?rest_route=
Disallow /wp-admin/
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /xmlrpc.php

*
adsbot-google

Rule Path
Disallow /wp-includes/
Allow /wp-includes/css/
Allow /wp-includes/js/

*
adsbot-google

Rule Path
Disallow /?s=
Disallow /page/*/?s=
Disallow /search/

adsbot-google

Rule Path
Disallow /*.woff2

*
adsbot-google

Rule Path
Disallow /porpoiseant/
Disallow /detroitchicago/
Disallow /beardeddragon/
Disallow /tardisrocinante/
Disallow /ezoic/

*
adsbot-google

Rule Path
Disallow /workers/

*
adsbot-google

Rule Path
Disallow /~partytown

*
adsbot-google

Rule Path
Disallow /wp-content/uploads/complianz/
Disallow /?wp-ajax=

*
adsbot-google

Rule Path
Disallow /cdn-cgi/bm/cv/
Disallow /cdn-cgi/challenge-platform/
Disallow /cdn-cgi/images/trace/
Disallow /cdn-cgi/rum
Disallow /cdn-cgi/scripts/
Disallow /cdn-cgi/styles/
Disallow /cdn-fpw/sxg/

nuclei
wikido
riddler
petalbot
zoominfobot
go-http-client
node/simplecrawler
cazoodlebot
dotbot/1.0
gigabot
barkrowler
blexbot
magpie-crawler
mj12bot
ahrefsbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://eddguide.org/sitemap_index.xml
Back to top

Comments

  • Block some general WP endpoints
  • -------------------------------
  • Special handling for /wp-includes/
  • ----------------------------------
  • Block internal search
  • ---------------------
  • Adsbot doesn't ever need to crawl fonts
  • ---------------------------------------
  • Block legacy Ezoic URLs
  • -----------------------
  • Block workers
  • -------------
  • Block partytown
  • ---------------
  • Block leaky plugins
  • -------------------
  • Block leaky Cloudflare endpoints
  • --------------------------------
  • Ban noisy bots
  • --------------
  • Sitemap
  • -------
Back to top