wpengine.com
robots.txt

Robots Exclusion Standard data for wpengine.com

Resource Scan

Scan Details

Site Domain wpengine.com
Base Domain wpengine.com
Scan Status Ok
Last Scan2024-09-15T04:38:33+00:00
Next Scan 2024-09-29T04:38:33+00:00

Last Scan

Scanned2024-09-15T04:38:33+00:00
URL https://wpengine.com/robots.txt
Domain IPs 104.18.37.43, 172.64.150.213
Response IP 104.18.37.43
Found Yes
Hash dc100615951f90e2b72fe4b783c239af18a0dd9b34989d44f1c76b8ea5048cd5
SimHash d270e105a998

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow */*/audio/
Disallow /site.webmanifest
Disallow /wp-json/wpe/v1/chat-group
Disallow /wp-json/wpe/v1/opt-in/
Disallow /search/
Disallow /solution-center/tag/
Disallow /*/2023-
Disallow /*/day/
Disallow /*eventDisplay
Disallow /*/?__hstc=
Disallow /*/?utm_
Disallow /*/?nabe=
Disallow /*/?print
Disallow /*/?wtime=
Disallow /*/?coupon=
Disallow /*/?wvideo=
Disallow /*/?_hsenc=
Disallow /*/?local-download=
Disallow /*/?_ga=
Disallow /*/?es_p=
Disallow /*/?w_agcid=
Disallow /*/?kaid=
Disallow /*/?tribe-bar-date=
Disallow /*/?amp=
Disallow /*/?fl_rand_seed=
Disallow /*/?clientId=
Disallow /*/?budget=
Disallow /*/?ss-track=
Disallow /*/?language=
Disallow /*/?inf_contact_key=
Disallow /*/?ref=
Disallow /*/?s=
Disallow /*/?sa=
Disallow /*/?a=
Disallow /*/?p=
Disallow /*/?_gl=
Disallow /*/?cid=
Disallow /*/?o=

Other Records

Field Value
sitemap https://wpengine.com/sitemap_index.xml
sitemap https://wpengine.com/builders/sitemap_index.xml
sitemap https://wpengine.com/support/sitemap_index.xml
sitemap https://wpengine.com/solution-center/sitemap_index.xml
sitemap https://wpengine.com/page-sitemap.xml

Comments

  • Apply to all bots
  • Block the admin area but allow ajax callbacks
  • WEB-3507 Rationale unknown
  • Block API endpoints that don't have indexable content
  • WEB-5082 Google has a number of junk URLs to both of these paths
  • Prevent builder calendar filters to be crawled
  • To optimize crawl budget for SEO
  • Link to Yoast sitemaps