Log In Sign Up

microchipdeveloper.com
robots.txt

Robots Exclusion Standard data for microchipdeveloper.com

Resource Scan

Scan Details

Site Domain microchipdeveloper.com
Base Domain microchipdeveloper.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2025-07-28T10:00:34+00:00
Next Scan 2025-10-26T10:00:34+00:00

Last Successful Scan

Scanned2025-03-08T09:17:40+00:00
URL https://microchipdeveloper.com/robots.txt
Redirect https://developerhelp.microchip.com/robots.txt
Redirect Domain developerhelp.microchip.com
Redirect Base microchip.com
Domain IPs 198.175.253.62
Redirect IPs 198.175.253.62
Response IP 198.175.253.62
Found Yes
Hash 9b64341b7add409310ef13c05e85c53760370ec69fac18dfbb03b6268f65c4d2
SimHash b947336dadf5

Groups

*

Rule Path
Disallow /xwiki/wiki/
Disallow /xwiki/bin/viewattachrev/
Disallow /xwiki/bin/viewrev/
Disallow /xwiki/bin/pdf/
Disallow /xwiki/bin/tex/
Disallow /xwiki/bin/edit/
Disallow /xwiki/bin/create/
Disallow /xwiki/bin/inline/
Disallow /xwiki/bin/preview/
Disallow /xwiki/bin/save/
Disallow /xwiki/bin/saveandcontinue/
Disallow /xwiki/bin/rollback/
Disallow /xwiki/bin/deleteversions/
Disallow /xwiki/bin/cancel/
Disallow /xwiki/bin/delete/
Disallow /xwiki/bin/deletespace/
Disallow /xwiki/bin/undelete/
Disallow /xwiki/bin/reset/
Disallow /xwiki/bin/register/
Disallow /xwiki/bin/propupdate/
Disallow /xwiki/bin/propadd/
Disallow /xwiki/bin/propdisable/
Disallow /xwiki/bin/propenable/
Disallow /xwiki/bin/propdelete/
Disallow /xwiki/bin/objectadd/
Disallow /xwiki/bin/commentadd/
Disallow /xwiki/bin/commentsave/
Disallow /xwiki/bin/objectsync/
Disallow /xwiki/bin/objectremove/
Disallow /xwiki/bin/attach/
Disallow /xwiki/bin/upload/
Disallow /xwiki/bin/download/
Disallow /xwiki/bin/temp/
Disallow /xwiki/bin/downloadrev/
Disallow /xwiki/bin/dot/
Disallow /xwiki/bin/svg/
Disallow /xwiki/bin/delattachment/
Disallow /xwiki/bin/login/
Disallow /xwiki/bin/loginsubmit/
Disallow /xwiki/bin/loginerror/
Disallow /xwiki/bin/logout/
Disallow /xwiki/bin/charting/
Disallow /xwiki/bin/lock/
Disallow /xwiki/bin/redirect/
Disallow /xwiki/bin/admin/
Disallow /xwiki/bin/export/
Disallow /xwiki/bin/import/
Disallow /xwiki/bin/distribution/
Disallow /xwiki/bin/imagecaptcha/
Disallow /xwiki/bin/unknown/
Disallow /xwiki/bin/view/XWiki/
Disallow /xwiki/bin/view/Development/
Disallow /xwiki/bin/view/assets/
Disallow /xwiki/bin/view/api/
Disallow /xwiki/bin/view/Sandbox/
Disallow /xwiki/bin/view/Admin/
Disallow /xwiki/bin/view/Stats/
Disallow /xwiki/bin/view/Panels/
Disallow /xwiki/bin/view/Main/Search
Disallow /xwiki/rest/
Disallow /xwiki/*?*xpage=*
Allow /xwiki/bin/get/Main/SiteMap
Allow /xwiki/bin/download/*.png$
Allow /xwiki/bin/download/*.jpg$
Allow /xwiki/bin/download/*.jpeg$
Allow /xwiki/bin/download/*.gif$
Back to top

Other Records

Field Value
sitemap https://developerhelp.microchip.com/xwiki/bin/get/Main/SiteMap
Back to top

Comments

  • Use * instead of /bin/ to also match path-based
  • subwikis like "/xwiki/wiki/wikialias/view/Space/Page"
  • Changed to bin and explicitly disallowed subwikis so pages with these words in titles won't get blocked
  • Disallow sub-wikis
  • Disallow specific actions on main wiki
  • Google officially recommends that you do not disallow crawling of the JS and CSS files which are now actually rendered by the crawler bot and used to better index your content.
  • Disallow: /xwiki/bin/skin/
  • Disallow: /xwiki/bin/jsx/
  • Disallow: /xwiki/bin/ssx/
  • Note: In addition, this matches both old /xwiki/bin/webjars/
  • and new /xwiki/webjars paths.
  • Google officially recommends that you do not disallow crawling of the JS and CSS files which are now actually rendered by the crawler bot and used to better index your content.
  • Disallow: /xwiki/bin/webjars/
  • Don't index additional UI-related resources.
  • Google officially recommends that you do not disallow crawling of the JS and CSS files which are now actually rendered by the crawler bot and used to better index your content.
  • Disallow: /xwiki/resources/
  • Don't index sandbox content since it's sample content
  • Don't index Admin space since it contains Admin stuff.
  • Note that the Admin space is protected by permissions
  • anyway but this acts as a safety net to not have private
  • info leaked on the internet ;)
  • Don't index Stats data (just because it's not useful and
  • those pages are a bit CPU intensive)
  • Don't index Panels data (because we don't want it
  • indexed on the internet)
  • Don't index the search page.
  • Don't index the REST API.
  • These are just UI elements which can cause infinite loops in
  • web crawlers. See https://jira.xwiki.org/browse/XWIKI-16915
  • These are legitimate pages that need to be allowed because their URL contains disallowed elements
Back to top