w3.org
robots.txt

Robots Exclusion Standard data for w3.org

Resource Scan

Scan Details

Site Domain w3.org
Base Domain w3.org
Scan Status Ok
Last Scan2025-08-12T18:35:37+00:00
Next Scan 2025-08-26T18:35:37+00:00

Last Scan

Scanned2025-08-12T18:35:37+00:00
URL https://w3.org/robots.txt
Redirect https://www.w3.org/robots.txt
Redirect Domain www.w3.org
Redirect Base w3.org
Domain IPs 104.18.22.19, 104.18.23.19, 2606:4700::6812:1613, 2606:4700::6812:1713
Redirect IPs 104.18.22.19, 104.18.23.19, 2606:4700::6812:1613, 2606:4700::6812:1713
Response IP 104.18.23.19
Found Yes
Hash 58efdcef17932beb4be8cd14198dde10378e37439bd85d902599c7352fcbf5ad
SimHash 502ea916072d

Groups

*

Rule Path
Disallow /*/wp-admin/
Disallow /*/wp-includes/
Disallow /*/wp-content/plugins/
Disallow /*/wp-content/cache/
Disallow /*/wp-content/themes/
Disallow /blog/*/trackback/
Disallow /blog/*/feed/
Disallow /blog/*/comments/
Disallow /blog/*/category/*/*
Disallow /blog/*/*/trackback/
Disallow /blog/*/*/feed/
Disallow /blog/*/*/comments/
Disallow /blog/*/*?
Disallow /community/trackback/
Disallow /community/feed/
Disallow /community/comments/
Disallow /community/category/*/*
Disallow /community/*/trackback/
Disallow /community/*/feed/
Disallow /community/*/comments/
Disallow /community/*/category/*/*
Disallow /community/*?
Disallow /Consortium/Offices/trackback/
Disallow /Consortium/Offices/feed/
Disallow /Consortium/Offices/comments/
Disallow /Consortium/Offices/category/*/*
Disallow /Consortium/Offices/*/trackback/
Disallow /Consortium/Offices/*/feed/
Disallow /Consortium/Offices/*/comments/
Disallow /Consortium/Offices/*?
Disallow /wiki/index.php?
Disallow /wiki/index.php/Help
Disallow /wiki/index.php/MediaWiki
Disallow /wiki/index.php/Special%3A
Disallow /wiki/index.php/Template
Disallow /wiki/Special%3A
Disallow /wiki/skins/
Disallow /*/wiki/index.php?
Disallow /*/wiki/index.php/Help
Disallow /*/wiki/index.php/MediaWiki
Disallow /*/wiki/index.php/Special%3A
Disallow /*/wiki/index.php/Template
Disallow /*/wiki/Special%3A
Disallow *//wiki/skins/
Disallow /2004/ontaria/basic
Disallow /Member/
Disallow /Team/
Disallow /Project/
Disallow /Web/
Disallow /Systems/
Disallow /Out-Of-Date
Disallow /2005/06/blog/
Disallow /2004/08/W3CTalks
Disallow /2007/11/Talks/search
Disallow /People/all/
Disallow /RDF/Validator/ARPServlet
Disallow /RDF/Validator/rdfval
Disallow /2003/03/Translations/byLanguage
Disallow /2003/03/Translations/byTechnology
Disallow /2005/11/Translations/Query
Disallow /2000/06/webdata/xslt
Disallow /2000/09/webdata/xslt
Disallow /2005/08/online_xslt/xslt
Disallow /Search/Mail/Public/
Disallow /2006/02/chartergen
Disallow /2004/01/pp-impl
Disallow /Consortium/supporters
Disallow /2012/pyRdfa/extract
Disallow /WAI/PF/comments/
Disallow /WAI/events/
Disallow /participate/conferences.xml
Disallow /scripts/
Disallow /People/domain/
Disallow /2005/01/yacker/
Disallow /2005/01/yacker?
Disallow /2005/07/pubrules?
Disallow /ns/hydra/console/?
Disallow /2007/08/grddl/?
Disallow /2009/07/webidl-check?
Disallow /RDF/Validator/ARPServlet?
Disallow /2000/06/webdata/xsv?
Disallow /2000/09/webdata/xsv?
Disallow /Style/CSS/members.be/
Disallow /services
Disallow /2021/05/view-gallery/
Disallow /International/questions/qa-html-language-declarations/icons/
Disallow /International/questions/qa-html-language-declarations/qa-html-language-declarations-data/icons/
Disallow /*%2C*
Disallow /WAI/beta/
Disallow /WAI/ut1/
Disallow /WAI/ut2/
Disallow /WAI/ut3/
Disallow /WAI/ut4/
Disallow /WAI/drafts/

w3c-checklink

Rule Path
Disallow

Comments

  • robots.txt for https://www.w3.org/
  • $Id: robots.txt,v 1.96 2025/07/10 17:25:43 gerald Exp $
  • the following settings apply to all bots
  • Blogs - WordPress
  • https://codex.wordpress.org/Search_Engine_Optimization_for_WordPress#Robots.txt_Optimization
  • Wikis - Mediawiki
  • https://www.mediawiki.org/wiki/Manual:Robots.txt
  • various other access-controlled or expensive areas
  • Allow W3C Link checker to check any paths on this site