diversityandability.com
robots.txt

Robots Exclusion Standard data for diversityandability.com

Resource Scan

Scan Details

Site Domain diversityandability.com
Base Domain diversityandability.com
Scan Status Ok
Last Scan2025-10-08T04:19:08+00:00
Next Scan 2025-11-07T04:19:08+00:00

Last Scan

Scanned2025-10-08T04:19:08+00:00
URL https://diversityandability.com/robots.txt
Redirect https://www.callingallminds.com/robots.txt
Redirect Domain www.callingallminds.com
Redirect Base callingallminds.com
Domain IPs 2001:8d8:100f:f000::200, 217.160.0.109
Redirect IPs 216.150.1.129, 216.150.16.129
Response IP 216.150.1.193
Found Yes
Hash 427ef6ed962abe87b7c389f3e7806657ad713d6b216314ead9ab1850989fd2b3
SimHash 6309685265a0

Groups

*

Rule Path Comment
Allow / -
Allow /resources/ -
Allow /resources/access-to-work -
Allow /resources/disabled-students-allowance -
Allow /resources/assistive-technology-guide -
Allow /workplace/ -
Allow /education/ -
Allow /services -
Allow /about -
Allow /insights -
Allow /images/ -
Allow /_next/static/ -
Allow /favicon.ico -
Disallow /api/ -
Disallow /_next/ -
Disallow /admin/ -
Disallow /private/ -
Disallow /*?* -
Disallow /* *

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://callingallminds.com/sitemap.xml

Comments

  • Robots.txt for Calling All Minds
  • Optimized for accessibility and neurodiversity content
  • High-priority pages for crawling
  • Allow important static assets
  • Block admin and private areas
  • Block duplicate content
  • Sitemap location
  • Crawl delay for respectful crawling
  • Special instructions for major search engines
  • Block AI training crawlers if desired (optional)
  • User-agent: GPTBot
  • Disallow: /
  • User-agent: ChatGPT-User
  • Disallow: /