i.turner.ncaa.com
robots.txt

Robots Exclusion Standard data for i.turner.ncaa.com

Resource Scan

Scan Details

Site Domain i.turner.ncaa.com
Base Domain ncaa.com
Scan Status Ok
Last Scan2025-10-26T20:23:09+00:00
Next Scan 2025-11-25T20:23:09+00:00

Last Scan

Scanned2025-10-26T20:23:09+00:00
URL https://i.turner.ncaa.com/robots.txt
Redirect https://www.ncaa.com/robots.txt
Redirect Domain www.ncaa.com
Redirect Base ncaa.com
Domain IPs 184.51.98.61, 2600:1413:5000:986::3134, 2600:1413:5000:98f::3134
Redirect IPs 184.51.98.61, 2600:1413:5000:986::3134, 2600:1413:5000:98f::3134
Response IP 184.51.98.61
Found Yes
Hash 6c302fdb1cbc7a265903d8864b18e3c0021811b3ec721b3e0a4d25003b82cdbc
SimHash 339e259486e2

Groups

*

Rule Path
Allow /core/*.css$
Allow /core/*.css?
Allow /core/*.js$
Allow /core/*.js?
Allow /core/*.gif
Allow /core/*.jpg
Allow /core/*.jpeg
Allow /core/*.png
Allow /core/*.svg
Disallow /core/
Disallow /README.txt
Disallow /web.config
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips
Disallow /node/add/
Disallow /search/
Disallow /webview
Disallow /user/register
Disallow /user/password
Disallow /user/login
Disallow /user/logout
Disallow /index.php/admin/
Disallow /index.php/comment/reply/
Disallow /index.php/filter/tips
Disallow /index.php/node/add/
Disallow /index.php/search/
Disallow /index.php/user/password
Disallow /index.php/user/register
Disallow /index.php/user/login
Disallow /index.php/user/logout
Disallow /includes/
Disallow /core/
Disallow /misc/
Disallow /modules/
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php
Disallow /saml_login
Disallow /simplesaml
Disallow /imce
Disallow /?q=*
Disallow /nst-fa/
Disallow /index.php
Disallow /index.php/*
Allow /sites/default/files/external/assets/
Allow /sites/default/files/external/printable-bracket/
Allow /sites/default/files/external/gametool/brackets/
Allow /sites/default/files/legacy/
Allow /modules/custom/*.css
Allow /modules/custom/*/css/
Allow /modules/custom/*/modules/*/css/
Allow /themes/custom/*/css/
Allow /modules/contrib/*.css
Allow /modules/contrib/*/css/
Allow /modules/contrib/*/modules/*/css/
Allow /themes/contrib/*/css/
Allow /modules/custom/*/img/
Allow /modules/custom/*/modules/*/img/
Allow /themes/custom/*/img/
Allow /themes/custom/*/assets/
Allow /modules/contrib/*/img/
Allow /modules/contrib/*/modules/*/img/
Allow /themes/contrib/*/img/
Allow /themes/contrib/*/assets/
Allow /modules/custom/*/js/
Allow /modules/custom/*/lib/
Allow /modules/custom/*/templates/compiled/
Allow /modules/custom/*/dist/
Allow /modules/custom/*/pkg/
Allow /themes/custom/*/js/
Allow /modules/contrib/*/js/
Allow /themes/contrib/*/js/
Allow /_ssr/live-updates/
Allow /_nuxt/
Allow /modules/custom/casablanca_libraries/

ai2bot
ai2bot-dolma
anthropic-ai
applebot-extended
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
cohere-training-data-crawler
diffbot
facebookbot
google-extended
gptbot
kangaroo bot
meta-externalagent
meta-externalagent
meta-externalfetcher
meta-externalfetcher
omgili
omgilibot
timpibot
webzio-extended

Rule Path
Disallow /

magpie-crawler
scrapy

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.ncaa.com/sites/default/files/public/sitemaps/article-sitemap.xml
sitemap https://www.ncaa.com/sites/default/files/public/sitemaps/brackets-sitemap.xml
sitemap https://www.ncaa.com/sites/default/files/public/sitemaps/superpage-sitemap.xml
sitemap https://www.ncaa.com/sites/default/files/public/sitemaps/video-sitemap.xml
sitemap https://www.ncaa.com/sites/default/files/public/sitemaps/page-sitemap.xml
sitemap https://www.ncaa.com/sites/default/files/public/sitemaps/iframe-sitemap.xml
sitemap https://www.ncaa.com/sites/default/files/public/sitemaps/ranking-sitemap.xml
sitemap https://www.ncaa.com/sites/default/files/public/sitemaps/layout_page-sitemap.xml
sitemap https://www.ncaa.com/sites/default/files/public/sitemaps/tabbed_content-sitemap.xml
sitemap https://www.ncaa.com/sites/default/files/public/sitemaps/history-sitemap.xml
sitemap https://www.ncaa.com/sitemap/news.xml
sitemap https://www.ncaa.com/sites/default/files/public/sitemaps/seo-sitemap.xml

Comments

  • Standard user agents
  • CSS, JS, Images
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • old/migrated file sources
  • css
  • img
  • js
  • libraries
  • AI Training Bots
  • Data Collection Bots
  • Sitemaps