Log In Sign Up

edtech.unc.edu
robots.txt

Robots Exclusion Standard data for edtech.unc.edu

Resource Scan

Scan Details

Site Domain edtech.unc.edu
Base Domain unc.edu
Scan Status Ok
Last Scan2025-08-30T13:58:13+00:00
Next Scan 2025-09-29T13:58:13+00:00

Last Scan

Scanned2025-08-30T13:58:13+00:00
URL https://edtech.unc.edu/robots.txt
Domain IPs 23.185.0.4, 2620:12a:8000::4, 2620:12a:8001::4
Response IP 23.185.0.4
Found Yes
Hash e7db9e28452dc5f1003ecdfe449d86433ca53773128d5f63ac2ad3ba3399775d
SimHash 02d74852e2a0

Groups

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

bingbot

Rule Path
Allow /

yandexbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

google-cloudvertexbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

duckassistbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

cohere-training-data-crawler

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

geedobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

pangubot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

webzio-extended

Rule Path
Disallow /

youbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

ai2bot

Rule Path
Disallow /

kangaroo bot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

*

Rule Path
Disallow /*.php$
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Disallow /wp-content/themes/
Disallow /readme.html
Disallow /license.txt
Disallow /wp-json/
Disallow */feed/
Disallow */trackback/
Disallow */page/*/?
Disallow /tag/
Disallow /category/*/*
Disallow /*?
Disallow /*?s=
Disallow /*%26s%3D
Disallow /search/
Disallow /author/
Disallow /archive/
Disallow */wp-login.php
Disallow */wp-json/*
Disallow */wp-admin/*
Disallow */?attachment_id=*
Disallow */?s=*
Disallow */?taxonomy=nav_menu*
Disallow */?eventDisplay=past*
Disallow */?eventDisplay=photo*
Disallow */?post_type=tribe_events&eventDisplay=day*
Disallow */?post_type=tribe_events&eventDisplay=week*
Disallow */?post_type=tribe_events&eventDisplay=month*
Disallow */?tribe-bar-date=*
Disallow *%26eventDisplay%3Dpast*
Disallow *%26eventDisplay%3Dphoto*
Disallow *%26tribe-bar-date%3D*
Disallow */2009/*
Disallow */2010/*
Disallow */2011/*
Disallow */2012/*
Disallow */2013/*
Disallow */2014/*
Disallow */2015/*
Disallow */2016/*
Disallow */2017/*
Disallow */2018/*
Disallow */2019/*
Disallow */2020/*
Disallow */2021/*
Disallow */2022/*
Disallow */2023/*
Disallow */author/*
Disallow */category/*
Disallow */events/*
Disallow */organizer/*
Disallow */scripts/webalert.js?__ver=*
Disallow */tag/*
Disallow */venue/*
Allow /wp-content/uploads/
Allow /wp-content/themes/*/assets/
Allow /wp-content/themes/*/images/
Allow /wp-content/themes/*/css/
Allow /wp-content/themes/*/js/
Allow /*?p=*
Allow /*?paged=*
Allow /*?sitemap=
Allow /page-sitemap.xml
Allow /post-sitemap.xml
Allow /category-sitemap.xml
Allow /author-sitemap.xml
Allow /sitemap_index.xml
Allow /news-sitemap.xml
Allow /product-sitemap.xml
Allow /portfolio-sitemap.xml

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://edtech.unc.edu/sitemap_index.xml
Back to top

Comments

  • Essential Search Engine Bots - Allow
  • Block AI Training & Large Language Model Bots
  • Block Social Media & Content Aggregator Bots
  • Block Aggressive SEO & Analysis Bots
  • Global Rules for All Other Bots
  • WordPress specific paths
  • Blocks access to all PHP files (including xmlrpc.php, wp-*.php, etc.)
  • Allow specific content directories
  • Allow pagination
  • Crawl-delay for remaining bots
  • Allow Yoast SEO sitemap patterns
  • START YOAST BLOCK
  • ---------------------------
  • ---------------------------
  • END YOAST BLOCK
Back to top