kuhf.org
robots.txt

Robots Exclusion Standard data for kuhf.org

Resource Scan

Scan Details

Site Domain kuhf.org
Base Domain kuhf.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-18T00:43:14+00:00
Next Scan 2024-06-17T00:43:14+00:00

Last Successful Scan

Scanned2024-04-25T00:42:02+00:00
URL https://kuhf.org/robots.txt
Redirect https://www.houstonpublicmedia.org//robots.txt
Redirect Domain www.houstonpublicmedia.org
Redirect Base houstonpublicmedia.org
Domain IPs 34.210.194.27
Redirect IPs 13.225.142.11, 13.225.142.118, 13.225.142.73, 13.225.142.97, 2600:9000:25ea:2400:3:f03:a940:93a1, 2600:9000:25ea:3000:3:f03:a940:93a1, 2600:9000:25ea:400:3:f03:a940:93a1, 2600:9000:25ea:6c00:3:f03:a940:93a1, 2600:9000:25ea:8600:3:f03:a940:93a1, 2600:9000:25ea:c00:3:f03:a940:93a1, 2600:9000:25ea:d600:3:f03:a940:93a1, 2600:9000:25ea:ec00:3:f03:a940:93a1
Response IP 18.165.171.30
Found Yes
Hash b6df2af80cdb512979c699ab0a72bd4c2942f8e9399a711744c28e6aa90e8ebd
SimHash d8205108e132

Groups

*

Rule Path
Disallow /ProdStage
Disallow /wp-admin/
Disallow /wp/wp-admin/
Allow /wp/wp-admin/admin-ajax.php
Disallow /videos/
Disallow /pages/
Disallow /news/awards/
Disallow /support/studio-society/members/
Disallow /support/affinity-council/members/
Disallow /wp-json/

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

peer39_crawler
peer39_crawler/1.0

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Allow /*?*smid=

twitterbot

Rule Path
Allow /*?*smid=

Other Records

Field Value
sitemap https://www.houstonpublicmedia.org/sitemap.xml
sitemap https://www.houstonpublicmedia.org/newssitemap.xml

Comments

  • robotstxt.org/
  • Other Bot Rules