Log In Sign Up

semissourian.com
robots.txt

Robots Exclusion Standard data for semissourian.com

Resource Scan

Scan Details

Site Domain semissourian.com
Base Domain semissourian.com
Scan Status Ok
Last Scan2025-03-27T04:27:50+00:00
Next Scan 2025-04-03T04:27:50+00:00

Last Scan

Scanned2025-03-27T04:27:50+00:00
URL https://semissourian.com/robots.txt
Redirect https://www.semissourian.com/robots.txt
Redirect Domain www.semissourian.com
Redirect Base semissourian.com
Domain IPs 76.76.21.21
Redirect IPs 3.170.229.106, 3.170.229.116, 3.170.229.18, 3.170.229.77
Response IP 3.170.229.77
Found Yes
Hash 9e29f9ca799b126467bcdf5bec1d80de71f073fdbf1af66a27bcf02ec1d45030
SimHash 49108900c4f5

Groups

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

youbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

*

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.semissourian.com/sitemap.xml
Back to top

Comments

  • Block specific AI crawlers
  • Allow all other crawlers
  • Host
  • Sitemaps
Back to top

Warnings

  • `host` is not a known field.
Back to top