Log In Sign Up

shaunc.com
robots.txt

Robots Exclusion Standard data for shaunc.com

Resource Scan

Scan Details

Site Domain shaunc.com
Base Domain shaunc.com
Scan Status Ok
Last Scan2024-09-09T04:56:13+00:00
Next Scan 2024-09-23T04:56:13+00:00

Last Scan

Scanned2024-09-09T04:56:13+00:00
URL https://shaunc.com/robots.txt
Domain IPs 172.93.52.73
Response IP 172.93.52.73
Found Yes
Hash 1be1051050055c339578304d58023f88f22f7eb3ce070aec15d404e3291c68dc
SimHash 509d425f25a1

Groups

ia_archiver

Rule Path
Disallow /

special_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

mixnodecache

Rule Path
Disallow /

checkmarknetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)

Rule Path
Disallow /

macocu

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

friendly_crawler

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

daum

Rule Path
Disallow /

senutobot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

sitescorebot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

dataprovider

Rule Path
Disallow /

neevabot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dnbcrawler-analytics

Rule Path
Disallow /

ioncrawl

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

*

Rule Path
Disallow /private_passwords/
Disallow /static/
Back to top

Other Records

Field Value
sitemap https://shaunc.com/blog/sitemap.xml
Back to top

Comments

  • Sitemap for blog content
Back to top