theshelbyreport.com
robots.txt

Robots Exclusion Standard data for theshelbyreport.com

Resource Scan

Scan Details

Site Domain theshelbyreport.com
Base Domain theshelbyreport.com
Scan Status Ok
Last Scan2026-03-01T12:15:41+00:00
Next Scan 2026-03-08T12:15:41+00:00

Last Scan

Scanned2026-03-01T12:15:41+00:00
URL https://theshelbyreport.com/robots.txt
Domain IPs 104.26.4.130, 104.26.5.130, 172.67.69.87
Response IP 104.26.4.130
Found Yes
Hash 53ff93f6f491d454e2d76ad4d33ccdd5ee9da7492cbf45bdd92c5fca31c3b722
SimHash d6204192c6bc

Groups

googlebot-news

Rule Path
Allow /
Allow /tag/

bingbot

Rule Path
Allow /
Allow /tag/

applebot

Rule Path
Allow /
Allow /feed/gn

*

Rule Path
Allow /wp-content/
Allow /wp-includes/
Disallow /wp-admin/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /feed/
Disallow /wp-login.php
Disallow /search/
Disallow *?s=*
Disallow *?p=*
Disallow *%26p%3D*
Disallow *%26preview%3D*
Disallow /tag/
Disallow /author/
Disallow /404-error/
Disallow /cgi-bin/
Disallow /envira/

semrushbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

archivebot

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

cohere-training-data-crawler

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

europarchive.org

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ia_archiver-web.archive.org

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

linkarchiver

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

nicecrawler

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

primalbot

Rule Path
Disallow /

web-archive-net.com.bot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

tiktokspider

Rule Path
Disallow /

iboubot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
sitemap https://www.theshelbyreport.com/sitemap_index.xml
sitemap https://theshelbyreport.com/news-sitemap.xml