10bestmedia.com
robots.txt

Robots Exclusion Standard data for 10bestmedia.com

Resource Scan

Scan Details

Site Domain 10bestmedia.com
Base Domain 10bestmedia.com
Scan Status Ok
Last Scan2025-03-06T15:35:37+00:00
Next Scan 2025-03-13T15:35:37+00:00

Last Scan

Scanned2025-03-06T15:35:37+00:00
URL https://10bestmedia.com/robots.txt
Redirect https://10best.usatoday.com/robots.txt
Redirect Domain 10best.usatoday.com
Redirect Base usatoday.com
Domain IPs 100.28.104.175, 2600:1f18:297:ba0e:7900:f622:48a8:f266, 2600:1f18:297:ba18:6ddc:5d8f:6053:f003, 2600:1f18:297:ba24:b508:76de:f4a:df34, 54.157.108.158, 98.83.184.133
Redirect IPs 151.101.130.62, 151.101.194.62, 151.101.2.62, 151.101.66.62
Response IP 199.232.46.62
Found Yes
Hash 6feb449c33edfd6b6563aed57ad4da5f13f7fd7dca212063f82ef1da756a45a7
SimHash 700d1951c480

Groups

*

Rule Path
Disallow /*/iframe/
Disallow /legal/
Disallow /insider/
Disallow /status.html
Disallow /awards/upload/
Disallow /awards/nominate/

anthropic-ai

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

news-please

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

peer39_crawler/1.0

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /
Allow /

Other Records

Field Value
sitemap https://10best.usatoday.com/sitemap.xml
sitemap https://10best.usatoday.com/sitemaps/sitemap-overview-and-cities-1.xml.gz
sitemap https://10best.usatoday.com/sitemaps/sitemap-articles-and-galleries-1.xml.gz
sitemap https://10best.usatoday.com/sitemaps/sitemap-restaurants-1.xml.gz
sitemap https://10best.usatoday.com/sitemaps/sitemap-hotels-1.xml.gz
sitemap https://10best.usatoday.com/sitemaps/sitemap-shopping-1.xml.gz
sitemap https://10best.usatoday.com/sitemaps/sitemap-nightlife-1.xml.gz
sitemap https://10best.usatoday.com/sitemaps/sitemap-attractions-1.xml.gz