Log In Sign Up

blog.symbolic.city
robots.txt

Robots Exclusion Standard data for blog.symbolic.city

Resource Scan

Scan Details

Site Domain blog.symbolic.city
Base Domain symbolic.city
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-12-17T05:20:20+00:00
Next Scan 2025-03-17T05:20:20+00:00

Last Successful Scan

Scanned2024-08-26T03:54:07+00:00
URL http://blog.symbolic.city/robots.txt
Domain IPs 159.89.45.21
Response IP 159.89.45.21
Found Yes
Hash 749d8dc9b481edbd4537294164d3f07bf3aec94ab64490766f23c548f7eb82cb
SimHash 7094d140c5a0

Groups

*

Rule Path
Disallow /assets/fonts/
Disallow /assets/images/

adidxbot

Rule Path
Disallow /

adsbot-google

Rule Path
Disallow /

adsbot-google-mobile

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

apis-google

Rule Path
Disallow /

applebot

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

cognitiveseo

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

googlebot-news

Rule Path
Disallow /

google-storebot

Rule Path
Disallow /

googleproducer

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mediapartners-google

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

msnbot-media

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

oncrawl

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

peer39_crawler/1.0

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

redditbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

youbot

Rule Path
Disallow /
Back to top

Comments

  • hides unnecessary files from all search engines
  • disallows specific bots
Back to top

Warnings

  • 1 invalid line.
Back to top