Log In Sign Up

botcity.dev
robots.txt

Robots Exclusion Standard data for botcity.dev

Resource Scan

Scan Details

Site Domain botcity.dev
Base Domain botcity.dev
Scan Status Ok
Last Scan2026-02-24T23:35:32+00:00
Next Scan 2026-03-26T23:35:32+00:00

Last Scan

Scanned2026-02-24T23:35:32+00:00
URL https://botcity.dev/robots.txt
Domain IPs 198.202.211.1
Response IP 198.202.211.1
Found Yes
Hash adc32127b45e1ca14069be927b5fc81b164261d4d984757081f2cdbb76e8dfef
SimHash 68145947c793

Groups

*

Rule Path
Disallow /

googlebot

Rule Path
Allow /

gptbot

Rule Path
Allow /

meta-externalads

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /

bingbot

Rule Path
Allow /

google adsbot

Rule Path
Allow /

google image proxy

Rule Path
Allow /

googleother

Rule Path
Allow /

pinterestbot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

semrushbotbacklinks

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

google images

Rule Path
Allow /

moz dotbot

Rule Path
Allow /

yandexbot

Rule Path
Allow /

petalbot

Rule Path
Allow /

applebot

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

google api

Rule Path
Allow /

prerender

Rule Path
Allow /

mj12bot

Rule Path
Allow /

google read aloud

Rule Path
Allow /

amazon adbot

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

new relic

Rule Path
Allow /

chrome-lighthouse

Rule Path
Allow /

google storebot

Rule Path
Allow /

microsoft preview

Rule Path
Allow /

cloudflare healthchecks

Rule Path
Allow /

yandexbot

Rule Path
Disallow /

googlebot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

duckduckbot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

yeti

Rule Path
Disallow /

slurp

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

googleother

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

sbintuitionsbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ai2bot

Rule Path
Disallow /

amazon q

Rule Path
Disallow /

claude-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

yandexgpt

Rule Path
Disallow /

mistralbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

youbot

Rule Path
Disallow /

neevaai

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://botcity.dev/sitemap.xml
Back to top

Comments

  • Block everything by default
  • --- Allowed Verified Bots ---
Back to top

Warnings

  • 2 invalid lines.
Back to top