citywire.com
robots.txt

Robots Exclusion Standard data for citywire.com

Resource Scan

Scan Details

Site Domain citywire.com
Base Domain citywire.com
Scan Status Ok
Last Scan2024-06-04T05:07:49+00:00
Next Scan 2024-06-18T05:07:49+00:00

Last Scan

Scanned2024-06-04T05:07:49+00:00
URL https://citywire.com/robots.txt
Domain IPs 45.60.242.95, 45.60.248.95
Response IP 45.60.242.95
Found Yes
Hash b42fea0cc15ee5f92e566d2c4ea036cb336e88b6f39c7e03ef66c4313009510f
SimHash 08524341f014

Groups

semrushbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

moodlebot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

https://hada.news

Rule Path
Disallow /

https://www.imediaethics.org

Rule Path
Disallow /

mojeek

Rule Path
Disallow /

jenkersbot

Rule Path
Disallow /

seekr

Rule Path
Disallow /

turnitin

Rule Path
Disallow /

youbot

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

yacy

Rule Path
Disallow /

yandex

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

*

Rule Path
Disallow /search$
Disallow /*/search$
Disallow /search?
Disallow /*/search?
Disallow /sign-in$
Disallow /*/sign-in$
Disallow /sign-in?
Disallow /*/sign-in?
Disallow /register$
Disallow /*/register$
Disallow /register?
Disallow /*/register?
Disallow /GetCommentsCount
Disallow /*/GetCommentsCount
Disallow /GetCommentsCount?
Disallow /*/GetCommentsCount?