Log In Sign Up

cityof.com
robots.txt

Robots Exclusion Standard data for cityof.com

Resource Scan

Scan Details

Site Domain cityof.com
Base Domain cityof.com
Scan Status Ok
Last Scan2024-10-19T01:07:44+00:00
Next Scan 2024-11-18T01:07:44+00:00

Last Scan

Scanned2024-10-19T01:07:44+00:00
URL https://cityof.com/robots.txt
Redirect https://www.cityof.com/robots.txt
Redirect Domain www.cityof.com
Redirect Base cityof.com
Domain IPs 138.197.52.62
Redirect IPs 138.197.52.62
Response IP 138.197.52.62
Found Yes
Hash 284b314669dba354a8332cb9e7b24ca2a4e4d00031f5e185127ac9f629dd6a65
SimHash f0781810c4cb

Groups

facebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

facebookexternalhit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

meta-externalfetcher

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

meta-externalagent

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

amazonbot

Rule Path
Disallow /

import.io

Rule Path
Disallow /

webhose.io

Rule Path
Disallow /

dexi.io

Rule Path
Disallow /

zyte

Rule Path
Disallow /

outwit hub

Rule Path
Disallow /

getleft

Rule Path
Disallow /

httrack

Rule Path
Disallow /

cyotek webcopy

Rule Path
Disallow /

helium scraper

Rule Path
Disallow /

sequentum

Rule Path
Disallow /

webharvy

Rule Path
Disallow /

visual scraper

Rule Path
Disallow /

parsehub

Rule Path
Disallow /

octoparse

Rule Path
Disallow /

oncrawl

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

sitebulb crawler

Rule Path
Disallow /

botify

Rule Path
Disallow /

jetoctopus

Rule Path
Disallow /

netpeak spider

Rule Path
Disallow /

contentking

Rule Path
Disallow /

botify

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

lumar

Rule Path
Disallow /

deepcrawl

Rule Path
Disallow /

cognitiveseo site explorer

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

awariobot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

deepcrawl

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

webzio-extended

Rule Path
Disallow /

youbot

Rule Path
Disallow /

*

Rule Path
Disallow /search
Disallow /get-listed/form
Disallow /admin
Disallow /internal
Disallow /checkout
Disallow /exclusive-offer
Disallow /get-listed/send-edit-link
Back to top

Comments

  • robots.txt for cityof.com
  • Data scraper bots
  • SEO company bots
  • AI model bots
Back to top

Warnings

  • 2 invalid lines.
Back to top