Log In Sign Up

cityseeker.com
robots.txt

Robots Exclusion Standard data for cityseeker.com

Resource Scan

Scan Details

Site Domain cityseeker.com
Base Domain cityseeker.com
Scan Status Ok
Last Scan2024-05-14T03:44:47+00:00
Next Scan 2024-06-13T03:44:47+00:00

Last Scan

Scanned2024-05-14T03:44:47+00:00
URL https://cityseeker.com/robots.txt
Domain IPs 108.157.254.109, 108.157.254.27, 108.157.254.43, 108.157.254.67
Response IP 108.157.254.27
Found Yes
Hash f4a7ff3426ea414859a291236b1beb759ee3f4796771efc5aaa3a06c153f40a9
SimHash e79f5f50c690

Groups

mauibot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

slurp china

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

hopper

Rule Path
Disallow /

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 30

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 30

linguee

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

bidswitchbot

Rule Path
Disallow /

seobility

Rule Path
Disallow /

cocolyzebot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

daum

Rule Path
Disallow /

yandexbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 30

yandexbot/3.0

Rule Path
Allow /

Other Records

Field Value
crawl-delay 30

yandex

Rule Path
Allow /

Other Records

Field Value
crawl-delay 30

neevabot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

awariorssbot
awariosmartbot
awariobot/1.0
awariobot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

applebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

applebot/0.1

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

facebookexternalhit

Rule Path
Allow /

Other Records

Field Value
crawl-delay 30

triplecheckerrobot/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0.2

zoombot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

twitterbot/1.0

Rule Path
Disallow /

triplecheckerrobot/1.0

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

startmebot

Rule Path
Disallow /

slackbot
slack-imgproxy

Rule Path
Disallow /

slackbot-linkexpanding

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /

monsidobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mediumbot-metatagfetcher

Rule Path
Disallow /

keys-so-bot

Rule Path
Disallow /

zapbot

Rule Path
Disallow /

zapbot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

ezoicbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

t3versionsbot

Rule Path
Disallow /

t3versionsbot

Rule Path
Disallow /

wellknownbot

Rule Path
Disallow /

wellknownbot

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

startmebot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

qwantify-prod4463

Rule Path
Disallow /

duckduckbot-https
duckduckbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

adsbot-google-mobile

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

snapchat

Rule Path
Disallow /

cincraw

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

facebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

twitterbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

diffbot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

adidxbot

Rule Path
Disallow /

ezoicbot-nicheiq

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://cityseeker.com/sitemap_v6.xml
Back to top