Log In Sign Up

cidadeurgente.com
robots.txt

Robots Exclusion Standard data for cidadeurgente.com

Resource Scan

Scan Details

Site Domain cidadeurgente.com
Base Domain cidadeurgente.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-04-07T13:38:18+00:00
Next Scan 2024-07-06T13:38:18+00:00

Last Successful Scan

Scanned2023-06-13T13:36:44+00:00
URL https://cidadeurgente.com/robots.txt
Domain IPs 18.161.97.116, 18.161.97.14, 18.161.97.42, 18.161.97.78
Response IP 52.222.214.61
Found Yes
Hash 7c77c242b1f812f6430fd7251f04c11b7fbfd643354863ec439d85bd73d9a7f8
SimHash 43d4107053be

Groups

accompanybot

Rule Path
Disallow /

admantx

Rule Path
Disallow /

ahrefs

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

archivebot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

baidu

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bomborabot

Rule Path
Disallow /

bot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

cognitiveseo

Rule Path
Disallow /

curebot

Rule Path
Disallow /

daum

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

detectify

Rule Path
Disallow /

deusu

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ecairn-grabber

Rule Path
Disallow /

exabot

Rule Path
Disallow /

eyeotabot

Rule Path
Disallow /

fatbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

ias_crawler

Rule Path
Disallow /

ias-va

Rule Path
Disallow /

ias-va/3.1

Rule Path
Disallow /

kinza

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

linguee

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

lmy47v

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mb2345browser

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

micromessenger

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mqqbrowser

Rule Path
Disallow /

newspaper/0.2.8

Rule Path
Disallow /

pagefreezer

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

psbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

roboto

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

screaming

Rule Path
Disallow /

screaming.frog

Rule Path
Disallow /

search365bot

Rule Path
Disallow /

searchblox

Rule Path
Disallow /

searchnz

Rule Path
Disallow /

securityresearch.bot

Rule Path
Disallow /

seekport

Rule Path
Disallow /

semanticscholarbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

siteimprove.com

Rule Path
Disallow /

slurp

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

sogou.web.spider

Rule Path
Disallow /

spbot

Rule Path
Disallow /

sqlmap

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ucbrowser

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

vebidoobot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximageresizer

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

yoozbot-2.2

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingpreview

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

adsbot-google-mobile-apps

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

adsbot-google-mobile

Rule Path
Allow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20
Back to top

Other Records

Field Value
sitemap https://cidadeurgente.com/sitemap.xml
Back to top

Comments

  • DENIED BOTS
  • REDUCED SPEED BOTS
  • FAST BOTS
  • OTHER BOTS
Back to top

Warnings

  • 4 invalid lines.
Back to top