Log In Sign Up

diariocidade.com
robots.txt

Robots Exclusion Standard data for diariocidade.com

Resource Scan

Scan Details

Site Domain diariocidade.com
Base Domain diariocidade.com
Scan Status Ok
Last Scan2024-06-05T18:29:34+00:00
Next Scan 2024-06-12T18:29:34+00:00

Last Scan

Scanned2024-06-05T18:29:34+00:00
URL https://diariocidade.com/robots.txt
Redirect https://www.diariocidade.com/robots.txt
Redirect Domain www.diariocidade.com
Redirect Base diariocidade.com
Domain IPs 18.161.6.72, 18.161.6.76, 18.161.6.81, 18.161.6.84
Redirect IPs 204.246.191.124, 204.246.191.59, 204.246.191.75, 204.246.191.82, 2600:9000:2024:3200:1e:b6fb:4740:93a1, 2600:9000:2024:4800:1e:b6fb:4740:93a1, 2600:9000:2024:9c00:1e:b6fb:4740:93a1, 2600:9000:2024:b000:1e:b6fb:4740:93a1, 2600:9000:2024:ce00:1e:b6fb:4740:93a1, 2600:9000:2024:ec00:1e:b6fb:4740:93a1, 2600:9000:2024:f400:1e:b6fb:4740:93a1, 2600:9000:2024:f800:1e:b6fb:4740:93a1
Response IP 3.160.246.48
Found Yes
Hash f2dcc2ce68619732a2de77fcb67a5b1c86a39541639d51435f3a05ac633db304
SimHash 61d4a07056aa

Groups

grapeshot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

ahrefs

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

archivebot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

baidu

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bomborabot

Rule Path
Disallow /

bot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

cognitiveseo

Rule Path
Disallow /

curebot

Rule Path
Disallow /

daum

Rule Path
Disallow /

detectify

Rule Path
Disallow /

deusu

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ecairn-grabber

Rule Path
Disallow /

exabot

Rule Path
Disallow /

eyeotabot

Rule Path
Disallow /

fatbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

kinza

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

linguee

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

lmy47v

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mb2345browser

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

micromessenger

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mqqbrowser

Rule Path
Disallow /

pagefreezer

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

roboto

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

screaming

Rule Path
Disallow /

screaming.frog

Rule Path
Disallow /

search365bot

Rule Path
Disallow /

searchblox

Rule Path
Disallow /

searchnz

Rule Path
Disallow /

securityresearch.bot

Rule Path
Disallow /

seekport

Rule Path
Disallow /

semanticscholarbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

siteimprove.com

Rule Path
Disallow /

slurp

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

sogou.web.spider

Rule Path
Disallow /

spbot

Rule Path
Disallow /

sqlmap

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ucbrowser

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

vebidoobot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximageresizer

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

yoozbot-2.2

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingpreview

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

googlebot-mobile

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

adsbot-google

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

adsbot-google-mobile

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

googlebot-image

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

googlebot-video

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

archive.org_bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

googlebot
*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20
Back to top

Other Records

Field Value
sitemap https://www.diariocidade.com/sitemap.xml
Back to top

Comments

  • Oracle Crawler - Premium Ads
  • DENIED BOTS
  • REDUCED SPEED BOTS
  • FAST BOTS
  • OTHER BOTS
Back to top

Warnings

  • 4 invalid lines.
Back to top