Log In Sign Up

universoautomotivo.com
robots.txt

Robots Exclusion Standard data for universoautomotivo.com

Resource Scan

Scan Details

Site Domain universoautomotivo.com
Base Domain universoautomotivo.com
Scan Status Ok
Last Scan2024-09-24T11:11:58+00:00
Next Scan 2024-10-01T11:11:58+00:00

Last Scan

Scanned2024-09-24T11:11:58+00:00
URL https://universoautomotivo.com/robots.txt
Redirect https://www.universoautomotivo.com/robots.txt
Redirect Domain www.universoautomotivo.com
Redirect Base universoautomotivo.com
Domain IPs 3.165.190.100, 3.165.190.17, 3.165.190.27, 3.165.190.58
Redirect IPs 2600:9000:2792:1200:16:6c03:82c0:93a1, 2600:9000:2792:1800:16:6c03:82c0:93a1, 2600:9000:2792:4200:16:6c03:82c0:93a1, 2600:9000:2792:8e00:16:6c03:82c0:93a1, 2600:9000:2792:b600:16:6c03:82c0:93a1, 2600:9000:2792:d400:16:6c03:82c0:93a1, 2600:9000:2792:d800:16:6c03:82c0:93a1, 2600:9000:2792:ea00:16:6c03:82c0:93a1, 3.164.182.121, 3.164.182.127, 3.164.182.43, 3.164.182.66
Response IP 65.9.112.41
Found Yes
Hash 514a7774f1da822446450de778df7f44404230dfc5e5b98515458cc542288757
SimHash 4bd4117057be

Groups

accompanybot

Rule Path
Disallow /

admantx

Rule Path
Disallow /

ahrefs

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

archivebot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

baidu

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bomborabot

Rule Path
Disallow /

bot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

cognitiveseo

Rule Path
Disallow /

curebot

Rule Path
Disallow /

daum

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

detectify

Rule Path
Disallow /

deusu

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ecairn-grabber

Rule Path
Disallow /

exabot

Rule Path
Disallow /

eyeotabot

Rule Path
Disallow /

fatbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

ias_crawler

Rule Path
Disallow /

kinza

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

linguee

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

lmy47v

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mb2345browser

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

micromessenger

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mqqbrowser

Rule Path
Disallow /

newspaper/0.2.8

Rule Path
Disallow /

pagefreezer

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

psbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

roboto

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

screaming

Rule Path
Disallow /

screaming.frog

Rule Path
Disallow /

search365bot

Rule Path
Disallow /

searchblox

Rule Path
Disallow /

searchnz

Rule Path
Disallow /

securityresearch.bot

Rule Path
Disallow /

seekport

Rule Path
Disallow /

semanticscholarbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

siteimprove.com

Rule Path
Disallow /

slurp

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

sogou.web.spider

Rule Path
Disallow /

spbot

Rule Path
Disallow /

sqlmap

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ucbrowser

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

vebidoobot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximageresizer

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

yoozbot-2.2

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingpreview

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

adsbot-google-mobile-apps

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

adsbot-google-mobile

Rule Path
Allow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20
Back to top

Other Records

Field Value
sitemap https://www.universoautomotivo.com/sitemap.xml
Back to top

Comments

  • DENIED BOTS
  • REDUCED SPEED BOTS
  • FAST BOTS
  • OTHER BOTS
Back to top

Warnings

  • 4 invalid lines.
Back to top