universalis.fr
robots.txt

Robots Exclusion Standard data for universalis.fr

Resource Scan

Scan Details

Site Domain universalis.fr
Base Domain universalis.fr
Scan Status Ok
Last Scan2024-09-30T11:10:46+00:00
Next Scan 2024-10-07T11:10:46+00:00

Last Scan

Scanned2024-09-30T11:10:46+00:00
URL https://universalis.fr/robots.txt
Redirect https://www.universalis.fr/robots.txt
Redirect Domain www.universalis.fr
Redirect Base universalis.fr
Domain IPs 137.74.25.45
Redirect IPs 137.74.25.45
Response IP 137.74.25.45
Found Yes
Hash 88c04533e28277cabf10323df75ae1e4e48a5ccc868148df7d8a3733b3eae527
SimHash ea57c2804122

Groups

psbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

soni

Rule Path
Disallow /

sogou

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

docomo

Rule Path
Disallow /

snapbot

Rule Path
Disallow /

ssearch_bot

Rule Path
Disallow /

vikspider

Rule Path
Disallow /

vik-robot

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

businessdbbot

Rule Path
Disallow /

happyfunbot

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

cityreview

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

mystbot

Rule Path
Disallow /

vik-robot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

ocelli

Rule Path
Disallow /

yeti

Rule Path
Disallow /

speedy

Rule Path
Disallow /

exabot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

comodospider

Rule Path
Disallow /

ayna

Rule Path
Disallow /

purebot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

discobot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

*

Rule Path
Disallow /identification/
Disallow /typo3/
Disallow /uploads/
Disallow /recherche/
Disallow /mobile/*
Disallow /typo3temp/tx_staticfilecache/
Disallow /1/
Disallow /carte-mentale/
Disallow /datapays/
Disallow /auteurs/e.u./
Disallow /auteurs/e.u./

verity/1.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

Warnings

  • 4 invalid lines.