univ-rennes1.fr
robots.txt

Robots Exclusion Standard data for univ-rennes1.fr

Resource Scan

Scan Details

Site Domain univ-rennes1.fr
Base Domain univ-rennes1.fr
Scan Status Ok
Last Scan2024-10-27T16:20:40+00:00
Next Scan 2024-11-26T16:20:40+00:00

Last Scan

Scanned2024-10-27T16:20:40+00:00
URL https://www.univ-rennes1.fr/robots.txt
Redirect https://www.univ-rennes.fr/robots.txt
Redirect Domain www.univ-rennes.fr
Redirect Base univ-rennes.fr
Domain IPs 129.20.126.134
Redirect IPs 129.20.126.139
Response IP 129.20.126.139
Found Yes
Hash e41296187588bfd01c6d2c76ae5b6451e4a0819998172522e319a768c08ea054
SimHash 2c14e00266e1

Groups

adsbot-google
adsbot-google-mobile
adsbot-google-mobile-apps
adidxbot
applebot
applenewsbot
bingbot
bingpreview
bublupbot
ccbot
cliqzbot
dazoobot
deusu
duckduckbot
duckduckgo-favicons-bot
euripbot
exploratodo
facebot
feedly
findxbot
gooblog
googlebot
googlebot-image
googlebot-mobile
googlebot-news
googlebot-video
mediapartners-google
orangebot
pinterest
qwantify
slurp
teoma
twitterbot
wotbox
yoozbot
youdaobot

Rule Path
Allow /themes/*/favicon.ico
Disallow /core/
Disallow /libraries/
Disallow /modules/
Disallow /profiles/
Disallow /themes/
Disallow /vendor/
Disallow /.csslintrc
Disallow /.eslintignore
Disallow /.eslintrc.json
Disallow /INSTALL.txt
Disallow /README.md
Disallow /README.txt
Disallow /web.config
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips
Disallow /node/add/
Disallow /search/
Disallow /user/register
Disallow /user/password
Disallow /user/login
Disallow /user/logout
Disallow /media/oembed
Disallow /*/media/oembed
Disallow /index.php/admin/
Disallow /index.php/comment/reply/
Disallow /index.php/filter/tips
Disallow /index.php/node/add/
Disallow /index.php/search/
Disallow /index.php/user/password
Disallow /index.php/user/register
Disallow /index.php/user/login
Disallow /index.php/user/logout
Disallow /index.php/media/oembed
Disallow /index.php/*/media/oembed
Disallow /fr/user/register
Disallow /fr/user/password
Disallow /fr/user/login
Disallow /fr/user/logout
Disallow /en/user/register
Disallow /en/user/password
Disallow /en/user/login
Disallow /en/user/logout
Disallow /index.php/fr/user/password
Disallow /index.php/fr/user/register
Disallow /index.php/fr/user/login
Disallow /index.php/fr/user/logout
Disallow /index.php/en/user/password
Disallow /index.php/en/user/register
Disallow /index.php/en/user/login
Disallow /index.php/en/user/logout
Disallow /interlocuteurs?*
Disallow /ressources?*
Disallow /seminaires?*
Disallow /interlocuteurs
Disallow /ressources
Disallow /seminaires
Disallow /interlocutors?*
Disallow /resources?*
Disallow /seminars?*
Disallow /interlocutors
Disallow /resources
Disallow /seminars
Disallow /en/interlocutors?*
Disallow /en/resources?*
Disallow /en/seminars?*
Disallow /en/interlocutors
Disallow /en/resources
Disallow /en/seminars
Disallow /fr/interlocuteurs?*
Disallow /fr/ressources?*
Disallow /fr/seminaires?*
Disallow /fr/interlocuteurs
Disallow /fr/ressources
Disallow /fr/seminaires
Disallow /index.php/interlocuteurs?*
Disallow /index.php/ressources?*
Disallow /index.php/seminaires?*
Disallow /index.php/interlocuteurs
Disallow /index.php/ressources
Disallow /index.php/seminaires
Disallow /index.php/interlocutors?*
Disallow /index.php/resources?*
Disallow /index.php/seminars?*
Disallow /index.php/interlocutors
Disallow /index.php/resources
Disallow /index.php/seminars
Disallow /index.php/en/interlocutors?*
Disallow /index.php/en/resources?*
Disallow /index.php/en/seminars?*
Disallow /index.php/en/interlocutors
Disallow /index.php/en/resources
Disallow /index.php/en/seminars
Disallow /index.php/fr/interlocuteurs?*
Disallow /index.php/fr/ressources?*
Disallow /index.php/fr/seminaires?*
Disallow /index.php/fr/interlocuteurs
Disallow /index.php/fr/ressources
Disallow /index.php/fr/seminaires
Disallow /*.css$
Disallow /*?
Disallow /*?f%5B*
Disallow /*%26f%5B*
Disallow /*?f%5B0*
Disallow /*%26f%5B1*
Disallow

*
bytespider
chatgpt-user
claudebot
dotbot
gptbot
imagesiftbot
mail.ru
mail.ru_bot
semrushbot
yandexbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10