Log In Sign Up

bernardaud.fr
robots.txt

Robots Exclusion Standard data for bernardaud.fr

Resource Scan

Scan Details

Site Domain bernardaud.fr
Base Domain bernardaud.fr
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-07-13T17:30:44+00:00
Next Scan 2024-10-11T17:30:44+00:00

Last Successful Scan

Scanned2023-05-28T16:22:25+00:00
URL https://www.bernardaud.fr/robots.txt
Domain IPs 52.212.52.84, 54.247.69.169, 63.32.161.232
Response IP 63.32.161.232
Found Yes
Hash 382e81b45593cf75e61f7381add3785d9e99b3950d9ccc22dffdf9ca53727886
SimHash a28c0c8dfa40

Groups

*

Rule Path
Disallow /admin
Disallow /admin_users/sign_in
Disallow /fr/declaration-de-conformite
Disallow /en-gb/compliance-with-regulations
Disallow /en/compliance-with-regulations

mj12bot

Rule Path
Disallow /

mj12bot/v1.4.5

Rule Path
Disallow /

mozilla/5.0 (compatible; mj12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mozilla/5.0 (compatible; ahrefsbot/5.0; +http://ahrefs.com/robot/)

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mozilla/5.0 (compatible; semrushbot-si/0.97; +http://www.semrush.com/bot.html)

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://bernardaud-prod.s3.eu-west-3.amazonaws.com/system/sitemap.xml.gz
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • Crawl-delay: 10
  • MJ12Bot
  • AhrefsBot
  • SemrushBot
  • Sitemap: https://www.bernardaud.com/sitemap.xml
Back to top