Log In Sign Up

bernardaud.com
robots.txt

Robots Exclusion Standard data for bernardaud.com

Resource Scan

Scan Details

Site Domain bernardaud.com
Base Domain bernardaud.com
Scan Status Ok
Last Scan2024-09-12T03:10:06+00:00
Next Scan 2024-10-12T03:10:06+00:00

Last Scan

Scanned2024-09-12T03:10:06+00:00
URL https://www.bernardaud.com/robots.txt
Domain IPs 52.212.52.84, 54.247.69.169, 63.32.161.232
Response IP 52.212.52.84
Found Yes
Hash e73eb5ae63415bc59e4450e6115ee10d535d6dc863753ec355fa8d2c1117788c
SimHash 229c0c9dfa40

Groups

*

Rule Path
Disallow /admin
Disallow /admin_users/sign_in
Disallow /fr/declaration-de-conformite
Disallow /en-gb/compliance-with-regulations
Disallow /en/compliance-with-regulations
Disallow /api/
Disallow /pro/

mj12bot

Rule Path
Disallow /

mj12bot/v1.4.5

Rule Path
Disallow /

mozilla/5.0 (compatible; mj12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mozilla/5.0 (compatible; ahrefsbot/5.0; +http://ahrefs.com/robot/)

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mozilla/5.0 (compatible; semrushbot-si/0.97; +http://www.semrush.com/bot.html)

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://bernardaud-prod.s3.eu-west-3.amazonaws.com/system/sitemap.xml.gz
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • Crawl-delay: 10
  • MJ12Bot
  • AhrefsBot
  • SemrushBot
  • Sitemap: https://www.bernardaud.com/sitemap.xml
Back to top