Log In Sign Up

auren.com
robots.txt

Robots Exclusion Standard data for auren.com

Resource Scan

Scan Details

Site Domain auren.com
Base Domain auren.com
Scan Status Ok
Last Scan2025-07-10T19:05:44+00:00
Next Scan 2025-08-09T19:05:44+00:00

Last Scan

Scanned2025-07-10T19:05:44+00:00
URL https://auren.com/robots.txt
Domain IPs 104.21.53.207, 172.67.218.180, 2606:4700:3031::6815:35cf, 2606:4700:3032::ac43:dab4
Response IP 172.67.218.180
Found Yes
Hash 6f4ce0bd39e2cc675574c7aae9bd092aff875fbbf8a5b021decf84afd8697d1b
SimHash 3a425543ede0

Groups

bytespider

Rule Path
Disallow /

bytedance

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

geedobot

Rule Path
Disallow /

netestate ne crawler (+http://www.website-datenbank.de/)

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

robozilla

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30
Back to top

Comments

  • To block Bytespider from crawling:
  • To block Bytedance from crawling:
  • To block Coccocbot from crawling:
  • To block Dotbot from crawling:
  • To block Common Crawl Bot from crawling:
  • To block Common Crawl Bot from crawling:
  • To block GeedoBot from crawling:
  • Block netEstate NE Crawler (+http://www.website-datenbank.de/)
  • To block PetalBot from crawling:
  • To block AhrefsBot from crawling:
  • Blocked SemrushBot since it is creating lot of requests
  • To block SemrushBot from crawling your site for different SEO and technical issues:
  • To block SemrushBot from crawling your site for Backlink Audit tool:
  • To block SemrushBot from crawling your site for On Page SEO Checker tool and similar tools:
  • To block SemrushBot from checking URLs on your site for SWA tool:
  • To block SemrushBot from crawling your site for Content Analyzer and Post Tracking tools:
  • To block SplitSignalBot from crawling your site for SplitSignal tool:
  • To block SemrushBot-COUB from crawling your site for Content Outline Builder tool:
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Block Etaospider
Back to top