blog.rac.me.uk
robots.txt

Robots Exclusion Standard data for blog.rac.me.uk

Resource Scan

Scan Details

Site Domain blog.rac.me.uk
Base Domain rac.me.uk
Scan Status Ok
Last Scan2024-10-04T05:47:49+00:00
Next Scan 2024-10-05T05:47:49+00:00

Last Scan

Scanned2024-10-04T05:47:49+00:00
URL https://blog.rac.me.uk/robots.txt
Domain IPs 2a07:7800:3:f::, 45.8.225.156
Response IP 45.8.225.156
Found Yes
Hash 0c4262e84957b49970afe2ba1d03fce6c6645c641c67791ff0bad6dfc764c20b
SimHash 5a78c2225e8a

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-includes/

dotbot
rogerbot
ahrefsbot
blexbot
magpie-crawler
semrushbot
mj12bot
daum

Rule Path
Disallow /

Comments

  • https://moz.com/help/moz-procedures/crawlers/dotbot
  • https://moz.com/help/moz-procedures/crawlers/rogerbot
  • http://ahrefs.com/robot/
  • http://webmeup-crawler.com/
  • https://www.brandwatch.com/legal/magpie-crawler/
  • https://www.semrush.com/bot/
  • https://mj12bot.com/
  • https://cs.daum.net/faq/15/4118.html?faqId=28966