Log In Sign Up

roll-of-arms.com
robots.txt

Robots Exclusion Standard data for roll-of-arms.com

Resource Scan

Scan Details

Site Domain roll-of-arms.com
Base Domain roll-of-arms.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-10-05T08:58:47+00:00
Next Scan 2026-01-03T08:58:47+00:00

Last Successful Scan

Scanned2025-05-15T23:37:40+00:00
URL https://roll-of-arms.com/robots.txt
Domain IPs 2602:294:0:b13::110, 38.46.223.205
Response IP 38.46.223.205
Found Yes
Hash abb1047f67f445155cdf8acd5b4389fbcacdb5683afee10efdc529b5b10867e0
SimHash 3247e85a85d0

Groups

*

Rule Path
Allow /w/api.php?action=mobileview&
Allow /w/load.php?
Disallow /w/
Disallow /geoip$
Disallow /rest_v1/

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2.5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20
Back to top

Other Records

Field Value
sitemap https://roll-of-arms.com/sitemap.xml
Back to top

Comments

  • robots.txt for Miraheze
  • Throttle access to certain pages
  • Do not include special pages and other pages where indexing is undesirable if they are likely to be linked to; use noindex instead.
  • That's because Google can still index pages in here without crawling them if the pages are linked to
  • See https://developers.google.com/search/docs/crawling-indexing/robots/intro
  • Block SemrushBot
  • Block AhrefsBot
  • Block Bytespider
  • Block PetalBot
  • Block DotBot
  • Block MegaIndex
  • Block serpstatbot
  • Block Barkrowler
  • Block SeekportBot
  • Keep Crawl-Delay rules at the bottom
  • Bots that don't understand Crawl-Delay might break when encountering it
  • See https://github.com/otwcode/otwarchive/pull/4411#discussion_r1044351129 (English) and https://webtan.impress.co.jp/e/2022/11/04/43611 (Japanese)
  • Throttle MJ12Bot
  • Throttle YandexBot
  • TODO: Crawl-delay is not respected since 2018
  • Throttle BingBot
  • ----------------------------------------------------------
  • Dynamic sitemap url
  • ----------------------------------------------------------
Back to top