wiki.melvoridle.com
robots.txt

Robots Exclusion Standard data for wiki.melvoridle.com

Resource Scan

Scan Details

Site Domain wiki.melvoridle.com
Base Domain melvoridle.com
Scan Status Ok
Last Scan2025-09-29T11:07:03+00:00
Next Scan 2025-10-29T11:07:03+00:00

Last Scan

Scanned2025-09-29T11:07:03+00:00
URL https://wiki.melvoridle.com/robots.txt
Domain IPs 104.26.6.193, 104.26.7.193, 172.67.69.234, 2606:4700:20::681a:6c1, 2606:4700:20::681a:7c1, 2606:4700:20::ac43:45ea
Response IP 104.26.7.193
Found Yes
Hash fcf47ad5c6e7c8e0d6a0e028b2ec5bcf1f722785f8feee4134b61be2b524d41c
SimHash 31f6ad52ec65

Groups

*

Rule Path
Disallow /w/Property%3A
Disallow /w/Property%3A
Disallow /w/property%3A
Disallow /*?title=Property%3A
Disallow /*?title=Property%3A
Disallow /*?*&title=Property%3A
Disallow /*?*&title=Property%3A
Disallow /w/Special%3A
Disallow /w/Special%3A
Disallow /w/special%3A
Disallow /*?title=Special%3A
Disallow /*?title=Special%3A
Disallow /*?*&title=Special%3A
Disallow /*?*&title=Special%3A
Disallow /api.php
Disallow /cors/
Disallow /geoip$
Disallow /rest_v1/
Disallow /tags/
Disallow /*?action=
Disallow /*?*&action=
Disallow /*?feed=
Disallow /*?*&feed=
Disallow /*?from=
Disallow /*?*&from=
Disallow /*?mobileaction=
Disallow /*?*&mobileaction=
Disallow /*?oldid=
Disallow /*?*&oldid=
Disallow /*?printable=
Disallow /*?*&printable=
Disallow /*?redirect=
Disallow /*?*&redirect=
Disallow /*?uselang=
Disallow /*?*&uselang=
Disallow /*?useskin=
Disallow /*?*&useskin=
Disallow /*?variant=
Disallow /*?*&variant=
Disallow /*?veaction=
Disallow /*?*&veaction=
Disallow /*?filefrom=
Disallow /*?*&filefrom=
Disallow /*?fileuntil=
Disallow /*?*&fileuntil=
Disallow /*?navbox=
Disallow /*?*&navbox=
Disallow /*?pageuntil=
Disallow /*?*&pageuntil=
Disallow /*?pagefrom=
Disallow /*?*&pagefrom=
Disallow /*?diff=
Disallow /*?*&diff=
Disallow /*?curid=
Disallow /*?*&curid=
Disallow /*?search=
Disallow /*?*&search=
Disallow /*?section=
Disallow /*?*&section=

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

neevabot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

seekport

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

Comments

  • SEO bots
  • Huawei's web crawler. Ignores Disallow and gets caught in loops accessing special pages. Produces the majority of uncachable requests.
  • AI bots