Log In Sign Up

en.wikifur.com
robots.txt

Robots Exclusion Standard data for en.wikifur.com

Resource Scan

Scan Details

Site Domain en.wikifur.com
Base Domain wikifur.com
Scan Status Ok
Last Scan2024-09-30T13:49:41+00:00
Next Scan 2024-10-30T13:49:41+00:00

Last Scan

Scanned2024-09-30T13:49:41+00:00
URL https://en.wikifur.com/robots.txt
Domain IPs 104.21.85.101, 172.67.204.114, 2606:4700:3031::6815:5565, 2606:4700:3036::ac43:cc72
Response IP 104.21.85.101
Found Yes
Hash 1b38c5d44343039afffa59316036178f4ffcceb31874451be3811942008c7822
SimHash 21561573c813

Groups

archive.org_bot

Rule Path
Disallow /

mediapartners-google*

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

acoonbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-seoab

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /w/index.php

*

Rule Path
Disallow /w/
Disallow /trap/
Disallow /wiki?QF_SID
Disallow /wiki/Sp%C3%A9cial%3A
Disallow /wiki/Speci%C3%A1ln%C3%AD%3A
Disallow /wiki/Especial%3A
Disallow /wiki/Special%3A
Disallow /wiki/Spesial%3A
Disallow /wiki/Spezial%3A
Disallow /wiki/Specjalna%3A
Disallow /wiki/Speciaal%3A
Disallow /wiki/Speciel%3A
Disallow /wiki/Speciale%3A
Disallow /wiki/Istimewa%3A
Disallow /wiki/Toiminnot%3A
Disallow /wiki/%D8%AE%D8%A7%D8%B5%3A
Disallow /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93%3A
Disallow /wiki/Speci%C3%A1lis%3A
Disallow /wiki/%E7%89%B9%E5%88%A5%3A
Disallow /wiki/Speci%C3%A1lis%3A
Disallow /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5%3A
Disallow /wiki/%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F%3A
Disallow /wiki/Eri%3A
Disallow /wiki/%ED%8A%B9%EC%88%98%EA%B8%B0%EB%8A%A5%3A
Disallow /wiki/Speciala%C4%B5o%3A
Disallow /wiki/Natatangi%3A
Disallow /wiki/MediaWiki%3A
Disallow /wiki/*?
Disallow /dbdumps/
Disallow /wiki/User_talk%3A1
Disallow /wiki/User_talk%3A2
Disallow /wiki/User_talk%3A3
Disallow /wiki/User_talk%3A4
Disallow /wiki/User_talk%3A5
Disallow /wiki/User_talk%3A6
Disallow /wiki/User_talk%3A7
Disallow /wiki/User_talk%3A8
Disallow /wiki/User_talk%3A9
Disallow /?author
Allow /wiki/special%3Aallpages
Allow /wiki/Special%3AAllPages
Allow /wiki/%3D%C3%90%C2%A1%C3%90%C2%BB%C3%91~C%C3%90%C2%B6%C3%90%C2%B5%C3%90%C2%B1%C3%90%C2%BD%C3%90%C2%B0%C3%91~O%3AAllPages
Allow /wiki/Especial%3ATodas
Allow /wiki/Speciale%3ATutteLePagine
Allow /wiki/%E7%89%B9%E5%88%A5%3A%E3%83%9A%E3%83%BC%E3%82%B8%E4%B8%80%E8%A6%A7
Allow /wiki/Special%3AAlla_sidor
Allow /wiki/Spezial%3AAlle_Seiten
Allow /wiki/Sp%C3%A9cial%3AToutes_les_pages
Allow /wiki/Speci%C3%A1ln%C3%AD%3AAllpages
Allow /wiki/Specjalna%3AWszystkie_strony
Allow /wiki/Special%3A%E6%89%80%E6%9C%89%E9%A1%B5%E9%9D%A2
Allow /wiki/Speci%C3%A1lis%3AAz_%C3%B6sszes_lap_list%C3%A1ja
Allow /wiki/Speciel%3AAlle_sider
Allow /wiki/Speciaal%3AAllePaginas
Allow /wiki/Eri%3AAllPages
Allow /wiki/%ED%8A%B9%EC%88%98%EA%B8%B0%EB%8A%A5%3A%EB%AA%A8%EB%93%A0%EB%AC%B8%EC%84%9C
Allow /wiki/Speciala%C4%B5o%3A%C4%88iuj_pa%C4%9Doj
Allow /wiki/Natatangi%3ALahat_ng_mga_pahina
Allow /w/images/
Disallow /w/images*/$
Allow /w/skins/
Disallow /w/index.php?
Allow /w/index.php?title=Special%3AAllPages
Allow /w/index.php?title=Especial%3ATodas
Allow /w/index.php?title=Spezial%3AAlle_Seiten
Allow /w/index.php?title=Special%3ANewPages&feed
Allow /w/index.php?title=Special%3ARecentChanges&feed
Allow /w/index.php?title=*variant=
Allow /w/index.php?title=Convention_map_script&action=raw
Allow /w/load.php
Disallow /w/index.php?title=*variant=*&trust
Disallow /w/index.php?title=Special%3AAllPages&from=*
Disallow /wiki/Porcelain_Joe
Back to top

Other Records

Field Value
sitemap https://en.wikifur.com/sitemap-index.xml
Back to top

Comments

  • See en.wikipedia.org/robots.txt for explanations
  • Don't accept * in search
  • Not folowing Disallow: /w/
  • Kept hitting the main page /wiki with QF_SID every few seconds
  • General rules
  • This should be captured above but some bots don't get *?
  • Need this for Google Rich Results
Back to top

Warnings

  • 3 invalid lines.
Back to top