Log In Sign Up

cyburbia.org
robots.txt

Robots Exclusion Standard data for cyburbia.org

Resource Scan

Scan Details

Site Domain cyburbia.org
Base Domain cyburbia.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-12T02:24:52+00:00
Next Scan 2024-12-11T02:24:52+00:00

Last Successful Scan

Scanned2022-11-16T05:27:42+00:00
URL https://cyburbia.org/robots.txt
Response IP 50.28.61.58
Found Yes
Hash 856a99a7c9ba43a7c524ab8616fdb9ab5242ac5a99755f90ab02c8c4c38deba3
SimHash da1665570cf8

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /forums/account/
Disallow /forums/admin.php/
Disallow /forums/conversations/
Disallow /forums/goto/
Disallow /forums/find-new/
Disallow /forums/find-threads/
Disallow /forums/login/
Disallow /forums/misc/
Disallow /forums/members/
Disallow /forums/online/
Disallow /forums/profile-posts/
Disallow /forums/posts/*/reactions
Disallow /forums/register/
Disallow /forums/search/
Disallow /forums/threads/*/reply*

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

a6-indexer
ahrefsbot
alphaseobot
alphaseobot-sa
awariosmartbot
baiduspider
baiduspi+
baiduspider-image
baiduspider-render
baiduspider-video
barkrowler
blackboard safeassign
blexbot
boardreader
bpimagewalker
bubing
buck
businessbot
bytespider
ccbot
clickagy
cloudfind
coccoc
coccocbot
crawler4j
dataforseobot
daum
daumoa
discobot
discoverybot
dotbot
exabot
fast enterprise
fast*
fastcrawler
garlick
gigabot
gluten free crawler
grapeshotcrawler
html validator by siteimprove.com
httpurlconnection
ichiro
jamesbot
jersey
jigsaw
jikespider
jikespidero
kraken
liebaofast
linkdex
linkwalker
linkxheck
lipperhey
ltx71
magpie-crawler
mail.ru
mail.ru_bot
manzama
mauibot
megaindex
megaindex.ru
mggbrowser
mj12bot
mlbot
mojeekbot
moodlebot
moreover
netseer
netseer
nimbostratus-bot
nutch
obot
onalyticabot
panscient.com
petalbot
plukkie
prlog
rankurbot
riddler
robots
rogerbot
scoutjet
scrapy
screaming frog seo spider
screenerbot
seekport crawler
semanticscholarbot
semrushbot
semrushbot-ba
semrushbot-sa
seokicks
seznambot
sitecheck-sitecrawl
siteimprove
siteimprove (accessibility)
smtbot
sogou
sogou blog
sogou inst spider
sogou orion spider
sogou spider
sogou spider2
sogou web spider
soso
sosospider
spbot
sputnikbot
the knowledge ai
turnitinbot
velenpublicwebcrawler
vegebot
yacybot
yandexblogs
yandexbot
yandexcatalog
yandeximages
yandexmedia
yeti
yisou
yodaobot
youdaobot
zoominfobot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.cyburbia.org/forums/sitemap.xml
Back to top

Comments

  • To keep Cyburbia running smoothly, we restrict bots and spiders that
  • refer few or no visitors to the site, yet still consume a lot of our
  • bandwidth and CPU load. It's nothing personal. We just want to have
  • our limited bandwidth used efficiently, where it will benefit our
  • community the most.
  • ____
  • / @ \==]|[=(] E X T E R M I N A T E ! !
  • |--------|
  • ========== . * *
  • ========== .\ * . *. * . * \ .
  • |||||||||||| \ * ./ * . * . \ \
  • |||||||||| --]%%%%%% .- =--=---=-=-=-=--=-=--=-==-----=-=-=-=-=-=
  • [=========\ ]===========( * . / /
  • [==============| / * \ . * * / .
  • C| @ @ @ @ @ @ D * * *
  • | \ . * *
  • C| @ @ @ @ @ @ D .
  • | \ * *
  • C| @ @ @ @ @ @ D
  • | \
  • C| @ @ @ @ @ @ D
  • | \
  • |@@@@@@@@@@@@@@@@@@@@@@@@@|
  • -------------------------
  • cyburbia.org robots.txt
  • Good bots (we assume)
  • Crawl-delay: 5
  • Disallow paths that are redundant, dead, or have little or no content
  • Disallow: /directory/
  • Disallow: /forums/gallery/
  • Annoying but still somewhat useful bots
  • Unwanted bots
Back to top

Warnings

  • 2 invalid lines.
Back to top