Log In Sign Up

suissebook.ch
robots.txt

Robots Exclusion Standard data for suissebook.ch

Resource Scan

Scan Details

Site Domain suissebook.ch
Base Domain suissebook.ch
Scan Status Ok
Last Scan2024-09-21T18:17:00+00:00
Next Scan 2024-09-28T18:17:00+00:00

Last Scan

Scanned2024-09-21T18:17:00+00:00
URL https://suissebook.ch/robots.txt
Domain IPs 104.21.75.207, 172.67.181.252
Response IP 172.67.181.252
Found Yes
Hash 606cba9c23635ef2a45a55dd1c2307d8412eff256f4ae3955191a110e8443d76
SimHash 6616f3024bee

Groups

*

Rule Path
Disallow /wp-admin/*
Disallow /*/feed/

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou blog

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

sogou news spider

Rule Path
Disallow /

sogou orion spider

Rule Path
Disallow /

sogou spider2

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

baiduspider
baiduspider+
baiduspider-video
baiduspider-image

Rule Path
Disallow /

yandex

Rule Path
Disallow /

bubing

Rule Path
Disallow /

msnbot
slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

seznambot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

adscanner

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

eurobot
twiceler
asterias
backdoorbot/1.0
black hole
blowfish/1.0
botalot
builtbottough
bullseye/1.0
bunnyslippers
cegbfeieh
cheesebot
cherrypicker
copyrightcheck
cosmos
crescent
dittospyder
emailcollector
emailsiphon
emailwolf
erocrawler
extractorpro
foobot
harvest/1.5
hloader
humanlinks
infonavirobot
jennybot
kenjin spider
keyword density/0.9
lexibot
libweb/clshttp
linkextractorpro
linkscan/8.1a unix
linkwalker
lnspiderguy
lwp-trivial
mata hari
miixpc
mister pix
moget
netants
nicerspro
offline explorer
openfind
openfind data gathere
propowerbot/2.14
prowebwalker
queryn metasearch
repomonkey
rma
sitesnagger
spankbot
spanner
suzuran
szukacz/1.4
teleport
teleportpro
telesoft
the intraformant
thenomad
tighttwatbot
titan
tocrawl/urldispatcher
true_robot
turingos
urly warning
vci
web image collector
webauto
webbandit
webbandit/3.50
webcopier
webenhancer
webmasterworldforumbot
websauger
website quester
webster pro
webstripper
www-collector-e
xenu's
zeus

Rule Path
Disallow /

msnbot
slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60
Back to top

Other Records

Field Value
sitemap https://v1.suissebook.ch/sitemap.xml
Back to top

Comments

  • For all robots
  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • dotbot without version
  • Block Gigabot
  • SoGou (CN) Info: http://www.sogou.com/docs/help/webmasters.htm#07
  • https://ahrefs.com/robot
  • http://law.di.unimi.it/BUbiNG.html
  • Crawl delay for bingbot and slurp. Google will ignor this.
  • Disallow mnore crap bots
  • Crawl delay for bingbot and slurp. Google will ignor this.
Back to top