Log In Sign Up

alphanet.cz
robots.txt

Robots Exclusion Standard data for alphanet.cz

Resource Scan

Scan Details

Site Domain alphanet.cz
Base Domain alphanet.cz
Scan Status Ok
Last Scan2025-10-23T09:18:51+00:00
Next Scan 2025-11-22T09:18:51+00:00

Last Scan

Scanned2025-10-23T09:18:51+00:00
URL http://alphanet.cz/robots.txt
Domain IPs 88.86.113.202
Response IP 88.86.113.202
Found Yes
Hash d9db231127eb5f0441b214b232a53a27e36899459a51a6d9c05c9e29709bb054
SimHash 3392bd51cd50

Groups

*

Rule Path
Disallow /ad/
Disallow /App_Data/
Disallow /App_Themes/
Disallow /aspnet_client/
Disallow /backup/
Disallow /bin/
Disallow /custom/
Disallow /data/
Disallow /feedback/
Disallow /headerImages/
Disallow /img/
Disallow /images/
Disallow /images-cache/
Disallow /images-cache-nonreg/
Disallow /javascript/
Disallow /log/
Disallow /pictures/
Disallow /services/
Disallow /portal.Master
Disallow /Global.asax

adsbot-google

Rule Path
Disallow

mediapartners-google

Rule Path
Disallow

googlebot-image

Rule Path
Disallow /

tineye

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

digout4u
extractorpro
getright
go-ahead-got-it
grub
httpclient
linkwalker
nearsite
netattache
newt activex
sitesnagger
teleport
tovektools web indexer
ubicrawler
web downloader
webtrends
webwhacker
webzip
ltx71
ahrefsbot
blexbot
dotbot
petalbot
barkrowler
amazonbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

youbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /
Back to top

Comments

  • # # # # # # # # # # #
  • ADS and IMAGES bots
  • # # # # # # # # # # #
  • # # # # # # # # # # #
  • Default robots.txt file
  • Last updated: 12.10.2020
  • Generated by: Bloghosts + MartyR
  • The below file is in place by default to block bad robots
  • from indexing your site while allowing good robots to browse
  • freely. Remove the below entries at your own risk.
  • 2020-04-30 added AhrefsBot by Martin
  • Open Link Profiler
  • Mozilla/5.0+(compatible;+spbot/4.4.2;++http://OpenLinkProfiler.org/bot+)
  • http://OpenLinkProfiler.org/bot
  • http://mj12bot.com - MAJESTIC
  • Crawl-Delay should be an integer number and it signifies number of seconds of wait between requests.
  • Crawl-Delay: 5
  • bingbot
  • Crawl-delay: 5
  • blocks access to whole site for All Yandex bots like YandexBot, YandexDirect, YandexDirectDyn, YandexMedia, YandexImages ....
  • block SEMrushBot
  • To block SEMrushBot from crawling your site for a webgraph of links:
  • To block SEMrushBot from crawling your site for different SEO and technical issues:
  • To block SEMrushBot from crawling your site for Backlink Audit tool:
  • To block SEMrushBot from crawling your site for On Page SEO Checker tool and similar tools:
  • To block SEMrushBot from checking URLs your site for SWA tool:
  • To block SEMrushBot from crawling your site for Content Analyzer and Post Tracking tools:
  • To block SEMrushBot from crawling your site for Brand Monitoring:
  • block Amazonbot
  • To block Amazonbot from crawling our site - Martin 23.1.2024
  • AmazonBot does not support the crawl-delay directive in robots.txt and robots meta tags on HTML pages such as “nofollow” and "noindex".
  • Here are AI bots I’m blocking using robots.txt
Back to top

Warnings

  • 1 invalid line.
Back to top