Log In Sign Up

hamisiburada.az
robots.txt

Robots Exclusion Standard data for hamisiburada.az

Resource Scan

Scan Details

Site Domain hamisiburada.az
Base Domain hamisiburada.az
Scan Status Ok
Last Scan2025-11-23T00:01:34+00:00
Next Scan 2025-12-23T00:01:34+00:00

Last Scan

Scanned2025-11-23T00:01:34+00:00
URL https://hamisiburada.az/robots.txt
Domain IPs 104.21.26.42, 172.67.168.56, 2606:4700:3032::ac43:a838, 2606:4700:3035::6815:1a2a
Response IP 172.67.168.56
Found Yes
Hash e342fd5545635ef915c84471b0747a0d1105b383258ec69d62b232503db478f4
SimHash 59176d8575d8

Groups

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

bingbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

duckduckbot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-bot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

linkedinbot

Rule Path
Disallow /

whatsapp

Rule Path
Disallow /

telegrambot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

wayback

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

netcraftsurveyagent

Rule Path
Disallow /

censysinspect

Rule Path
Disallow /

shodanbot

Rule Path
Disallow /

crawler

Rule Path
Disallow /

spider

Rule Path
Disallow /

bot

Rule Path
Disallow /

scraper

Rule Path
Disallow /

*

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 86400
Back to top

Other Records

Field Value
sitemap https://hamisiburada.az/sitemap.xml
Back to top

Comments

  • robots.txt - Block all bots except Google
  • This file works with .htaccess bot blocking for comprehensive protection
  • Allow only Googlebot for SEO
  • Allow Googlebot Image for image indexing
  • Allow Googlebot News for news content
  • Allow Googlebot Video for video content
  • Block all other major search engine bots
  • Block SEO and marketing crawlers
  • Block AI training bots
  • Block social media crawlers
  • Block e-commerce crawlers
  • Block archiving and backup crawlers
  • Block security and research crawlers
  • Block generic crawlers and spiders
  • Default rule - block all other bots
  • Sitemap location (update with your actual sitemap URL if you have one)
  • Crawl-delay for any remaining bots (in seconds)
Back to top