Log In Sign Up

passport.et
robots.txt

Robots Exclusion Standard data for passport.et

Resource Scan

Scan Details

Site Domain passport.et
Base Domain passport.et
Scan Status Ok
Last Scan2025-10-25T13:46:44+00:00
Next Scan 2025-11-01T13:46:44+00:00

Last Scan

Scanned2025-10-25T13:46:44+00:00
URL https://passport.et/robots.txt
Domain IPs 104.21.51.157, 172.67.182.127, 2606:4700:3030::6815:339d, 2606:4700:3036::ac43:b67f
Response IP 172.67.182.127
Found Yes
Hash 9e12de8ea8e3cb30497fe8643a628dcddabbb4c2ee2de2581b86a9c1a5b68128
SimHash 609cdf00f5b6

Groups

*

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

googlebot
googlebot-image
googlebot-news
googlebot-video

Rule Path
Allow /

bingbot
bingpreview

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

yandex
yandexbot

Rule Path
Allow /

gptbot
chatgpt-user

Rule Path
Allow /

claude-web
claudebot

Rule Path
Allow /

google-extended

Rule Path
Allow /

ccbot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

facebookbot
facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

pinterestbot

Rule Path
Allow /

telegrambot

Rule Path
Allow /

slackbot
slackbot-linkexpanding

Rule Path
Allow /

whatsapp

Rule Path
Allow /

discordbot

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

semrushbot

Rule Path
Allow /

rogerbot
dotbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://passport.et/sitemap.xml
Back to top

Comments

  • Robots.txt - Open to All Search Engines & AI Bots
  • Website: https://passport.et
  • Updated: 2024
  • ==============================================
  • ALLOW ALL BOTS - WELCOME ALL CRAWLERS
  • ==============================================
  • Default: Allow all robots
  • ==============================================
  • MAJOR SEARCH ENGINE BOTS (Explicitly Listed)
  • ==============================================
  • Google
  • Bing
  • Yahoo
  • DuckDuckGo
  • Baidu (China)
  • Yandex (Russia)
  • ==============================================
  • AI & LLM CRAWLERS (Training & Research)
  • ==============================================
  • OpenAI (ChatGPT, GPT-4)
  • Anthropic (Claude)
  • Google AI (Bard, Gemini)
  • Common Crawl (Open dataset)
  • Perplexity AI
  • Cohere
  • Facebook/Meta AI
  • ==============================================
  • SOCIAL MEDIA & PREVIEW BOTS
  • ==============================================
  • Twitter/X
  • LinkedIn
  • Pinterest
  • Telegram
  • Slack
  • WhatsApp
  • Discord
  • ==============================================
  • SEO & ANALYTICS TOOLS
  • ==============================================
  • Ahrefs
  • SEMrush
  • Moz
  • ==============================================
  • SITEMAP LOCATION
  • ==============================================
Back to top