Log In Sign Up

dialaflight.com
robots.txt

Robots Exclusion Standard data for dialaflight.com

Resource Scan

Scan Details

Site Domain dialaflight.com
Base Domain dialaflight.com
Scan Status Ok
Last Scan2025-11-16T00:09:44+00:00
Next Scan 2025-12-16T00:09:44+00:00

Last Scan

Scanned2025-11-16T00:09:44+00:00
URL https://dialaflight.com/robots.txt
Redirect https://www.dialaflight.com/robots.txt
Redirect Domain www.dialaflight.com
Redirect Base dialaflight.com
Domain IPs 79.173.173.75
Redirect IPs 79.173.173.75
Response IP 79.173.173.75
Found Yes
Hash 3bcbb85fe1e4d9900070736b2bceb9e8b3a014438e4b75d915abe5d988865fc8
SimHash f3124210d7f2

Groups

*

Rule Path
Disallow /images/
Disallow /flights/duplicatelocationsearch.aspx
Disallow /businesstravel/duplicatelocationsearch.aspx
Disallow /utility/callbackrequest.aspx
Disallow /utility/datedflightsearchjson.aspx
Disallow /utility/verticalbuttonimg.aspx*
Disallow /utility/excursionimg.aspx*
Disallow /*searchresults.aspx*
Disallow /*globalpropertyid*
Disallow /*duplicatelocationsearch.aspx*
Disallow /featuredlocations/*/restaurants/images/
Disallow /featuredlocations/*/bars/images/
Disallow /featuredlocations/*
Disallow /insight/*
Disallow /corporatetravel/img/*
Disallow /corporatetravel/manage/*
Disallow /static/

oai-searchbot

Rule Path
Allow /

chatgpt-user
chatgpt-user/2.0

Rule Path
Allow /

gptbot

Rule Path
Allow /

anthropic-ai

Product Comment
anthropic-ai bulk model training
Rule Path
Allow /

claudebot
claude-web

Product Comment
claudebot chat citation fetch
claude-web web-focused crawl
Rule Path
Allow /

perplexitybot

Product Comment
perplexitybot index builder
Rule Path
Allow /

perplexity-user

Product Comment
perplexity-user human-triggered visit
Rule Path
Allow /

googlebot
google-extended

Rule Path
Allow /

bingbot
bingbot/2.0
microsoft-extended

Rule Path
Allow /

amazonbot

Rule Path
Allow /

applebot
applebot-extended

Rule Path
Allow /

facebookbot
meta-externalagent
meta-externalfetcher
facebookexternalhit

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

bytespider

Rule Path
Allow /

duckassistbot

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

ai2bot
ccbot
diffbot
omgili

Rule Path
Allow /

timpibot
youbot

Rule Path
Allow /

mj12bot

Rule Path
Disallow /

*

Rule Path
Disallow /images/hotels/cache/
Back to top

Other Records

Field Value
sitemap https://www.dialaflight.com/Sitemap.xml
sitemap https://www.dialaflight.com/VideoSitemap.xml
Back to top

Comments

  • You can paste any of these blocks into robots.txt or a firewall rule. grouped by company to make things readable for y'all.
  • ——— OPENAI ———
  • Search (shows my webpages as links inside ChatGPT search). NOT used for model training.
  • User-driven browsing from ChatGPT and Custom GPTs. Acts after a human click.
  • Model-training crawler. Opt-out here if I don’t want content in GPT-4o or GPT-5.
  • ——— ANTHROPIC (Claude) ———
  • ——— PERPLEXITY ———
  • ——— GOOGLE (Gemini) ———
  • ——— MICROSOFT (Bing / Copilot) ———
  • ——— AMAZON ———
  • ——— APPLE ———
  • ——— META ———
  • ——— LINKEDIN ———
  • ——— BYTEDANCE ———
  • ——— DUCKDUCKGO ———
  • ——— COHERE ———
  • ——— ALLEN INSTITUTE / COMMON CRAWL / OTHER RESEARCH ———
  • ——— EMERGING SEARCH START-UPS ———
Back to top