Log In Sign Up

m.justfly.com
robots.txt

Robots Exclusion Standard data for m.justfly.com

Resource Scan

Scan Details

Site Domain m.justfly.com
Base Domain justfly.com
Scan Status Ok
Last Scan2025-12-15T23:52:14+00:00
Next Scan 2025-12-29T23:52:14+00:00

Last Scan

Scanned2025-12-15T23:52:14+00:00
URL https://m.justfly.com/robots.txt
Redirect https://www.justfly.com/robots.txt
Redirect Domain www.justfly.com
Redirect Base justfly.com
Domain IPs 104.18.12.48, 104.18.13.48, 2606:4700::6812:c30, 2606:4700::6812:d30
Redirect IPs 104.18.12.48, 104.18.13.48, 2606:4700::6812:c30, 2606:4700::6812:d30
Response IP 104.18.12.48
Found Yes
Hash ef4fa0e61e9afb559366c4d43a6abf2d45ef32664467e7a4bdfb18cc6b499708
SimHash 28128c72dbd6

Groups

*

Rule Path
Allow /$
Allow /?
Allow /flights/
Allow /flights?
Allow /flights$
Allow /cheap-flights
Allow /email-deals
Allow /fr/cheap-flights
Allow /fr/email-deals
Allow /cf-assets/images/
Allow /cf-assets
Allow /airports
Disallow /font-awesome
Disallow /bootstrap
Disallow /dispute
Disallow /self-transfer
Disallow /cf-fonts
Disallow /css
Disallow /fonts
Disallow /gass
Disallow /nuyp
Disallow /js
Disallow /storefront-api
Disallow /material-icomoon
Disallow /mobile-flags
Disallow /upsell
Disallow /seatmap
Disallow /surveys
Disallow /flight-search-api/
Disallow /autocomplete
Disallow /storefront-api/
Disallow /gass/
Disallow /cdn-cgi/
Disallow /fare-alerts
Disallow /privacy-policy
Disallow /privacypolicy
Disallow /terms-and-conditions
Disallow /fares
Disallow /account
Disallow /my-account
Disallow /checkout
Disallow /booking
Disallow /support
Disallow /hotel
Disallow /flight
Disallow /cars
Disallow /hotels
Disallow /service
Disallow /landing-flights/ajax
Disallow /fr/fare-alerts
Disallow /fr/privacy-policy
Disallow /fr/privacypolicy
Disallow /fr/terms-and-conditions
Disallow /fr/fares
Disallow /fr/account
Disallow /fr/my-account
Disallow /fr/checkout
Disallow /fr/booking
Disallow /fr/support
Disallow /fr/hotel
Disallow /fr/flight
Disallow /fr/cars
Disallow /fr/hotels
Disallow /fr/service
Disallow /fr/landing-flights/ajax

google-hoteladsverifier
adsbot-google-mobile-apps
adsbot-google-mobile
adsbot-google
meta-externalads
meta-externalagent
meta-externalfetcher
criteobot
adidxbot

Rule Path
Allow /$
Allow /?
Allow /cheap-flights
Allow /email-deals
Allow /fr/cheap-flights
Allow /fr/email-deals
Allow /cf-assets/images/
Disallow /

mediapartners-google

Rule Path
Disallow

semrushbot
eyemonit
buck
mediatoolkitbot
seznambot
taboolabot
mail.ru
bidswitchbot
mj12bot
zoombot
seostar
neevabot
zoominfobot
linespider
ahrefsbot
dotbot
blexbot
megaindex.ru
megaindexbot
seokicks-robot
spbot
barkrowler
petalbot
bytespider
mauibot
baiduspider
sogou spider
yandexbot
ccbot

Rule Path
Disallow /
Back to top

Comments

  • General catchall rules
  • Common Paths
  • English Paths
  • French Paths (same order as English)
  • Advertiser Group: Paid Advertising Crawlers
  • Google Ads bots
  • Meta Ads bots
  • Criteo Ads bot
  • Microsoft Ads bot
  • Advertiser: Google AdSense - Requires broad access for ad display validation
  • Unfavorable Bots
  • SemrushBot - Semrush Inc. - Will hammer the site endlessly for competition intel
  • EyeMonIT - EyeMonIT - Constant health checks that bury your server
  • Buck - Buck - Consumes resources for shady marketing data
  • Mediatoolkitbot - Mediatoolkit - Scrapes content non-stop for social metrics
  • SeznamBot - Seznam.cz - Floods pages for Czech search, slowing global users
  • Taboolabot - Taboola - Sucks bandwidth to serve more native ads elsewhere
  • Mail.Ru - Mail.ru - Heavy Russian crawler draining crawl budget
  • bidswitchbot - BidSwitch - Harvests pages for ad bidding profiles around the clock
  • MJ12bot - Majestic - Builds backlink maps while saturating your logs
  • ZoomBot - Unknown - Aggressive spider that chokes small servers
  • seostar - SEOstar - Obsessive SEO crawler that doesn’t respect rate limits
  • Neevabot - Neeva - Crawls relentlessly even though search engine is defunct
  • ZoominfoBot - ZoomInfo - Mines personal data for B2B dossiers 24/7
  • Linespider - Unknown - Spams requests causing performance hits
  • AhrefsBot - Ahrefs - Eats up bandwidth mapping links for competitors
  • DotBot - Moz - Crawls full site to fuel Moz index at massive scale
  • BLEXBot - Blex - Extracts links quickly causing spike in CPU
  • MegaIndex.ru - MegaIndex - Russian SEO crawler notorious for high request rates
  • MegaIndexBot - MegaIndex - Duplicate aggressive crawler variant
  • SEOkicks-Robot - SEOkicks - German SEO bot flooding pages for link data
  • spbot - Seo-Profiler - Rapid-fire requests that exhaust resources
  • Barkrowler - Common Crawl - Full-site archive sweeps causing heavy load
  • PetalBot - Huawei - Huge crawler building Petal search index globally
  • Bytespider - ByteDance - TikTok parent collects data aggressively
  • MauiBot - Unknown - Exotic bot that blitzes pages without mercy
  • Baiduspider - Baidu - Massive Chinese crawler that overloads sites
  • Sogou Spider - Sogou - Another Chinese search bot taxing bandwidth
  • YandexBot - Yandex - Russian crawler known for deep recursive fetches
  • CCBot - Common Crawl - Huge web archiving bot devouring crawl budget
Back to top