Log In Sign Up

allflyingjobs.com
robots.txt

Robots Exclusion Standard data for allflyingjobs.com

Resource Scan

Scan Details

Site Domain allflyingjobs.com
Base Domain allflyingjobs.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-11-04T08:16:39+00:00
Next Scan 2025-02-02T08:16:39+00:00

Last Successful Scan

Scanned2024-04-15T08:14:51+00:00
URL https://allflyingjobs.com/robots.txt
Domain IPs 104.21.74.188, 172.67.205.194, 2606:4700:3030::ac43:cdc2, 2606:4700:3037::6815:4abc
Response IP 104.21.74.188
Found Yes
Hash 72ddbc38d392413b9a194143b1b0a001b16814ba187decaf0d22d41f1201fb5e
SimHash 780457f8ce78

Groups

gptbot

Rule Path
Disallow /

honolulubot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

hybridbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

tweetmemebot

Rule Path
Disallow /

livelapbot

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

anderspinkbot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

epicbot

Rule Path
Disallow /

nsrbot

Rule Path
Disallow /

coccocbot-image

Rule Path
Disallow /

moatbot

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

fyrebot

Rule Path
Disallow /

extlinksbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

obot

Rule Path
Disallow /

mappy

Rule Path
Disallow /

spbot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

linkbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

leikibot

Rule Path
Disallow /

buzzbot

Rule Path
Disallow /

findxbot

Rule Path
Disallow /

ruby

Rule Path
Disallow /

everyonesocialbot

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

mangoway

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

*

Rule Path
Allow /misc/*.css$
Allow /misc/*.css?
Allow /misc/*.js$
Allow /misc/*.js?
Allow /misc/*.gif
Allow /misc/*.jpg
Allow /misc/*.jpeg
Allow /misc/*.png
Allow /modules/*.css$
Allow /modules/*.css?
Allow /modules/*.js$
Allow /modules/*.js?
Allow /modules/*.gif
Allow /modules/*.jpg
Allow /modules/*.jpeg
Allow /modules/*.png
Allow /profiles/*.css$
Allow /profiles/*.css?
Allow /profiles/*.js$
Allow /profiles/*.js?
Allow /profiles/*.gif
Allow /profiles/*.jpg
Allow /profiles/*.jpeg
Allow /profiles/*.png
Allow /themes/*.css$
Allow /themes/*.css?
Allow /themes/*.js$
Allow /themes/*.js?
Allow /themes/*.gif
Allow /themes/*.jpg
Allow /themes/*.jpeg
Allow /themes/*.png
Disallow /includes/
Disallow /misc/
Disallow /modules/
Disallow /profiles/
Disallow /scripts/
Disallow /themes/
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /INSTALL.sqlite.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips/
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /?q=admin%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=filter%2Ftips%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F
Disallow /?q=user%2Flogout%2F
Disallow /users/
Disallow /user/
Disallow /user
Disallow /forum/
Disallow /forums/
Disallow /notifications/
Disallow /applications/
Disallow /content/
Disallow /apply/
Disallow /apply2/
Disallow /apply-redirect/
Disallow /apply-redirect2/
Disallow /flag/
Disallow /static/
Disallow /recruiter/register
Disallow /recruiter/password
Disallow /recruiter
Disallow /pilot/register
Disallow /pilot/password
Disallow /filter/tips
Disallow /modules/statistics/statistics.php
Disallow /modules/radioactivity/emit.php
Disallow /*qt-left
Disallow /*qt-sidebar
Disallow /*?s=*
Disallow /*?0=*
Disallow /*?f%5B*%5D=
Disallow /*?ts=
Disallow /*recruiter-name/jobs*
Disallow /*recruiter_name/jobs*
Disallow /*jobs/*location/*/location
Disallow /*jobs/*aircraft-type/*/aircraft-type
Allow /jobs-text-search?op=*&s=$
Allow /*jobs/*aircraft-type/*/location/*?page=0
Allow /*jobs/*location/*/aircraft-type/*?page=0
Allow /cabin-crew-jobs/*location/*?page=0
Allow /aircraft-engineer-jobs/*location/*?page=0
Allow /air-traffic-controller-jobs/*location/*?page=0
Allow /ground-operations-aviation-jobs/*location/*?page=0
Allow /aviation-intern-stage-jobs/*location/*?page=0
Allow /aviation-ground-jobs/*location/*?page=0
Allow /aviation-flight-training-offers/*location/*?page=0
Back to top

Other Records

Field Value
sitemap https://www.allflyingjobs.com/sitemap.xml
Back to top

Comments

  • User-agent: AhrefsBot
  • Disallow: /
  • User-agent: SemrushBot
  • Disallow: /
  • User-agent: MJ12bot
  • Disallow: /
  • CSS, JS, Images
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Custom
  • Disallow: /privacy
  • Disallow: /terms-of-use
  • Disallow: /contact
  • Disallow: /*jobs/*aircraft-type/*/location/*?page=*
  • Disallow: /*jobs/*location/*/aircraft-type/*?page=*
  • Disallow: /cabin-crew-jobs/*location/*?page=*
  • Disallow: /aircraft-engineer-jobs/*location/*?page=*
  • Disallow: /air-traffic-controller-jobs/*location/*?page=*
  • Disallow: /ground-operations-aviation-jobs/*location/*?page=*
  • Disallow: /aviation-intern-stage-jobs/*location/*?page=*
  • Disallow: /aviation-ground-jobs/*location/*?page=*
  • Disallow: /aviation-flight-training-offers/*location/*?page=*
  • Disallow: /jobs-text-search?op=*&s=*
Back to top

Warnings

  • 3 invalid lines.
Back to top