Log In Sign Up

airnav.com
robots.txt

Robots Exclusion Standard data for airnav.com

Resource Scan

Scan Details

Site Domain airnav.com
Base Domain airnav.com
Scan Status Ok
Last Scan2024-09-28T06:39:01+00:00
Next Scan 2024-10-28T06:39:01+00:00

Last Scan

Scanned2024-09-28T06:39:01+00:00
URL https://airnav.com/robots.txt
Domain IPs 178.128.135.124
Response IP 178.128.135.124
Found Yes
Hash 3140eaee0beaa03473df97c080c0941fae8cccc373fff17c414849a4ee58e42f
SimHash 305c53c1c9a3

Groups

teleport pro

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

scoutjet

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

blekkobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

hyperrobot

Rule Path
Disallow /

teoma

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

voilabot

Rule Path
Disallow /

iltrovatore-setaccio

Rule Path
Disallow /

diamondbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

*

Rule Path
Disallow /iws/

anthropic-ai

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 240

claudebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 240

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

awariobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

awariorssbot
awariosmartbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120
Back to top

Comments

  • Teleport Pro is too dumb, rude and agressive
  • Voila bot is too agressive, and there is no real benefit in being listed in voila.com
  • iltrovatore bot is too agressive, and there is no real benefit in being listed in iltrovatore
  • Disallow: /
  • Disallow: /
Back to top

Warnings

  • 3 invalid lines.
Back to top