airlinesinfocare.com
robots.txt

Robots Exclusion Standard data for airlinesinfocare.com

Resource Scan

Scan Details

Site Domain airlinesinfocare.com
Base Domain airlinesinfocare.com
Scan Status Ok
Last Scan2024-09-24T15:54:23+00:00
Next Scan 2024-10-01T15:54:23+00:00

Last Scan

Scanned2024-09-24T15:54:23+00:00
URL https://airlinesinfocare.com/robots.txt
Domain IPs 154.49.142.222, 2a02:4780:b:1096:0:6d4:bbdf:2
Response IP 154.49.142.222
Found Yes
Hash e3dd9d6c307b02ad223c522583768e35e1b33d13117b22b890de35bc053e22c5
SimHash 6100d97156d9

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

googlebot
googlebot-image
mediapartners-google
msnbot
msnbot-media
slurp
yahoo-blogs
duckduckbot
baiduspider
yandexbot
ia_archiver
yahoo-mmcrawler

Rule Path
Disallow /cgi-bin/

Other Records

Field Value
crawl-delay 6

Comments

  • but allow only important bots
  • Directories