airlive.net
robots.txt

Robots Exclusion Standard data for airlive.net

Resource Scan

Scan Details

Site Domain airlive.net
Base Domain airlive.net
Scan Status Ok
Last Scan2024-11-15T13:43:59+00:00
Next Scan 2024-11-22T13:43:59+00:00

Last Scan

Scanned2024-11-15T13:43:59+00:00
URL https://airlive.net/robots.txt
Domain IPs 104.21.76.227, 172.67.201.236, 2606:4700:3031::ac43:c9ec, 2606:4700:3034::6815:4ce3
Response IP 172.67.201.236
Found Yes
Hash 9a9a416664c174a9e610ab7717a3ecac37f1cd0e0d90cd136ce274bb40f02086
SimHash 6b245a8ad6a3

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

googlebot-news

Rule Path
Allow /category/exclusive/
Disallow /category/emergency/
Disallow /category/flight-attendant/
Disallow /category/history/
Disallow /category/infographic/
Disallow /category/mh17/
Disallow /category/mh370/
Disallow /category/infographic/
Disallow /category/military/
Disallow /category/news/
Disallow /category/newsletter/
Disallow /category/qz8501/
Disallow /category/space/
Disallow /category/spotting/