flightcentre.co.za
robots.txt
Robots Exclusion Standard data for flightcentre.co.za
Resource Scan
Scan Details
Site Domain | flightcentre.co.za |
Base Domain | flightcentre.co.za |
Scan Status | Ok |
Last Scan | 2024-05-23T18:51:51+00:00 |
Next Scan | 2024-06-22T18:51:51+00:00 |
Last Scan
Scanned | 2024-05-23T18:51:51+00:00 |
URL | https://flightcentre.co.za/robots.txt |
Redirect | https://www.flightcentre.co.za/robots.txt |
Redirect Domain | www.flightcentre.co.za |
Redirect Base | flightcentre.co.za |
Domain IPs | 54.252.210.73 |
Redirect IPs | 108.156.133.100, 108.156.133.5, 108.156.133.84, 108.156.133.9, 2600:9000:2755:1000:1b:2cd9:2c40:93a1, 2600:9000:2755:1200:1b:2cd9:2c40:93a1, 2600:9000:2755:4c00:1b:2cd9:2c40:93a1, 2600:9000:2755:5800:1b:2cd9:2c40:93a1, 2600:9000:2755:5c00:1b:2cd9:2c40:93a1, 2600:9000:2755:a400:1b:2cd9:2c40:93a1, 2600:9000:2755:d400:1b:2cd9:2c40:93a1, 2600:9000:2755:fc00:1b:2cd9:2c40:93a1 |
Response IP | 108.156.133.84 |
Found | Yes |
Hash | 90ec9fb3d7995b875fb0ddad0a45d8e2f005efe9f449386621be7565786177bb |
SimHash | 38963d01c770 |
Groups
*
Rule | Path |
---|---|
Allow | /misc/*.css$ |
Allow | /misc/*.css? |
Allow | /misc/*.js$ |
Allow | /misc/*.js? |
Allow | /misc/*.gif |
Allow | /misc/*.jpg |
Allow | /misc/*.jpeg |
Allow | /misc/*.png |
Allow | /modules/*.css$ |
Allow | /modules/*.css? |
Allow | /modules/*.js$ |
Allow | /modules/*.js? |
Allow | /modules/*.gif |
Allow | /modules/*.jpg |
Allow | /modules/*.jpeg |
Allow | /modules/*.png |
Allow | /profiles/*.css$ |
Allow | /profiles/*.css? |
Allow | /profiles/*.js$ |
Allow | /profiles/*.js? |
Allow | /profiles/*.gif |
Allow | /profiles/*.jpg |
Allow | /profiles/*.jpeg |
Allow | /profiles/*.png |
Allow | /themes/*.css$ |
Allow | /themes/*.css? |
Allow | /themes/*.js$ |
Allow | /themes/*.js? |
Allow | /themes/*.gif |
Allow | /themes/*.jpg |
Allow | /themes/*.jpeg |
Allow | /themes/*.png |
Disallow | /includes/ |
Disallow | /misc/ |
Disallow | /modules/ |
Disallow | /profiles/ |
Disallow | /scripts/ |
Disallow | /themes/ |
Disallow | /CHANGELOG.txt |
Disallow | /cron.php |
Disallow | /INSTALL.mysql.txt |
Disallow | /INSTALL.pgsql.txt |
Disallow | /INSTALL.sqlite.txt |
Disallow | /install.php |
Disallow | /INSTALL.txt |
Disallow | /LICENSE.txt |
Disallow | /MAINTAINERS.txt |
Disallow | /update.php |
Disallow | /UPGRADE.txt |
Disallow | /xmlrpc.php |
Disallow | /admin/ |
Disallow | /comment/reply/ |
Disallow | /filter/tips/ |
Disallow | /node/add/ |
Disallow | /search/ |
Disallow | /user/register/ |
Disallow | /user/password/ |
Disallow | /user/login/ |
Disallow | /user/logout/ |
Disallow | /?q=admin%2F |
Disallow | /?q=comment%2Freply%2F |
Disallow | /?q=filter%2Ftips%2F |
Disallow | /?q=node%2Fadd%2F |
Disallow | /?q=search%2F |
Disallow | /?q=user%2Fpassword%2F |
Disallow | /?q=user%2Fregister%2F |
Disallow | /?q=user%2Flogin%2F |
Disallow | /?q=user%2Flogout%2F |
Disallow | /*/search? |
Disallow | /*/search$ |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
bingbot
Rule | Path |
---|---|
Disallow | /*/search? |
Disallow | /*/search$ |
Disallow | /search- |
Disallow | /search? |
Other Records
Field | Value |
---|---|
crawl-delay | 30 |
rogerbot
Rule | Path |
---|---|
Disallow | /product/ |
Disallow | /deals/*?int_source |
Disallow | /cruise/details |
Disallow | /*/%7B%7B%20profileURL%20%7D%7D |
Disallow | /*/*/%7B%7B%20profileURL%20%7D%7D |
Disallow | /*?*page=* |
Disallow | /accommodation/%5B%5E/%5D%2B/%5B%5E/%5D%2B.* |
Other Records
Field | Value |
---|---|
crawl-delay | 30 |
googlebot
Rule | Path |
---|---|
Disallow | /product/* |
Disallow | /cruises/product/* |
Disallow | /flights/*/1* |
Disallow | /flights/*/2* |
Disallow | /flights/*/3* |
Disallow | /flights/*/4* |
Disallow | /flights/*/5* |
Disallow | /holidays/*/1* |
Disallow | /holidays/*/2* |
Disallow | /holidays/*/3* |
Disallow | /holidays/*/4* |
Disallow | /holidays/*/5* |
Disallow | /tours/*/1* |
Disallow | /tours/*/2* |
Disallow | /tours/*/3* |
Disallow | /tours/*/4* |
Disallow | /tours/*/5* |
Disallow | /cruises/*/1* |
Disallow | /cruises/*/2* |
Disallow | /cruises/*/3* |
Disallow | /cruises/*/4* |
Disallow | /cruises/*/5* |
Disallow | /accommodation/*/1* |
Disallow | /accommodation/*/2* |
Disallow | /accommodation/*/3* |
Disallow | /accommodation/*/4* |
Disallow | /accommodation/*/5* |
Disallow | /luxury/*/1* |
Disallow | /luxury/*/2* |
Disallow | /luxury/*/3* |
Disallow | /luxury/*/4* |
Disallow | /luxury/*/5* |
Other Records
Field | Value |
---|---|
crawl-delay | 30 |
Other Records
Field | Value |
---|---|
sitemap | https://www.flightcentre.co.za/sitemap.xml |
sitemap | https://www.flightcentre.co.za/aurora/api/sitemap/en-ZA/content-sitemap.xml?types=blog_post,rich_page,blogs_landing,flights_landing,flights_destination,flights_p2p,holidays_landing,holidays_destination,holidays_type,travel_guide_page |
sitemap | https://www.flightcentre.co.za/stores/sitemap.xml |
Comments