Log In Sign Up

travel.earth
robots.txt

Robots Exclusion Standard data for travel.earth

Resource Scan

Scan Details

Site Domain travel.earth
Base Domain travel.earth
Scan Status Ok
Last Scan2024-06-26T08:18:37+00:00
Next Scan 2024-07-26T08:18:37+00:00

Last Scan

Scanned2024-06-26T08:18:37+00:00
URL https://travel.earth/robots.txt
Domain IPs 3.110.0.163
Response IP 3.110.0.163
Found Yes
Hash a6889aea2f8e7ee9052583eb7c198401b89a79eb1996c9a15712280ca12e8fbf
SimHash 65158541f67e

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /linkout/
Disallow /recommended/
Disallow /comments/feed/
Disallow /trackback/
Disallow /index.php
Disallow /xmlrpc.php
Disallow /tag/
Disallow /page/
Allow /romantic-rome-italy/amp/
Allow /15-of-the-most-exciting-places-to-visit-in-dubai/amp/
Allow /visa-exemptions-for-foreigners-seeking-medical-treatment-india/amp/
Allow /best-geneva-in-winter/amp/
Allow /category/europe/denmark/amp/

rogerbot

Rule Path
Disallow /tag/
Disallow /quizs*

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterestbot/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterest/0.2 (+https://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (compatible; pinterestbot/1.0; +https://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (linux; android 6.0.1; nexus 5x build/mmb29p) applewebkit/537.36 (khtml, like gecko) chrome/41.0.2272.96 mobile safari/537.36 (compatible; pinterestbot/1.0; +https://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot-sa

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mediapartners-google*

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

seekport crawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

seekbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yandex

Rule Path
Disallow /admin/*

mozilla/5.0 (compatible; mj12bot/v1.4.8; https://mj12bot.com/)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

facebookexternalhit/1.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

facebookexternalhit/1.1 (+https://www.facebook.com/externalhit_uatext.php)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mozilla/5.0 (macintosh; intel mac os x 10_9) applewebkit/537.71 (khtml, like gecko) version/7.0 safari/537.71 (rival iq, rivaliq.com)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (macintosh; intel mac os x 10_11_6) applewebkit/537.36 (khtml, like gecko) chrome/52.0.2743.116 safari/537.36 (rival iq, rivaliq.com)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0(macintosh; intel mac os x 10_11_6)applewebkit/537.36(khtml, like gecko)chrome/52.0.2743.116safari/537.36(rival iq, rivaliq.com)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (macintosh; intel mac os x 10_11_6) applewebkit/537.36 (khtml, like gecko) chrome/52.0.2743.116 safari/537.36 (rival iq, rivaliq.com)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (macintosh; intel mac os x 10_9) applewebkit/537.71 (khtml, like gecko) version/7.0 safari/537.71 (rival iq, rivaliq.com)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

haosouspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0+(compatible; uptimerobot/2.0; https://www.uptimerobot.com/)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (compatible; bingbot/2.0; +https://www.bing.com/bingbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 applewebkit/537.36 (khtml, like gecko; compatible; bingbot/2.0; +https://www.bing.com/bingbot.htm) chrome/w.x.y.z safari/537.36 edg/w.x.y.z

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 (linux; android 6.0.1; nexus 5x build/mmb29p) applewebkit/537.36 (khtml, like gecko) chrome/w.x.y.z mobile safari/537.36 edg/w.x.y.z (compatible; bingbot/2.0; +https://www.bing.com/bingbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 (compatible; adidxbot/2.0; +https://www.bing.com/bingbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 (iphone; cpu iphone os 7_0 like mac os x) applewebkit/537.51.1 (khtml, like gecko) version/7.0 mobile/11a465 safari/9537.53 (compatible; adidxbot/2.0; +https://www.bing.com/bingbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 (windows phone 8.1; arm; trident/7.0; touch; rv:11.0; iemobile/11.0; nokia; lumia 530) like gecko (compatible; adidxbot/2.0; +https://www.bing.com/bingbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 (windows nt 6.1; wow64) applewebkit/534+ (khtml, like gecko) bingpreview/1.0b

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 (windows phone 8.1; arm; trident/7.0; touch; rv:11.0; iemobile/11.0; nokia; lumia 530) like gecko bingpreview/1.0b

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2
Back to top

Other Records

Field Value
sitemap https://travel.earth/sitemap_index.xml
Back to top

Comments

  • Crawl-delay: 30
  • User-agent: Slurp
  • Crawl-delay: 10
  • User-agent: DuckDuckBot
  • Crawl-delay: 10
  • User-agent: ia_archiver
  • Crawl-delay: 10
  • User-agent: amazonbot
  • Crawl-delay: 10
  • User-agent: NinjaBot
  • Disallow: /
  • Allow Seekbot with delay - https://www.seekport.co.uk/seekbot/
  • Allow Seekbot with delay - https://www.seekport.co.uk/seekbot/
  • User-agent: Mozilla/5.0 (compatible; YaK/1.0; https://linkfluence.com/; bot@linkfluence.com)
  • Disallow: /
  • User-agent: Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)
  • Disallow: /
  • User-agent: Mozilla/5.0 (compatible; Daum/4.1; +https://cs.daum.net/faq/15/4118.html?faqId=28966)
  • Disallow: /
  • new bots info added
  • User-agent: Yandex
  • Crawl-delay: 10
  • User-agent: Mozilla/5.0 (compatible; Semanticbot/1.0; +https://sempi.tech/bot.html)
  • Crawl-delay: 10
  • User-agent: Baiduspider
  • Crawl-delay: 10
  • User-agent: PetalBot
  • Disallow: /
  • User-agent: Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot)
  • Disallow: /
  • Source: https://www.rocketrental.co.nz/robots.txt
  • Header: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
  • Header: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; https://www.haosou.com/help/help_3_2.html)
  • Source: https://developers.whatismybrowser.com/useragents/parse/1109735-uptime-robot
  • bing other agents delay goes here. Source: https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0
  • and siteground.com/kb/how_to_decrease_the_crawl_rate_of_bings_search_engine_bot/
Back to top

Warnings

  • 2 invalid lines.
Back to top