Log In Sign Up

virtualglobetrotting.com
robots.txt

Robots Exclusion Standard data for virtualglobetrotting.com

Resource Scan

Scan Details

Site Domain virtualglobetrotting.com
Base Domain virtualglobetrotting.com
Scan Status Ok
Last Scan2024-09-21T02:35:42+00:00
Next Scan 2024-09-28T02:35:42+00:00

Last Scan

Scanned2024-09-21T02:35:42+00:00
URL https://virtualglobetrotting.com/robots.txt
Domain IPs 64.34.182.211
Response IP 64.34.182.211
Found Yes
Hash 7cf9196e2278e8ed36dd0b57179d73e40ee69ec385b0283480547ff9a1db54fb
SimHash bcbc1b0d45b0

Groups

mediapartners-google

Rule Path
Disallow

*

Rule Path
Disallow *STARTNUM*
Disallow *export-1.kml$
Disallow *export-2.kml$
Disallow *export-3.kml$
Disallow *export-4.kml$
Disallow *export-5.kml$
Disallow *export-6.kml$
Disallow *export-7.kml$
Disallow *export-8.kml$
Disallow *quickpage*
Allow *?s=0&*
Disallow *?s=1*
Disallow *?s=2*
Disallow *?s=3*
Disallow *?s=4*
Disallow *?s=5*
Disallow *?s=6*
Disallow *?s=7*
Disallow *?s=8*
Disallow *?s=9*
Allow */0/?v=
Disallow *0/?v=
Disallow *1/?v=
Disallow *2/?v=
Disallow *3/?v=
Disallow *4/?v=
Disallow *5/?v=
Disallow *6/?v=
Disallow *7/?v=
Disallow *8/?v=
Disallow *9/?v=
Disallow /ajax/
Disallow /ll/
Disallow /map/*/archive/
Disallow /map/*/graph.png
Disallow /map/*/nearby/
Disallow /report/
Disallow /rss_comments.php
Disallow /rss_comments/
Disallow /favorites/
Disallow /my_favorites/
Disallow /achievement_info.php
Disallow /search/
Disallow /search.php
Disallow /search-results-*.kml
Disallow /share/
Disallow /submit_thumb/
Disallow /show.php
Disallow /send_message/
Disallow /forums/printthread.php
Disallow /forums/showprofile.php
Disallow /forums/showthreaded.php

omniexplorer_bot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://virtualglobetrotting.com/sitemaps/sitemap_index.xml
Back to top

Comments

  • from parsed URLs
  • don't crawl other filters for KMLs
  • don't crawl non-first pages
  • don't crawl ajax requests
  • coordinates
  • map extra details
  • report map apge
  • rss
  • logged in user
  • search pages - excluding 2024-04 because of Search Results SPAM
  • search results KML
  • submit pages
  • shouldn't be able links to this anymore
  • Messages
  • Forums
  • same content as in showflat
  • forum profiles are unimportant
  • don't need both flat and threaded crawled
  • Crawling too much
  • Crawling too much
  • Crawling too much
Back to top