Log In Sign Up

goabroad.com
robots.txt

Robots Exclusion Standard data for goabroad.com

Resource Scan

Scan Details

Site Domain goabroad.com
Base Domain goabroad.com
Scan Status Ok
Last Scan2024-05-15T03:04:22+00:00
Next Scan 2024-06-14T03:04:22+00:00

Last Scan

Scanned2024-05-15T03:04:22+00:00
URL https://goabroad.com/robots.txt
Redirect https://www.goabroad.com/robots.txt
Redirect Domain www.goabroad.com
Redirect Base goabroad.com
Domain IPs 35.168.250.62, 52.45.44.127, 52.71.199.3, 54.162.40.191
Redirect IPs 207.120.40.165, 207.120.40.166, 207.120.40.167, 207.120.40.168, 207.120.40.169, 207.120.40.170
Response IP 207.120.40.165
Found Yes
Hash 1c19c82690004b8e3414415899691ed44d06cc757168c9650e31484395461409
SimHash 26125a9a6ffd

Groups

googlebot

Rule Path
Allow /

twitterbot

Rule Path
Allow /

spider

Rule Path
Disallow /

bot-

Rule Path
Disallow /

bot/

Rule Path
Disallow /

linkchecker

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

irlbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

java

Rule Path
Disallow /

nicebot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

powermarks

Rule Path
Disallow /

missigua_locator

Rule Path
Disallow /

web downloader

Rule Path
Disallow /

lanshanbot

Rule Path
Disallow /

custo

Rule Path
Disallow /

cfnetwork

Rule Path
Disallow /

httrack off-line browser

Rule Path
Disallow /

nutchcvs

Rule Path
Disallow /

t-h-u-n-d-e-r-s-t-o-n-e

Rule Path
Disallow /

jakarta commons-httpclient

Rule Path
Disallow /

htmlparser

Rule Path
Disallow /

crawl

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

larbin

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

npbot

Rule Path
Disallow /

*

Rule Path
Allow /index.php
Disallow /api/*
Disallow /accommodations-abroad/
Disallow /feeds/
Disallow /rssfeeds/
Disallow /newsletter/
Disallow /newsletter*.cfm/
Disallow /bounce.cfm/
Disallow /bounce.cfm/
Disallow /bou_nce.cfm/
Disallow /*bounce*.cfm/
Disallow /bounce*.cfm/
Disallow /hitcounter.cfm/
Disallow /tracker.cfm/
Disallow /*program.cfm/
Disallow /news/wp-content/*.asp
Disallow /testimonial/replyToParticipant/
Disallow /testimonial/new/
Disallow /blog/feed/
Disallow /blog/trackback/
Disallow /blog/wp-admin/
Disallow /blog/wp-includes/
Disallow /blog/xmlrpc.php
Back to top

Other Records

Field Value
sitemap https://www.goabroad.com/sitemap_index.xml
sitemap https://www.goabroad.com/providers_sitemap_index.xml
sitemap https://www.goabroad.com/adventure-travel-abroad/sitemap_index.xml
sitemap https://www.goabroad.com/adventure-travel-abroad/country_sitemap_index.xml
sitemap https://www.goabroad.com/adventure-travel-abroad/region_sitemap_index.xml
sitemap https://www.goabroad.com/degree-abroad/sitemap_index.xml
sitemap https://www.goabroad.com/degree-abroad/country_sitemap_index.xml
sitemap https://www.goabroad.com/degree-abroad/region_sitemap_index.xml
sitemap https://www.goabroad.com/gap-year/sitemap_index.xml
sitemap https://www.goabroad.com/gap-year/country_sitemap_index.xml
sitemap https://www.goabroad.com/gap-year/region_sitemap_index.xml
sitemap https://www.goabroad.com/highschool-study-abroad/sitemap_index.xml
sitemap https://www.goabroad.com/highschool-study-abroad/country_sitemap_index.xml
sitemap https://www.goabroad.com/highschool-study-abroad/region_sitemap_index.xml
sitemap https://www.goabroad.com/intern-abroad/sitemap_index.xml
sitemap https://www.goabroad.com/intern-abroad/country_sitemap_index.xml
sitemap https://www.goabroad.com/intern-abroad/region_sitemap_index.xml
sitemap https://www.goabroad.com/language-study-abroad/sitemap_index.xml
sitemap https://www.goabroad.com/language-study-abroad/country_sitemap_index.xml
sitemap https://www.goabroad.com/language-study-abroad/region_sitemap_index.xml
sitemap https://www.goabroad.com/study-abroad/sitemap_index.xml
sitemap https://www.goabroad.com/study-abroad/country_sitemap_index.xml
sitemap https://www.goabroad.com/study-abroad/region_sitemap_index.xml
sitemap https://www.goabroad.com/teach-abroad/sitemap_index.xml
sitemap https://www.goabroad.com/teach-abroad/country_sitemap_index.xml
sitemap https://www.goabroad.com/teach-abroad/region_sitemap_index.xml
sitemap https://www.goabroad.com/tefl-courses/sitemap_index.xml
sitemap https://www.goabroad.com/tefl-courses/country_sitemap_index.xml
sitemap https://www.goabroad.com/tefl-courses/region_sitemap_index.xml
sitemap https://www.goabroad.com/volunteer-abroad/sitemap_index.xml
sitemap https://www.goabroad.com/volunteer-abroad/country_sitemap_index.xml
sitemap https://www.goabroad.com/volunteer-abroad/region_sitemap_index.xml
sitemap https://www.goabroad.com/scholarships-abroad/sitemap_index.xml
sitemap https://www.goabroad.com/blog/sitemap_index.xml
Back to top

Comments

  • Some bots are known to be trouble, particularly those designed to copy
  • entire sites. Please obey robots.txt.
  • Doesn't follow robots.txt anyway, but...
  • A capture bot, downloads gazillions of pages with no public benefit
  • http://www.webreaper.net/
  • Hits many times per second, not acceptable
  • http://www.nameprotect.com/botinfo.html
Back to top