yachtcharterfleet.com
robots.txt
Robots Exclusion Standard data for yachtcharterfleet.com
Resource Scan
Scan Details
| Site Domain | yachtcharterfleet.com |
| Base Domain | yachtcharterfleet.com |
| Scan Status | Ok |
| Last Scan | 2024-05-23T09:38:09+00:00 |
| Next Scan | 2024-06-22T09:38:09+00:00 |
Last Scan
| Scanned | 2024-05-23T09:38:09+00:00 |
| URL | https://yachtcharterfleet.com/robots.txt |
| Redirect | https://www.yachtcharterfleet.com/robots.txt |
| Redirect Domain | www.yachtcharterfleet.com |
| Redirect Base | yachtcharterfleet.com |
| Domain IPs | 178.32.61.230 |
| Redirect IPs | 54.39.26.122 |
| Response IP | 54.39.26.122 |
| Found | Yes |
| Hash | 7f09c2adf62f27843beb9d1f8881ea260dcd45bae0f549e0f6d136d9a8e3b09a |
| SimHash | 783a4900c7d0 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /my-account/ |
| Disallow | /feeds/ |
| Disallow | /yachts/save_search.htm |
| Disallow | /events/map.htm |
| Disallow | /directory/compare-yachts.htm |
| Disallow | /yachts/pdf-brochure/ |
| Disallow | /yachts/save_search.htm |
| Disallow | /yachts/yacht-shortlist.htm |
| Disallow | /instagram/embed_proxy |
crawler4j
crawler4j (http://code.google.com/p/crawler4j/)
curious george - www.analyticsseo.com/crawler
curious george - www.analyticsseo.com
megaindex.ru/2.0
megaindex.ru/
mozilla/5.0 (compatible; megaindex.ru/2.0; +https://www.megaindex.ru/?tab=linkanalyze)
mj12
mozilla/5.0 (compatible; steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
steeler
microsoft-webdav-miniredir/6.1.7601
mozilla/5.0 (compatible; findxbot/1.0; +http://www.findxbot.com)
findxbot
mozilla/5.0 (compatible; seznambot/3.2; +http://fulltext.sblog.cz/)
seznambot
seznam
mozilla/4.0 (compatible; vagabondo/4.0; http://webagent.wise-guys.nl/)
mozilla/4.0 (compatible; vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
vagabondo
jakarta commons-httpclient/3.0.1
lwp::simple/5.822
mozilla/5.0 (compatible; smtbot/1.0; +http://www.similartech.com/smtbot)
smtbot
mozilla/5.0 (compatible; memorybot/1.22.56 +http://internetmemory.org/en/)
memorybot
mozilla/5.0 (compatible; smrjbot/0.0.20)
smrjbot
mozilla/5.0 (compatible; spbot/4.4.2; +http://openlinkprofiler.org/bot )
spbot
mauibot (crawler.feedback+dc@gmail.com)
ia_archiver
blexbot
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.yachtcharterfleet.com/sitemap.xml |
Warnings
- 2 invalid lines.
Comments