Log In Sign Up

marinesource.com
robots.txt

Robots Exclusion Standard data for marinesource.com

Resource Scan

Scan Details

Site Domain marinesource.com
Base Domain marinesource.com
Scan Status Ok
Last Scan2024-11-12T22:38:28+00:00
Next Scan 2024-11-19T22:38:28+00:00

Last Scan

Scanned2024-11-12T22:38:28+00:00
URL https://marinesource.com/robots.txt
Domain IPs 142.214.209.18
Response IP 142.214.209.18
Found Yes
Hash 673eea72e40142621dbf313fbcedb337b59eff5929d757db9d7b7cf62a9ccd88
SimHash c433d811623a

Groups

clickagy*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

ccbot*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

baiduspider-image

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

baiduspider-video

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

baiduspider-news

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

baiduspider-favo

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

baiduspider-cpro

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

baiduspider-ads

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

google*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

semrushbot*

Rule Path
Disallow /boats-for-sale/listing_print.cfm
Disallow /boats-for-sale/listing_print_pretty.cfm
Disallow /brokers/listing_print.cfm
Disallow /brokers/listing_print_pretty.cfm

Other Records

Field Value
crawl-delay 10

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

barkrowler*

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

*

Rule Path
Disallow /400.cfm
Disallow /403.cfm
Disallow /404.cfm
Disallow /410.cfm
Disallow /404handler.cfm
Disallow /error_standard.cfm
Disallow /error_extreme.cfm
Disallow /newsletter_signup.cfm
Disallow /prolink.cfm
Disallow /searchresults.cfm
Disallow /sitewideerrorhandler.cfm
Disallow /_errors/
Disallow /_FCKe/
Disallow /_mates/
Disallow /_popyachts/
Disallow /_yatco/
Disallow /_admin/
Disallow /boats-for-sale/listing_print.cfm
Disallow /boats-for-sale/listing_print_pretty.cfm
Disallow /brokers/listing_print.cfm
Disallow /brokers/listing_print_pretty.cfm
Back to top

Other Records

Field Value
sitemap https://marinesource.com/sitemap.xml
Back to top

Warnings

  • 2 invalid lines.
Back to top