surfline.com
robots.txt

Robots Exclusion Standard data for surfline.com

Resource Scan

Scan Details

Site Domain surfline.com
Base Domain surfline.com
Scan Status Ok
Last Scan2024-11-15T17:20:51+00:00
Next Scan 2024-11-22T17:20:51+00:00

Last Scan

Scanned2024-11-15T17:20:51+00:00
URL https://surfline.com/robots.txt
Redirect https://www.surfline.com/robots.txt
Redirect Domain www.surfline.com
Redirect Base surfline.com
Domain IPs 13.57.94.154, 54.177.28.187
Redirect IPs 104.16.244.29, 104.16.245.29, 2606:4700::6810:f41d, 2606:4700::6810:f51d
Response IP 104.16.245.29
Found Yes
Hash d177cc52c03eb6442e337881a0c6840ea91695886d252335dc17d7a375cfd231
SimHash d11241da8e25

Groups

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yrspider

Rule Path
Disallow /

camontspider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

purebot

Rule Path
Disallow /

pikimal

Rule Path
Disallow /

pik-a-part

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

israbot

Rule Path
Disallow /

orthogaffe

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

spyfu

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linkdex

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

discobot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

bender

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 500

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 500

*

Rule Path
Disallow /account_shared
Disallow /admin
Disallow /advertiser
Disallow /alpha
Disallow /api
Disallow /audio
Disallow /Audio
Disallow /badh
Disallow /bamp_v02
Disallow /bamp_v03
Disallow /bapv
Disallow /barl
Disallow /barl_v02
Disallow /basp
Disallow /bavp02
Disallow /bavs_v02
Disallow /beta
Disallow /blank
Disallow /bookmarks
Disallow /build
Disallow /bw
Disallow /bwcam
Disallow /campaign
Disallow /calendar
Disallow /cfide
Disallow /comments
Disallow /create-account
Disallow /dashboard
Disallow /dtd
Disallow /dvd
Disallow /errors
Disallow /external_assets
Disallow /externalcam
Disallow /eyeblaster
Disallow /ezprints
Disallow /faces
Disallow /flash
Disallow /flashcam
Disallow /flashcam2
Disallow /flashcam2ub
Disallow /flashcam3
Disallow /flashcam4
Disallow /flashcam5
Disallow /footer
Disallow /for_review
Disallow /forums
Disallow /ftp
Disallow /functions
Disallow /globalnav
Disallow /globalnav2
Disallow /globalnav2OLD
Disallow /globalnav3
Disallow /globalnav4
Disallow /hdcam
Disallow /hdcam2
Disallow /hdcam3
Disallow /hdml
Disallow /hlsec
Disallow /holiday
Disallow /home2
Disallow /home3
Disallow /homeocean
Disallow /lite
Disallow /lite2
Disallow /lite2go
Disallow /lite3
Disallow /lite5
Disallow /live_site_media
Disallow /liza_blog
Disallow /memcached
Disallow /modules
Disallow /music
Disallow /mutinymedia
Disallow /partners
Disallow /podcast
Disallow /popups
Disallow /ppt
Disallow /ppv
Disallow /promobox
Disallow /prune_files
Disallow /redirect
Disallow /registration
Disallow /reports2
Disallow /rr
Disallow /rsession
Disallow /rules
Disallow /sched
Disallow /search/
Disallow /sign-in
Disallow /slwl
Disallow /sms
Disallow /soquel
Disallow /sound
Disallow /store_email
Disallow /sub_msg
Disallow /surfline20
Disallow /surgient
Disallow /swf
Disallow /syndication
Disallow /teaser_images
Disallow /test
Disallow /tinymce
Disallow /util
Disallow /video-old
Disallow /vimation
Disallow /vodcast
Disallow /w3c
Disallow /wavetraks
Disallow /webpoll
Disallow /widgets2
Disallow /wml
Disallow /wx
Disallow /freestreams
Disallow /fusion

Other Records

Field Value
sitemap https://www.surfline.com/sitemaps/index.xml

Comments

  • Robots.txt file for https://www.surfline.com
  • Wikipedia work bots: