friendseat.com
robots.txt

Robots Exclusion Standard data for friendseat.com

Resource Scan

Scan Details

Site Domain friendseat.com
Base Domain friendseat.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-08-03T14:56:59+00:00
Next Scan 2024-11-01T14:56:59+00:00

Last Successful Scan

Scanned2023-01-12T03:45:14+00:00
URL https://friendseat.com/robots.txt
Domain IPs 104.18.26.48, 104.18.27.48, 2606:4700::6812:1a30, 2606:4700::6812:1b30
Response IP 104.18.27.48
Found Yes
Hash fa21b3fa89cf91857e8895bfd2a4fcc28368d6b0a0b49aec947f087885ee76d0
SimHash 22125bb1c353

Groups

mediapartners-google

Rule Path
Disallow

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 1

adsbot-google

Rule Path
Disallow

Other Records

Field Value
crawl-delay 1

googlebot-image

Rule Path
Disallow

Other Records

Field Value
crawl-delay 2

googlebot-mobile

Rule Path
Disallow

Other Records

Field Value
crawl-delay 2

msnbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 2

bingbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 3

slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 2

yahoo! slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 3

*

Rule Path
Disallow /include/
Disallow /theme/

sogou spider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

lexxebot

Rule Path
Disallow /

hopperbot

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

http://bimeon.com (crawler_bootv10.1.5_link)

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

purebot

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

discobot

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

speedy

Rule Path
Disallow /

exabot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-news

Rule Path
Disallow /

baiduspider-favo

Rule Path
Disallow /

baiduspider-cpro

Rule Path
Disallow /

baiduspider-ads

Rule Path
Disallow /

baidu

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

hopperbot/1.1

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

Comments

  • Time Delay

Warnings

  • 4 invalid lines.