broker-robinhood.com
robots.txt

Robots Exclusion Standard data for broker-robinhood.com

Resource Scan

Scan Details

Site Domain broker-robinhood.com
Base Domain broker-robinhood.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-10-05T17:05:54+00:00
Next Scan 2025-01-03T17:05:54+00:00

Last Successful Scan

Scanned2022-03-31T07:40:16+00:00
URL https://broker-robinhood.com/robots.txt
Response IP 104.21.42.71
Found Yes
Hash 0521e768bd8a2528bcb253a6b5b49dc23d9c3bf54afa7483d8cda921a3d7be2b
SimHash 0a19444a2d20

Groups

mediapartners-google

Rule Path
Disallow

criteobot/0.1

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

cis455crawler

Rule Path
Disallow /

netseer

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mail.ru_bot*

Rule Path
Disallow /

paradigmcrawler*

Rule Path
Disallow /

dotbot*

Rule Path
Disallow /

clickagy*

Rule Path
Disallow /

pu_in*

Rule Path
Disallow /

crazywebcrawler*

Rule Path
Disallow /

fast*

Rule Path
Disallow /

bender

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

npbot

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

open*

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

scooter*

Rule Path
Disallow /

websearch*

Rule Path
Disallow /

zyborg*

Rule Path
Disallow /

wget*

Rule Path
Disallow /

turnitinbot*

Rule Path
Disallow /

slysearch*

Rule Path
Disallow /

seekbot*

Rule Path
Disallow /

netresearchserver*

Rule Path
Disallow /

seekbot*

Rule Path
Disallow /

mozdex*

Rule Path
Disallow /

link*

Rule Path
Disallow /

larbin*

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

charlotte

Rule Path
Disallow /

discobot

Rule Path
Disallow /

*

Rule Path
Disallow /p/
Disallow /d/
Disallow /_/misc/
Disallow /_/search/
Disallow /_/cite.htm
Disallow /_/gr.aspx
Disallow /_/hp/Controls/printer-friendly.aspx
Disallow /unscramble/

Warnings

  • 2 invalid lines.