myfreeblack.com
robots.txt

Robots Exclusion Standard data for myfreeblack.com

Resource Scan

Scan Details

Site Domain myfreeblack.com
Base Domain myfreeblack.com
Scan Status Ok
Last Scan2024-09-16T20:53:57+00:00
Next Scan 2024-10-16T20:53:57+00:00

Last Scan

Scanned2024-09-16T20:53:57+00:00
URL https://myfreeblack.com/robots.txt
Redirect https://www.myfreeblack.com/robots.txt
Redirect Domain www.myfreeblack.com
Redirect Base myfreeblack.com
Domain IPs 104.21.234.170, 104.21.234.171, 2606:4700:3038::6815:eaaa, 2606:4700:3038::6815:eaab
Redirect IPs 104.21.234.170, 104.21.234.171, 2606:4700:3038::6815:eaaa, 2606:4700:3038::6815:eaab
Response IP 104.21.234.171
Found Yes
Hash 822afda2a52f38dad80fed6fdb81f74717a90141ef9b76cf702998f91c64ca2a
SimHash 511cc1404d78

Groups

*

Rule Path
Allow /

*

Rule Path
Disallow /livegirl/
Disallow /livecam/
Disallow /camsex/
Disallow /cam/
Disallow /live/
Disallow /cams/

yandexvideoparser

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

special_archiver

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

special_archiver

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

proximic

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

Warnings

  • 2 invalid lines.