rateyourmusic.com
robots.txt
Robots Exclusion Standard data for rateyourmusic.com
Resource Scan
Scan Details
| Site Domain | rateyourmusic.com |
| Base Domain | rateyourmusic.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-05-27T12:15:42+00:00 |
| Next Scan | 2024-06-03T12:15:42+00:00 |
Last Successful Scan
| Scanned | 2024-05-19T12:15:00+00:00 |
| URL | https://rateyourmusic.com/robots.txt |
| Domain IPs | 104.22.28.74, 104.22.29.74, 172.67.11.193 |
| Response IP | 104.22.28.74 |
| Found | Yes |
| Hash | 19de8c22fbe75bf6c80ed38a148c4f60e13a31ef2812ef917071b948ba9e1916 |
| SimHash | 7014db5ce27d |
Groups
googlebot-image
| Rule | Path |
|---|---|
| Disallow | /account |
| Disallow | /redir/ |
| Disallow | /admin/ |
| Disallow | /artist/profile_ac |
| Disallow | /compare |
| Disallow | /lists/new_item |
| Disallow | /favorited |
| Disallow | /find_similar_users |
| Disallow | /releases/ac |
| Disallow | /user_albums_export/ |
| Disallow | /contributed/ |
| Disallow | /stats/ |
| Disallow | /messages/ |
| Disallow | /recommend/ |
| Disallow | /recs/ |
| Disallow | /rdescriptor/set |
| Disallow | /rgenre/set |
| Disallow | /contact |
| Disallow | /search |
| Disallow | /artist/profile_details |
| Disallow | /images/upload |
| Disallow | /images/all |
| Disallow | /rymzilla |
googlebot
| Rule | Path |
|---|---|
| Disallow | /account |
| Disallow | /redir/ |
| Disallow | /admin/ |
| Disallow | /artist/profile_ac |
| Disallow | /compare |
| Disallow | /lists/new_item |
| Disallow | /favorited |
| Disallow | /find_similar_users |
| Disallow | /releases/ac |
| Disallow | /user_albums_export/ |
| Disallow | /contributed/ |
| Disallow | /stats/ |
| Disallow | /messages/ |
| Disallow | /recommend/ |
| Disallow | /recs/ |
| Disallow | /rdescriptor/set |
| Disallow | /rgenre/set |
| Disallow | /contact |
| Disallow | /search |
| Disallow | /artist/profile_details |
| Disallow | /images/upload |
| Disallow | /images/all |
| Disallow | /rymzilla |
twitterbot
| Rule | Path |
|---|---|
| Disallow | /account |
| Disallow | /redir/ |
| Disallow | /admin/ |
| Disallow | /artist/profile_ac |
| Disallow | /compare |
| Disallow | /lists/new_item |
| Disallow | /favorited |
| Disallow | /find_similar_users |
| Disallow | /releases/ac |
| Disallow | /user_albums_export/ |
| Disallow | /contributed/ |
| Disallow | /stats/ |
| Disallow | /messages/ |
| Disallow | /recommend/ |
| Disallow | /recs/ |
| Disallow | /rdescriptor/set |
| Disallow | /rgenre/set |
| Disallow | /contact |
| Disallow | /search |
| Disallow | /artist/profile_details |
| Disallow | /images/upload |
| Disallow | /images/all |
| Disallow | /rymzilla |
msnbot
| Rule | Path |
|---|---|
| Disallow | /account |
| Disallow | /redir/ |
| Disallow | /admin/ |
| Disallow | /artist/profile_ac |
| Disallow | /compare |
| Disallow | /lists/new_item |
| Disallow | /favorited |
| Disallow | /find_similar_users |
| Disallow | /releases/ac |
| Disallow | /user_albums_export/ |
| Disallow | /contributed/ |
| Disallow | /stats/ |
| Disallow | /messages/ |
| Disallow | /recommend/ |
| Disallow | /recs/ |
| Disallow | /rdescriptor/set |
| Disallow | /rgenre/set |
| Disallow | /contact |
| Disallow | /search |
| Disallow | /artist/profile_details |
| Disallow | /images/upload |
| Disallow | /images/all |
| Disallow | /rymzilla |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
slurp
| Rule | Path |
|---|---|
| Disallow | /account |
| Disallow | /redir/ |
| Disallow | /admin/ |
| Disallow | /artist/profile_ac |
| Disallow | /compare |
| Disallow | /lists/new_item |
| Disallow | /favorited |
| Disallow | /find_similar_users |
| Disallow | /releases/ac |
| Disallow | /user_albums_export/ |
| Disallow | /contributed/ |
| Disallow | /stats/ |
| Disallow | /messages/ |
| Disallow | /recommend/ |
| Disallow | /recs/ |
| Disallow | /rdescriptor/set |
| Disallow | /rgenre/set |
| Disallow | /contact |
| Disallow | /search |
| Disallow | /artist/profile_details |
| Disallow | /images/upload |
| Disallow | /images/all |
| Disallow | /rymzilla |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 12 |
*
| Rule | Path |
|---|---|
| Disallow | / |
Comments