alfiegoodrich.com
robots.txt
Robots Exclusion Standard data for alfiegoodrich.com
Resource Scan
Scan Details
Site Domain | alfiegoodrich.com |
Base Domain | alfiegoodrich.com |
Scan Status | Ok |
Last Scan | 2024-06-23T14:47:30+00:00 |
Next Scan | 2024-07-23T14:47:30+00:00 |
Last Scan
Scanned | 2024-06-23T14:47:30+00:00 |
URL | https://alfiegoodrich.com/robots.txt |
Domain IPs | 198.57.184.124 |
Response IP | 198.57.184.124 |
Found | Yes |
Hash | b0f9a0419dfac60f24f9cb345cf00f7bcaa64531302ba432b1dfc68e626ef4da |
SimHash | eeb65159ccf7 |
Groups
mozilla/5.0 (tweetmemebot/4.0; +http://datasift.com/bot.html) gecko/20100101 firefox/31.0
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0 (tweetmemebot/4.0; +http://datasift.com/bot.html) gecko/20100101 firefox/31.0
Rule | Path |
---|---|
Disallow | / |
*
Rule | Path |
---|---|
Disallow | /media/ |
Disallow | /search/ |
Disallow | /basie/ |
Disallow | /login/ |
Disallow | /publications/article/*/email/ |
Disallow | /account/ |
Disallow | */checkout/* |
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
googlebot
Rule | Path |
---|---|
Disallow | /media/ |
Disallow | /search/ |
Disallow | /basie/ |
Disallow | /login/ |
Disallow | /publications/article/*/email/ |
Disallow | /account/ |
Disallow | */checkout/* |
bingbot
Rule | Path |
---|---|
Disallow | /media/ |
Disallow | /search/ |
Disallow | /basie/ |
Disallow | /login/ |
Disallow | /publications/article/*/email/ |
Disallow | /account/ |
Disallow | */checkout/* |
rogerbot
Rule | Path |
---|---|
Disallow | /media/ |
Disallow | /search/ |
Disallow | /basie/ |
Disallow | /login/ |
Disallow | /publications/article/*/email/ |
Disallow | /account/ |
Disallow | */checkout/* |
msnbot
Rule | Path |
---|---|
Disallow | /media |
Disallow | /search |
Disallow | /basie |
Disallow | /login |
Disallow | /account |
Disallow | */checkout/* |
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
msnbot-newsblogs/1.1 (+http://search.msn.com/msnbot.htm)
Rule | Path |
---|---|
Disallow | /media |
Disallow | /search |
Disallow | /basie |
Disallow | /login |
Disallow | /account |
Disallow | */checkout/* |
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
Rule | Path |
---|---|
Disallow | /media |
Disallow | /search |
Disallow | /basie |
Disallow | /login |
Disallow | /account |
Disallow | */checkout/* |
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
Warnings
- 3 invalid lines.
- `host-load` is not a known field.
Comments