illustrationsource.com
robots.txt
Robots Exclusion Standard data for illustrationsource.com
Resource Scan
Scan Details
Site Domain | illustrationsource.com |
Base Domain | illustrationsource.com |
Scan Status | Ok |
Last Scan | 2024-06-17T07:16:19+00:00 |
Next Scan | 2024-07-17T07:16:19+00:00 |
Last Scan
Scanned | 2024-06-17T07:16:19+00:00 |
URL | https://www.illustrationsource.com/robots.txt |
Domain IPs | 66.175.215.172 |
Response IP | 66.175.215.172 |
Found | Yes |
Hash | 871ce837a3a65a9af13abc02ef0e0623c8004114d2a78f72e3a2e4655fae5a9d |
SimHash | ae3a5159ccf7 |
Groups
mozilla/5.0 (tweetmemebot/4.0; +http://datasift.com/bot.html) gecko/20100101 firefox/31.0
Rule | Path |
---|---|
Disallow | / |
*
Rule | Path |
---|---|
Disallow | /media/ |
Disallow | /login/ |
Disallow | /account/ |
Disallow | */checkout/* |
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
msnbot
Rule | Path |
---|---|
Disallow | /media |
Disallow | /login |
Disallow | /account |
Disallow | */checkout/* |
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
msnbot-newsblogs/1.1 (+http://search.msn.com/msnbot.htm)
Rule | Path |
---|---|
Disallow | /media |
Disallow | /login |
Disallow | /account |
Disallow | */checkout/* |
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
Rule | Path |
---|---|
Disallow | /media |
Disallow | /login |
Disallow | /account |
Disallow | */checkout/* |
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
Warnings
- 2 invalid lines.
- `host-load` is not a known field.
Comments