italpress.com
robots.txt
Robots Exclusion Standard data for italpress.com
Resource Scan
Scan Details
Site Domain | italpress.com |
Base Domain | italpress.com |
Scan Status | Ok |
Last Scan | 2024-09-20T20:33:18+00:00 |
Next Scan | 2024-09-27T20:33:18+00:00 |
Last Scan
Scanned | 2024-09-20T20:33:18+00:00 |
URL | https://italpress.com/robots.txt |
Domain IPs | 104.26.2.221, 104.26.3.221, 172.67.70.146, 2606:4700:20::681a:2dd, 2606:4700:20::681a:3dd, 2606:4700:20::ac43:4692 |
Response IP | 104.26.2.221 |
Found | Yes |
Hash | b98f6ad375e7e624481566bf086050e28eecbdb4401e241850fc25b3cd0443b3 |
SimHash | 3093d053d2b7 |
Groups
dlvr.it/1.0
Rule | Path |
---|---|
Allow | /rss |
Allow | /rss/ |
Allow | /atom |
Allow | /atom/ |
Other Records
Field | Value |
---|---|
crawl-delay | 60 |
mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0 (compatible; yahoo! slurp china; http://misc.yahoo.com.cn/help.html)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+becomebot/3.0;++http://www.become.com/site_owners.html)
Rule | Path |
---|---|
Disallow | / |
Warnings
- 12 invalid lines.
Comments