agentgrabber.com
robots.txt
Robots Exclusion Standard data for agentgrabber.com
Resource Scan
Scan Details
| Site Domain | agentgrabber.com |
| Base Domain | agentgrabber.com |
| Scan Status | Ok |
| Last Scan | 2025-08-14T10:45:48+00:00 |
| Next Scan | 2025-09-13T10:45:48+00:00 |
Last Scan
| Scanned | 2025-08-14T10:45:48+00:00 |
| URL | https://agentgrabber.com/robots.txt |
| Domain IPs | 104.26.2.4, 104.26.3.4, 172.67.70.196, 2606:4700:20::681a:204, 2606:4700:20::681a:304, 2606:4700:20::ac43:46c4 |
| Response IP | 172.67.70.196 |
| Found | Yes |
| Hash | 8f6dc2adf8be9b7ec9442b11d18a362f8895351bfc5ba487f5d730663c4dd8d5 |
| SimHash | 22667d43e991 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /search/ |
| Disallow | /author/ |
| Disallow | /users/ |
| Disallow | */trackback |
| Disallow | */feed |
| Disallow | */rss |
| Disallow | */embed |
| Disallow | /dostavka/ |
| Disallow | /oplata/ |
| Disallow | /garantiya/ |
| Disallow | /terms-of-use/ |
| Disallow | /cart-2/ |
| Disallow | /en/cart-3/ |
| Disallow | /checkout-2/ |
| Disallow | /en/checkout-3/ |
| Disallow | /en/dostavka/ |
| Disallow | /en/oplata/ |
| Disallow | /en/garantiya/ |
| Disallow | /en/terms-of-use/ |
| Disallow | /wp-admin/ |
| Disallow | /wp-includes/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://agentgrabber.com/sitemap_index.xml |
Warnings
- `host` is not a known field.
Comments