agrilife.org
robots.txt
Robots Exclusion Standard data for agrilife.org
Resource Scan
Scan Details
| Site Domain | agrilife.org |
| Base Domain | agrilife.org |
| Scan Status | Ok |
| Last Scan | 2025-11-08T12:10:33+00:00 |
| Next Scan | 2025-11-22T12:10:33+00:00 |
Last Scan
| Scanned | 2025-11-08T12:10:33+00:00 |
| URL | https://agrilife.org/robots.txt |
| Domain IPs | 141.193.213.10, 141.193.213.11 |
| Response IP | 141.193.213.11 |
| Found | Yes |
| Hash | 1c13c103229b6cbed3d83b693e66dad4b84782b96a14d3557a55b91e89838290 |
| SimHash | 4a5ef844cf92 |
Groups
bingbot
| Rule | Path |
|---|---|
| Disallow | /wp-login.php |
| Disallow | /wp-admin/ |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 600 | Set delay to 10 minutes to reduce load |
yandex
| Rule | Path |
|---|---|
| Disallow | /wp-login.php |
| Disallow | /wp-admin/ |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 600 | Set delay to 10 minutes to manage server impact |
amazonbot
| Rule | Path |
|---|---|
| Disallow | /wp-login.php |
| Disallow | /wp-admin/ |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 600 | Set delay to 10 minutes |
ahrefsbot
| Rule | Path |
|---|---|
| Disallow | /wp-login.php |
| Disallow | /wp-admin/ |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 600 | Set delay to 10 minutes |
claudebot
| Rule | Path |
|---|---|
| Disallow | /wp-login.php |
| Disallow | /wp-admin/ |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 600 | Set delay to 10 minutes |
facebookexternalhit
| Rule | Path |
|---|---|
| Disallow | /wp-login.php |
| Disallow | /wp-admin/ |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 600 | Set delay to 10 minutes |
seekportbot
| Rule | Path |
|---|---|
| Disallow | /wp-login.php |
| Disallow | /wp-admin/ |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 600 | Set delay to 10 minutes |
feedburner
| Rule | Path |
|---|---|
| Disallow | /wp-login.php |
| Disallow | /wp-admin/ |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 600 | Set delay to 10 minutes |
*
| Rule | Path |
|---|---|
| Disallow | /wp-login.php |
| Disallow | /wp-admin/ |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 600 | General delay for other bots set to 10 minutes |
Other Records
| Field | Value |
|---|---|
| sitemap | https://meat.tamu.edu/sitemap_index.xml |
| sitemap | https://elp.tamu.edu/sitemap_index.xml |
| sitemap | https://agrilife.org/bell4h/sitemap_index.xml |
| sitemap | https://travis-tx.tamu.edu/sitemap_index.xml |
| sitemap | https://dallas-tx.tamu.edu/sitemap_index.xml |
| sitemap | https://naturetourism.tamu.edu/sitemap_index.xml |
| sitemap | https://agrilife.org/brazoria4h/sitemap_index.xml |
| sitemap | https://agrilife.org/qualitycounts/sitemap_index.xml |
| sitemap | https://feralhogs.tamu.edu/sitemap_index.xml |
| sitemap | https://bbq.tamu.edu/sitemap_index.xml |
| sitemap | https://citybugs.tamu.edu/sitemap_index.xml |
| sitemap | https://texas4hcenter.tamu.edu/sitemap_index.xml |
Comments