fao.org
robots.txt
Robots Exclusion Standard data for fao.org
Resource Scan
Scan Details
| Site Domain | fao.org |
| Base Domain | fao.org |
| Scan Status | Ok |
| Last Scan | 2024-09-20T04:21:29+00:00 |
| Next Scan | 2024-10-20T04:21:29+00:00 |
Last Scan
| Scanned | 2024-09-20T04:21:29+00:00 |
| URL | https://fao.org/robots.txt |
| Redirect | https://www.fao.org/robots.txt |
| Redirect Domain | www.fao.org |
| Redirect Base | fao.org |
| Domain IPs | 104.18.22.5, 104.18.23.5 |
| Redirect IPs | 104.18.10.41, 104.18.11.41, 2606:4700::6812:a29, 2606:4700::6812:b29 |
| Response IP | 104.18.11.41 |
| Found | Yes |
| Hash | e56d01e7df0c923e8290bb472fb000bcf413c0816c7163a4fbbfb3ba9b54b7ec |
| SimHash | 700f1d5afdf5 |
Groups
*
| Rule | Path | Comment |
|---|---|---|
| Disallow | /index.php | - |
| Disallow | /t3lib/ | Nothing to see here |
| Disallow | /typo3/ | Nothing to see here |
| Disallow | /*?id=* | Disable non-realurl - re-instated 10/Oct/2013 |
| Disallow | /*%26type%3D98 | - specified in Google webmaster tools for the Google exclusion - re-instated 10/Oct/2013 |
| Disallow | /fileadmin/user_upload/PermRep/ | don't need to be indexed (23/06/2014 - nw) |
| Disallow | /fileadmin/user_upload/en/ | don't need to be indexed (11/07/2014 - permreps) |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.fao.org/newsroom/sitemap/sitemap.gz |
| sitemap | https://www.fao.org/in-action/ectad/sitemap/sitemap.gz |
| sitemap | https://www.fao.org/americas/sitemap/sitemap.gz |
| sitemap | https://www.fao.org/director-general/sitemap/sitemap-index.xml |
| sitemap | https://www.fao.org/europe/sitemap/sitemap.gz |
Comments