wiki.astro.com
robots.txt
Robots Exclusion Standard data for wiki.astro.com
Resource Scan
Scan Details
| Site Domain | wiki.astro.com |
| Base Domain | astro.com |
| Scan Status | Ok |
| Last Scan | 2024-10-31T06:12:54+00:00 |
| Next Scan | 2024-11-30T06:12:54+00:00 |
Last Scan
| Scanned | 2024-10-31T06:12:54+00:00 |
| URL | http://wiki.astro.com/robots.txt |
| Redirect | http://www.astro.com/robots.txt |
| Redirect Domain | www.astro.com |
| Redirect Base | astro.com |
| Domain IPs | 192.53.104.242 |
| Redirect IPs | 192.53.104.242 |
| Response IP | 192.53.104.242 |
| Found | Yes |
| Hash | 3ac80f29c1a24ca12caf94086cb6669a8cc8174e97a859da41d8db3e3cc137b1 |
| SimHash | 201a59f4e518 |
Groups
openindexspider
| Rule | Path |
|---|---|
| Disallow | /*.pdf$ |
| Disallow | /cgi/chart.cgi |
| Disallow | /astro-databank/Category%3A |
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /apache |
| Disallow | /adb/ |
| Disallow | /design |
| Disallow | /hfot |
| Disallow | /h/ |
| Disallow | /htm/ |
| Disallow | /license |
| Disallow | /sstat |
| Disallow | /stats |
| Disallow | /tmpd |
| Disallow | /trans |
| Disallow | /webp |
| Disallow | /cgi/sub.cgi |
| Disallow | /cgi/bugrep.cgi |
| Disallow | /cgi/ovi.cgi |
| Disallow | /cgi/ade.cgi |
| Disallow | /astrowiki/de/Benutzer |
| Disallow | /astrowiki/de/Diskussion |
| Disallow | /astrowiki/en/User |
| Disallow | /astrowiki/en/Talk |
| Disallow | /astro-databank/Private%3A |
| Disallow | /wiki/*/index.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2.0 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.astro.com/sitemap_index.xml |
Warnings
- 2 invalid lines.