astronomytoday.com
robots.txt
Robots Exclusion Standard data for astronomytoday.com
Resource Scan
Scan Details
| Site Domain | astronomytoday.com |
| Base Domain | astronomytoday.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Couldn't connect to server. |
| Last Scan | 2025-11-06T05:44:47+00:00 |
| Next Scan | 2026-02-04T05:44:47+00:00 |
Last Successful Scan
| Scanned | 2024-01-16T03:13:59+00:00 |
| URL | http://astronomytoday.com/robots.txt |
| Redirect | http://www.astronomytoday.com/robots.txt |
| Redirect Domain | www.astronomytoday.com |
| Redirect Base | astronomytoday.com |
| Domain IPs | 192.251.238.4 |
| Redirect IPs | 192.251.238.4 |
| Response IP | 192.251.238.4 |
| Found | Yes |
| Hash | c70bb61cdab17ba867630486df6974b606ffdb0e5cb2918de1481133d057efcf |
| SimHash | ec36f042cb90 |
Groups
teoma
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /newsletter/ |
| Disallow | /contact.html |
| Disallow | /refer/ |
| Disallow | /forum/ |
| Disallow | /blog/wp-includes/ |
| Disallow | /blog/wp-content/ |
| Disallow | /blog/wp-admin/ |
| Disallow | /blog/xmlrpc.php |
| Disallow | /feed/ |
| Disallow | /feed |
| Disallow | /trackback/ |
| Disallow | /rss/ |
| Disallow | /comments/feed/ |
| Disallow | /page/ |
| Disallow | /date/ |
| Disallow | /comments/ |
slurp
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /newsletter/ |
| Disallow | /contact.html |
| Disallow | /refer/ |
| Disallow | /forum/ |
| Disallow | /blog/wp-includes/ |
| Disallow | /blog/wp-content/ |
| Disallow | /blog/wp-admin/ |
| Disallow | /blog/xmlrpc.php |
| Disallow | /feed/ |
| Disallow | /feed |
| Disallow | /trackback/ |
| Disallow | /rss/ |
| Disallow | /comments/feed/ |
| Disallow | /page/ |
| Disallow | /date/ |
| Disallow | /comments/ |
msnbot
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /newsletter/ |
| Disallow | /contact.html |
| Disallow | /refer/ |
| Disallow | /forum/ |
| Disallow | /blog/wp-includes/ |
| Disallow | /blog/wp-content/ |
| Disallow | /blog/wp-admin/ |
| Disallow | /blog/xmlrpc.php |
| Disallow | /blog/wp-admin/ |
| Disallow | /feed/ |
| Disallow | /feed |
| Disallow | /trackback/ |
| Disallow | /rss/ |
| Disallow | /comments/feed/ |
| Disallow | /page/ |
| Disallow | /date/ |
| Disallow | /comments/ |
googlebot
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /newsletter/ |
| Disallow | /forum/search.php |
| Disallow | /forum/indexg.php |
| Disallow | /forum/faq.php |
| Disallow | /forum/profile.php |
| Disallow | /forum/posting.php |
| Disallow | /forum/privmsg.php |
| Disallow | /forum/login.php |
| Disallow | /forum/viewonline.php |
| Disallow | /forum/groupcp.php |
| Disallow | /refer/ |
| Disallow | /refer/form.php |
| Disallow | /blog/wp-includes/ |
| Disallow | /blog/wp-content/ |
| Disallow | /blog/wp-admin/ |
| Disallow | /blog/xmlrpc.php |
| Disallow | /blog/wp-admin/ |
| Disallow | /feed/ |
| Disallow | /feed |
| Disallow | /trackback/ |
| Disallow | /rss/ |
| Disallow | /comments/feed/ |
| Disallow | /date/ |
| Disallow | /comments/ |