directory9.net
robots.txt
Robots Exclusion Standard data for directory9.net
Resource Scan
Scan Details
| Site Domain | directory9.net |
| Base Domain | directory9.net |
| Scan Status | Ok |
| Last Scan | 2025-05-04T01:38:40+00:00 |
| Next Scan | 2025-06-03T01:38:40+00:00 |
Last Scan
| Scanned | 2025-05-04T01:38:40+00:00 |
| URL | https://directory9.net/robots.txt |
| Domain IPs | 154.38.165.218 |
| Response IP | 154.38.165.218 |
| Found | Yes |
| Hash | 9ab7a8296881c554993c0dbed8e59671ff4e3723053cad12817896a8ed256ccf |
| SimHash | e106145ec217 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /custom/domain* |
| Disallow | /custom/log |
| Disallow | /custom/profile |
| Disallow | /custom/tmp |
| Disallow | /layout |
| Disallow | /includes |
| Disallow | /search/map/summary |
| Allow | /custom/domain*/sitemap* |
| Allow | /custom/domain*/tmp*.js |
| Allow | /custom/domain*/tmp*.css |
| Allow | /custom/domain*/theme* |
| Allow | /custom/domain*/image_files/* |
| Allow | /custom/domain*/content_files/* |
| Allow | / |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://directory9.net/custom/domain_1/sitemap/index.xml |