trawlbens.id
robots.txt
Robots Exclusion Standard data for trawlbens.id
Resource Scan
Scan Details
| Site Domain | trawlbens.id |
| Base Domain | trawlbens.id |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Couldn't connect to server. |
| Last Scan | 2025-09-26T14:41:54+00:00 |
| Next Scan | 2025-11-25T14:41:54+00:00 |
Last Successful Scan
| Scanned | 2025-07-06T14:32:07+00:00 |
| URL | https://trawlbens.id/robots.txt |
| Redirect | https://troben.id/robots.txt |
| Redirect Domain | troben.id |
| Redirect Base | troben.id |
| Domain IPs | 104.21.59.88, 172.67.220.164, 2606:4700:3032::6815:3b58, 2606:4700:3034::ac43:dca4 |
| Redirect IPs | 104.21.6.71, 172.67.134.150, 2606:4700:3031::6815:647, 2606:4700:3033::ac43:8696 |
| Response IP | 104.21.6.71 |
| Found | Yes |
| Hash | 5896500e3ef67469451d890edb9286a0f091c3cc987645f830b831e8a06d59f5 |
| SimHash | 6905dec507b0 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /cek-tarif?* |
| Disallow | /karir |
| Disallow | /karir?* |
| Disallow | /karir/ |
| Disallow | /karir/* |
| Disallow | /artikel?category=* |
| Disallow | /mitra?utm_source=* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://troben.id/sitemap.xml |
| sitemap | https://troben.id/sitemap-page.xml |
| sitemap | https://troben.id/sitemap-layanan.xml |
| sitemap | https://troben.id/sitemap-cek-tarif.xml |
| sitemap | https://troben.id/sitemap-content.xml |
| sitemap | https://troben.id/sitemap-karir.xml |
| sitemap | https://troben.id/sitemap-promo.xml |
| sitemap | https://troben.id/sitemap-mitra.xml |
| sitemap | https://troben.id/sitemap-artikel.xml |
| sitemap | https://troben.id/blog/sitemap_index.xml |
| sitemap | https://troben.id/blog/post-sitemap1.xml |
| sitemap | https://troben.id/blog/post-sitemap2.xml |
| sitemap | https://troben.id/blog/page-sitemap.xml |
Warnings
- `host` is not a known field.
Comments