honestdocs.id
robots.txt

Robots Exclusion Standard data for honestdocs.id

Resource Scan

Scan Details

Site Domain honestdocs.id
Base Domain honestdocs.id
Scan Status Ok
Last Scan2025-07-14T06:26:30+00:00
Next Scan 2025-07-28T06:26:30+00:00

Last Scan

Scanned2025-07-14T06:26:30+00:00
URL https://honestdocs.id/robots.txt
Redirect https://www.honestdocs.id/robots.txt
Redirect Domain www.honestdocs.id
Redirect Base honestdocs.id
Domain IPs 104.21.21.53, 172.67.196.124, 2606:4700:3033::6815:1535, 2606:4700:3034::ac43:c47c
Redirect IPs 104.21.21.53, 172.67.196.124, 2606:4700:3033::6815:1535, 2606:4700:3034::ac43:c47c
Response IP 104.21.21.53
Found Yes
Hash 1c6335647c42130d6c2dd3a2792f5888b37927a6b60e2858a89e70f72dec16be
SimHash 475fd410c590

Groups

*

Rule Path
Disallow /universal-search*

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

uptimerobot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.honestdocs.id/sitemaps/sitemap.xml.gz