Log In Sign Up

documentgeneratorhub.online
robots.txt

Robots Exclusion Standard data for documentgeneratorhub.online

Resource Scan

Scan Details

Site Domain documentgeneratorhub.online
Base Domain documentgeneratorhub.online
Scan Status Ok
Last Scan2025-12-03T11:25:40+00:00
Next Scan 2025-12-10T11:25:40+00:00

Last Scan

Scanned2025-12-03T11:25:40+00:00
URL https://documentgeneratorhub.online/robots.txt
Domain IPs 185.199.108.153, 185.199.109.153, 185.199.110.153, 185.199.111.153, 2606:50c0:8000::153, 2606:50c0:8001::153, 2606:50c0:8002::153, 2606:50c0:8003::153
Response IP 185.199.110.153
Found Yes
Hash 7348cad3042df829078dd430625cfa308ad94194d2d33ee8931374a1daeb570e
SimHash 490f4873f535

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /
Allow /generators/
Allow /assets/
Allow /about.html
Allow /contact.html
Allow /faq.html
Allow /privacy-policy.html
Allow /terms-of-service.html
Allow /disclaimer.html

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://documentgeneratorhub.online/sitemap.xml
Back to top

Comments

  • Sitemap
  • Allow all search engines to crawl the site
  • Crawl-delay for general bots
  • Disallow admin or private areas (if any)
  • Disallow: /admin/
  • Disallow: /private/
  • Allow all document generators
  • Allow all assets
  • Allow all main pages
Back to top