Log In Sign Up

iq.product-hub.app
robots.txt

Robots Exclusion Standard data for iq.product-hub.app

Resource Scan

Scan Details

Site Domain iq.product-hub.app
Base Domain product-hub.app
Scan Status Ok
Last Scan2025-11-19T14:30:48+00:00
Next Scan 2025-12-19T14:30:48+00:00

Last Scan

Scanned2025-11-19T14:30:48+00:00
URL https://iq.product-hub.app/robots.txt
Domain IPs 104.21.76.85, 172.67.191.151, 2606:4700:3032::6815:4c55, 2606:4700:3037::ac43:bf97
Response IP 172.67.191.151
Found Yes
Hash be3055ac9f822986f564d8d2ad5444cae1777a0321d992bc95eff57f3a738c05
SimHash 6c060090e5f5

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /_next/
Disallow /cancelled
Disallow /auth

Other Records

Field Value
crawl-delay 1

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

whatsapp

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://iq.product-hub.app/sitemap.xml
Back to top

Comments

  • Specific rules for major search engines
  • Block access to sensitive areas
  • Allow social media crawlers
  • Sitemap location
Back to top