Log In Sign Up

procurementresource.com
robots.txt

Robots Exclusion Standard data for procurementresource.com

Resource Scan

Scan Details

Site Domain procurementresource.com
Base Domain procurementresource.com
Scan Status Ok
Last Scan2026-02-27T19:37:08+00:00
Next Scan 2026-03-29T19:37:08+00:00

Last Scan

Scanned2026-02-27T19:37:08+00:00
URL https://procurementresource.com/robots.txt
Redirect https://www.procurementresource.com/robots.txt
Redirect Domain www.procurementresource.com
Redirect Base procurementresource.com
Domain IPs 104.21.20.148, 172.67.193.36, 2606:4700:3034::ac43:c124, 2606:4700:3037::6815:1494
Redirect IPs 104.21.20.148, 172.67.193.36, 2606:4700:3034::ac43:c124, 2606:4700:3037::6815:1494
Response IP 172.67.193.36
Found Yes
Hash b645955c2d45ab304a947963239736fab48feec98abf011b9582431034e27456
SimHash 15753fd164d9

Groups

*

Rule Path
Allow /
Disallow /_next/
Disallow /_error
Disallow /404
Disallow /500
Disallow /admin/
Disallow /api/
Disallow /dashboard/
Disallow /login/
Disallow /signin/
Disallow /register/
Disallow /account/
Disallow /cart/
Disallow /checkout/
Disallow /payment/
Disallow /order/
Disallow /thank-you/
Disallow /purchase/
Disallow /*?utm_
Disallow /*?ref=
Disallow /*?source=
Disallow /*?campaign=
Disallow /*?filter=
Disallow /*?sort=
Disallow /*?s=
Disallow /*?q=
Disallow /*?print=
Disallow /search/
Disallow /tag/
Disallow /tags/
Disallow /feed/
Disallow /tmp/
Disallow /cache/
Disallow /private/

ahrefsbot

Rule Path
Allow /

semrushbot

Rule Path
Allow /

dataforseobot

Rule Path
Allow /

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

claudebot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

google-extended

Rule Path
Allow /

applebot-extended

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.procurementresource.com/sitemap.xml
Back to top

Comments

  • robots.txt for procurementresource.com
  • Last Updated: February 2026
  • Platform: Next.js
  • ================================
  • GENERAL RULES - ALL BOTS
  • ================================
  • Block Next.js internal files
  • Block admin & backend
  • Block checkout & payment flow
  • Block URL parameters (duplicate content)
  • Block thin/low value pages
  • ================================
  • SEO TOOLS - ALLOW
  • ================================
  • ================================
  • AI CRAWLERS - ALLOW
  • ================================
  • ================================
  • BLOCK USELESS BOTS
  • ================================
  • ================================
  • ALLOW GOOD BOTS FULLY
  • ================================
  • ================================
  • SITEMAP
  • ================================
Back to top