avr-getbot.com
robots.txt

Robots Exclusion Standard data for avr-getbot.com

Resource Scan

Scan Details

Site Domain avr-getbot.com
Base Domain avr-getbot.com
Scan Status Ok
Last Scan2025-10-17T17:04:28+00:00
Next Scan 2025-10-24T17:04:28+00:00

Last Scan

Scanned2025-10-17T17:04:28+00:00
URL https://avr-getbot.com/robots.txt
Redirect https://avr-global.com/robots.txt
Redirect Domain avr-global.com
Redirect Base avr-global.com
Domain IPs 67.215.9.138
Redirect IPs 67.215.9.138
Response IP 67.215.9.138
Found Yes
Hash ee01a21b86e5310ecd7f379d5fb25432e3801f17b322d368c9ac68cf35e5395e
SimHash 6690490060a3

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /my-account
Disallow /login
Allow /about
Allow /aerospace
Allow /mro-aerospace
Allow /energy
Allow /medical
Allow /general-manufacturing
Allow /abrasive-finishing-products
Allow /visual-inspection-products
Allow /operating-programming-software
Allow /customer-service

chatgpt-user

Rule Path
Allow /

gptbot

Rule Path
Allow /

google-extended

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

ccbot

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

ai-crawler

Rule Path
Allow /

bingbot

Rule Path
Allow /

msnbot

Rule Path
Allow /

Comments

  • General rule for all crawlers
  • Key pages allowed
  • Explicitly allow AI and major search crawlers