alobariatrics.com
robots.txt

Robots Exclusion Standard data for alobariatrics.com

Resource Scan

Scan Details

Site Domain alobariatrics.com
Base Domain alobariatrics.com
Scan Status Ok
Last Scan2025-11-07T04:42:41+00:00
Next Scan 2025-12-07T04:42:41+00:00

Last Scan

Scanned2025-11-07T04:42:41+00:00
URL https://alobariatrics.com/robots.txt
Domain IPs 104.251.216.190
Response IP 104.251.216.190
Found Yes
Hash 71bbe5c66d6a955d4934f25fd0ed6da9ba163f7fb6e158f09619b8e03f497826
SimHash 205c1951c577

Groups

*

Rule Path
Disallow /wp-json/
Disallow /?rest_route=

oai-searchbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

perplexitybot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

firecrawlagent

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

andibot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

exabot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

phindbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

coherebot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.alobariatrics.com/sitemap_index.xml

Comments

  • -----------------------------------------------
  • Global Disallow Rules for All Bots
  • -----------------------------------------------
  • -----------------------------------------------
  • Sitemap Declaration
  • -----------------------------------------------
  • -----------------------------------------------
  • Allowed AI Bots for Live Browsing (Not Training)
  • -----------------------------------------------
  • -----------------------------------------------
  • SEO Search Engine Bots
  • -----------------------------------------------
  • -----------------------------------------------
  • Disallowed AI Crawlers
  • -----------------------------------------------