froghire.ai
robots.txt

Robots Exclusion Standard data for froghire.ai

Resource Scan

Scan Details

Site Domain froghire.ai
Base Domain froghire.ai
Scan Status Ok
Last Scan2025-12-12T22:08:31+00:00
Next Scan 2026-01-11T22:08:31+00:00

Last Scan

Scanned2025-12-12T22:08:31+00:00
URL https://froghire.ai/robots.txt
Redirect https://www.froghire.ai/robots.txt
Redirect Domain www.froghire.ai
Redirect Base froghire.ai
Domain IPs 104.21.60.216, 172.67.201.204, 2606:4700:3032::ac43:c9cc, 2606:4700:3034::6815:3cd8
Redirect IPs 104.21.60.216, 172.67.201.204, 2606:4700:3032::ac43:c9cc, 2606:4700:3034::6815:3cd8
Response IP 172.67.201.204
Found Yes
Hash e54170bde92982d30649921d10534925757bcf746dcc2c66e1caf51427ec4164
SimHash 60100950c411

Groups

*

Rule Path
Allow /
Disallow /cdn-cgi/
Disallow /company?search*
Disallow /company/*

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.froghire.ai/sitemap-index.xml

Comments

  • AI and LLM Crawling