Log In Sign Up

teamflexo.com
robots.txt

Robots Exclusion Standard data for teamflexo.com

Resource Scan

Scan Details

Site Domain teamflexo.com
Base Domain teamflexo.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-03-25T07:23:38+00:00
Next Scan 2024-06-23T07:23:38+00:00

Last Successful Scan

Scanned2023-02-05T18:13:32+00:00
URL https://teamflexo.com/robots.txt
Domain IPs 104.26.4.93, 104.26.5.93, 172.67.73.22, 2606:4700:20::681a:45d, 2606:4700:20::681a:55d, 2606:4700:20::ac43:4916
Response IP 172.67.73.22
Found Yes
Hash c3515f284c74dee2cdb7c19c5276843161226f2e3f8d279aff22b960d75f2365
SimHash 805ec93a2976

Groups

*

Rule Path
Allow /

rogerbot

Rule Path
Disallow

dotbot

Rule Path
Disallow

google

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ccbot/2.0

Rule Path
Disallow /
Back to top

Comments

  • Was disallowing /wp-*. Rankings declined. Allowing as of 9/30/17 to see if they increase.
  • Moz's Bot. We want them to crawl us. This entry may be needed.
  • Adding for Moz
  • Allow Googlebot to crawl. No entry really needed but being safe.
  • AhrefsBot - Commercial. Huge crawling activity. Useless. Blocked 12/17/17. Last crawl 12/20/17
  • MJ12bot - Commercial. Large crawling activity. Useless. Blocked 2/17/2018.
  • CCBot/2.0 - Common Crawl Non-Profit for researchers. Usefulness limited. Blocked 2/17/2018.
Back to top