cgstud.io
robots.txt

Robots Exclusion Standard data for cgstud.io

Resource Scan

Scan Details

Site Domain cgstud.io
Base Domain cgstud.io
Scan Status Ok
Last Scan2024-11-05T03:08:25+00:00
Next Scan 2024-12-05T03:08:25+00:00

Last Scan

Scanned2024-11-05T03:08:25+00:00
URL https://cgstud.io/robots.txt
Redirect https://free3d.com:443/robots.txt
Redirect Domain free3d.com
Redirect Base free3d.com
Domain IPs 13.248.178.125, 76.223.35.209
Redirect IPs 13.33.88.101, 13.33.88.25, 13.33.88.33, 13.33.88.85
Response IP 13.33.88.85
Found Yes
Hash 5ca897116691806775611fa3db237212d4b24a55ab88fcc7bcac942f0f52a0f3
SimHash a8b55976e7d2

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /tmp/
Disallow /include/
Disallow /aws-tf3dm/
Disallow /affiliate/
Disallow /panel/
Disallow /scraper/
Disallow /webfonts/

gptbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

Comments

  • Disallow public gptbot
  • Disallow meta agent
  • Disallow CCBot