varbes.com
robots.txt
Robots Exclusion Standard data for varbes.com
Resource Scan
Scan Details
| Site Domain | varbes.com |
| Base Domain | varbes.com |
| Scan Status | Ok |
| Last Scan | 2024-09-21T03:37:23+00:00 |
| Next Scan | 2024-09-28T03:37:23+00:00 |
Last Scan
| Scanned | 2024-09-21T03:37:23+00:00 |
| URL | https://varbes.com/robots.txt |
| Redirect | https://www.varbes.com/robots.txt |
| Redirect Domain | www.varbes.com |
| Redirect Base | varbes.com |
| Domain IPs | 198.49.23.145 |
| Redirect IPs | 2600:9000:21f8:1c00:13:448e:1100:93a1, 2600:9000:21f8:2800:13:448e:1100:93a1, 2600:9000:21f8:5000:13:448e:1100:93a1, 2600:9000:21f8:8800:13:448e:1100:93a1, 2600:9000:21f8:be00:13:448e:1100:93a1, 2600:9000:21f8:ec00:13:448e:1100:93a1, 2600:9000:21f8:f200:13:448e:1100:93a1, 2600:9000:21f8:fc00:13:448e:1100:93a1, 52.84.45.127, 52.84.45.42, 52.84.45.45, 52.84.45.47 |
| Response IP | 108.156.22.21 |
| Found | Yes |
| Hash | a3e14fbfbd3db338be7e1018438206d7a4022833de6701adac8f708a32e10cdb |
| SimHash | d31ec9c1c090 |
Groups
ccbot
gptbot
chatgpt-user
mj12bot
turnitinbot
petalbot
dotbot
ahrefsbot
imagesiftbot
amazonbot
google-extended
bytespider
facebookbot
brightbot
crystalsemanticsbot
dataforseobot
semrushbot
anthropic-ai
cohere-ai
claude-web
omgili
omgilibot
perplexitybot
applebot-extended
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /click/ |
| Disallow | /search/ |
Comments