willows-grove.com
robots.txt
Robots Exclusion Standard data for willows-grove.com
Resource Scan
Scan Details
| Site Domain | willows-grove.com |
| Base Domain | willows-grove.com |
| Scan Status | Ok |
| Last Scan | 2025-10-30T21:15:01+00:00 |
| Next Scan | 2025-11-29T21:15:01+00:00 |
Last Scan
| Scanned | 2025-10-30T21:15:01+00:00 |
| URL | https://willows-grove.com/robots.txt |
| Domain IPs | 173.236.141.46 |
| Response IP | 173.236.141.46 |
| Found | Yes |
| Hash | f0221b058b7d891837ccd7bfad0bd003610705ecea9eaf01e390fec77790f6f9 |
| SimHash | 7015d111a290 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /user/ |
| Disallow | /masterlist/ |
| Disallow | /sublist/ |
| Disallow | /character/ |
| Disallow | /myo/ |
| Disallow | /gallery/ |
| Disallow | /submissions/ |
| Disallow | /claims/ |
| Disallow | /reports/ |
| Disallow | /trades/ |
| Disallow | /comment/ |
| Disallow | /images/ |
| Disallow | /files/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
amazonbot
anthropic-ai
applebot-extended
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
diffbot
facebookbot
facebookexternalhit
friendlycrawler
google-extended
gptbot
icc-crawler
imagesiftbot
img2dataset
meta-externalagent
oai-searchbot
omgili
omgilibot
perplexitybot
petalbot
scrapy
timpibot
velenpublicwebcrawler
youbot
semrushbot
| Rule | Path |
|---|---|
| Disallow | / |