caulfeild.co.uk
robots.txt
Robots Exclusion Standard data for caulfeild.co.uk
Resource Scan
Scan Details
| Site Domain | caulfeild.co.uk |
| Base Domain | caulfeild.co.uk |
| Scan Status | Ok |
| Last Scan | 2025-08-03T23:08:48+00:00 |
| Next Scan | 2025-09-02T23:08:48+00:00 |
Last Scan
| Scanned | 2025-08-03T23:08:48+00:00 |
| URL | https://caulfeild.co.uk/robots.txt |
| Domain IPs | 2a02:4780:15:9ec0:47b6:f2f:1ccf:a953, 2a02:4780:38:fc6f:2cde:59d9:8ddd:4a37, 84.32.84.131, 84.32.84.146 |
| Response IP | 93.127.187.8 |
| Found | Yes |
| Hash | 54fb0c43af4505fe4ea389fb7cc2dd94e0cfcba0f64ecdbe64b0beb16cbaa6cd |
| SimHash | 47864a010682 |
Groups
admantx
ai2bot
adsbot
aisearchbot
ahrefsbot
ai2bot-dolma
aliyunsecbot
amazonbot
andibot
antbot
applebot
aspiegelbot
awario
blexbot
barkrowler
brightbot
bytespider
ccbot
censysinspect
chatgpt-user
claude-searchbot
claude-user
claude-web
claudebot
cotoyogi
crawlspace
dataforseobot
datenbank crawler
devin
diffbot
dotbot
duckassistbot
echobot bot
echoboxbot
expanse
facebookbot
factset_spyderbot
firecrawlagent
foregenix
friendlycrawler
gptbot
gemini-deep-research
go-http-client
google-cloudvertexbot
google-extended
googleagent-mariner
googleother
grapeshot
honolulu-bot
icc-crawler
isscyberriskcrawler
imagesiftbot
ioncrawl
java
kangaroo bot
linguee
mj12bot
megaindex.ru
meta-externalagent
meta-externalfetcher
mistralai-user
mycentralaiscraperbot
novaact
oai-searchbot
operator
pangubot
panscient
perplexity-user
perplexitybot
petalbot
phindbot
poseidon research crawler
qualifiedbot
quillbot
sbintuitionsbot
seokicks
scrapy
seekportbot
semrushbot
sidetrade indexer bot
sitekiosk
summalybot
thinkbot
tiktokspider
timpibot
tinytestbot
turnitin
velenpublicwebcrawler
wardbot
webzio-extended
xovibot
yandexadditional
yisouspider
youbot
zoominfobot
aihitbot
aiohttp
anthropic-ai
bedrockbot
cohere-ai
cohere-training-data-crawler
facebookexternalhit
fidget-spinner-bot
iaskspider
img2dataset
internet-measurement
linabot
meta-externalagent
meta-externalfetcher
netestate
omgili
panscient
phxbot
proximic
python-requests
quillbot.com
wpbot
serpstatbot
test-bot
wp_is_mobile
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /admin |
| Disallow | /manager |
| Disallow | /editor |
| Disallow | /account |
| Disallow | /tree/Caulfeild/ancestors |
| Disallow | /tree/Caulfeild/calendar |
| Disallow | /tree/Caulfeild/compact |
| Disallow | /tree/Caulfeild/descendants |
| Disallow | /tree/Caulfeild/family-book |
| Disallow | /tree/Caulfeild/family-list |
| Disallow | /tree/Caulfeild/fan-chart |
| Disallow | /tree/Caulfeild/hourglass |
| Disallow | /tree/Caulfeild/individual-list |
| Disallow | /tree/Caulfeild/lifespans |
| Disallow | /tree/Caulfeild/pedigree |
| Disallow | /tree/Caulfeild/report |
| Disallow | /tree/Caulfeild/timeline |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://caulfeild.co.uk/sitemap.xml |
Comments