foswiki.org
robots.txt
Robots Exclusion Standard data for foswiki.org
Resource Scan
Scan Details
| Site Domain | foswiki.org |
| Base Domain | foswiki.org |
| Scan Status | Ok |
| Last Scan | 2025-07-26T22:17:49+00:00 |
| Next Scan | 2025-08-25T22:17:49+00:00 |
Last Scan
| Scanned | 2025-07-26T22:17:49+00:00 |
| URL | https://foswiki.org/robots.txt |
| Domain IPs | 2.56.98.129, 2a03:4000:3e:14d::1 |
| Response IP | 2.56.98.129 |
| Found | Yes |
| Hash | 0275de49d88357983190bcc09c24ff002ce6d240f1be14d5814404ddd356bab1 |
| SimHash | 6846d11b8cc7 |
Groups
amazonbot
anthropic-ai
applebot
baiduspider
bytespider
chatgpt-user
claudebot
claude-web
diffbot
facebookbot
google-extended
gptbot
imagesiftbot
omgili
omgilibot
perplexitybot
turnitinbot
yandex
youbot
petalbot
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /bin/attach |
| Disallow | /bin/changes |
| Disallow | /bin/configure |
| Disallow | /bin/edit |
| Disallow | /bin/geturl |
| Disallow | /bin/installpasswd |
| Disallow | /bin/login |
| Disallow | /bin/logon |
| Disallow | /bin/logos |
| Disallow | /bin/mailnotify |
| Disallow | /bin/manage |
| Disallow | /bin/oops |
| Disallow | /bin/passwd |
| Disallow | /bin/preview |
| Disallow | /bin/rdiff |
| Disallow | /bin/rdiffauth |
| Disallow | /bin/register |
| Disallow | /bin/rename |
| Disallow | /bin/resetpasswd |
| Disallow | /bin/rest |
| Disallow | /bin/save |
| Disallow | /bin/savemulti |
| Disallow | /bin/search |
| Disallow | /bin/setlib.cfg |
| Disallow | /bin/statistics |
| Disallow | /bin/testenv |
| Disallow | /bin/upload |
| Disallow | /bin/viewauth |
| Disallow | /bin/viewfile |
| Disallow | /list/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |
-ai
_ai
ai.
ai-
ai_
ai=
agentic
agentql
ai agent
ai article writer
ai chat
ai content detector
ai detection
ai dungeon
ai journalist
ai search
ai seo crawler
ai training
ai web
ai writer
ai2
aibot
aihitbot
aimatrix
aisearchbot
aitraining
alexa
alice yandex
aligenie
alpha ai
alphaai
amazon
amelia
anderspinkbot
andibot
anonymous ai
anthropic
anypicker
anyword
apple
aria ai
aria browse
articoolo
ask ai
autoglm
automated writer
automl
autonomous rag
awariorssbot
awariosmartbot
aws trainium
azure
babyagi
bardbot
basic rag
bedrock
brave leo
brightbot
bytedance
bytespider
carynai
catboost
cc-crawler
ccbot
chai
charstar ai
chatbot
chatglm
chatuser
chinchilla
claude
clearscope
cognitive ai
cohere
common crawl
commoncrawl
content harmony
content king
content optimizer
content samurai
contentatscale
contentbot
contentedge
contentshake
conversion ai
copilot
copyai
copymatic
copyscape
coreweave
corrective rag
cotoyogi
crab
crawlq ai
crawlspace
crew ai
crewai
crushon ai
dall-e
darkbard
dataforai
dataforseobot
dataprovider
datenbank crawler
deepai
deepl
deepmind
deepseek
diffbot
doubao ai
duckassistbot
echobox
elixir
facebookbot
facebookexternalhit
factset
falcon
fire-1
firecrawl
flux
flyriver
frase ai
friendlycrawler
gato
gemini
gemma
genai
genspark
ghostwriter
gigachat
glm
goose
gpt
grammarly
grendizer
grok
gt bot
gtbot
hemingway editor
hugging face
hybrid search rag
hypotenuse ai
iaskspider
icc-crawler
imagegen
imagesiftbot
img2dataset
imgproxy
ink editor
inkforall
intelliseek
inferkit
isscyberriskcrawler
janitor ai
jasperai
jenni ai
kafkai
kaggle
kangaroo
keyword density ai
knowledge
komobot
langchain
le chat
lensa
lightpanda
llama
llm
local rag agent
magpie-crawler
manus
marketmuse
meltwater
meta.ai
meta-ai
meta-external
meta ai
metaai
metatagbot
midjourney
minimax
mistral
model-training
monica
mycentralaiscraperbot
narrative
neevabot
neural text
neuralseo
ninjaai
nova act
novaact
oai-searchbot
oai searchbot
oasis
omgili
open ai
open deep research
open perflexity
openai
openbot
opentext ai
operator
outwrite
page analyzer ai
pangubot
panscient
paperlibot
paraphraser.io
peer39_crawler
perplexity
petalbot
phind
piplbot
prowritingaid
proximic
puppeteer
qualified
quark
quillbot
qwen
rag agent
rag is
rag pipeline
rag search
rag with
raptor
redis ai rag
robotspider
rytr
saplingai
sbintuitionsbot
scala
scalenut
scrapegraph
scraper
scrapy
scriptbook
seekr
seo content machine
seo robot
semrushbot
sentibot
sidetrade
simplified ai
sitefinity
skydancer
slickwrite
smartbot
smartscrape
sonic
sora
spin rewriter
spinbot
stability
stablediffusionbot
sudowrite
summalybot
super agent
surfer ai
text blaze
textcortex
thinkbot
tiktokspider
timpibot
together ai
traefik
turnitinbot
velenpublicwebcrawler
venus chub ai
vidnami ai
vision rag
webscrape
webtext
webzio
wechat
whisper
wordai
wordtune
wormsgtp
wpbot
writecream
writerzen
writescope
writesonic
xai
xbot
yaml
yandexadditional
youbot
zero
zhipu
zhuque ai
zimm
| Rule | Path |
|---|---|
| Disallow | / |
Warnings
- 1 invalid line.
- `disallowaitraining` is not a known field.
Comments