ndcs.org.uk
robots.txt
Robots Exclusion Standard data for ndcs.org.uk
Resource Scan
Scan Details
| Site Domain | ndcs.org.uk |
| Base Domain | ndcs.org.uk |
| Scan Status | Ok |
| Last Scan | 2025-07-19T16:19:24+00:00 |
| Next Scan | 2025-08-18T16:19:24+00:00 |
Last Scan
| Scanned | 2025-07-19T16:19:24+00:00 |
| URL | https://www.ndcs.org.uk/robots.txt |
| Domain IPs | 34.147.168.147 |
| Response IP | 34.147.168.147 |
| Found | Yes |
| Hash | 35955397d4a659533141f8c4b190a78c56f4c1cecb66cf8cdc32ea91d507d135 |
| SimHash | 504c1606df96 |
Groups
addthis.com
ahrefs
aiohttp
alexa site audit
alipesnewsbot
amazonbot
amazon-route53-health-check-service
apachebench
appdynamics
applebot
archivebot
archive-it
aspiegelbot
assetnote
axios
azure-logic-apps
baiduspider
barkrowler
bingbot
blexbot
blp_bbot
bluechipbacklinks
buck
bytespider
catchbot
ccbot
check_http
claudebot
cloudflare-prefetch
cludo.com bot
colly
contentkingapp
cookiebot
copperegg
crawler4j
csnibot
curebot
curl
cyotekwebcopy
daum
datadog agent
dataforseobot
detectify
dotbot
dow jones searchbot
duckduckbot
facebookexternalhit
faraday
feedburner
feedfetcher-google
feedonomics
feroxbuster
fess
funnelback
fuzz faster u fool
gachecker
ghost inspector
gptbot
grapeshot
gobuster
gocolly
googlestackdrivermonitoring
go-http-client
go-resty
guzzlehttp
headlesschrome
heritrix
hokifybot
honolulu-bot
httrack
hubspot crawler
icc-crawler
imperva
ioncrawl
jooble
kauaibot
kinza
liebaofast
linabot
linespider
linguee
linkchecker
linkedinbot
linkupbot
linuxgeturl
lmy47v
macoutlook
magnet.me
magus bot
mail.ru_bot
mauibot
mb2345browser
megaindex
microsoft office
microsoft outlook
microsoft word
micromessenger
mindbreeze-crawler
mirrorweb.com
mj12bot
monitoring-plugins
monsidobot
mqqbrowser
msnbot
msoffice
mtrobot
nagios-plugins
nettle
neevabot
newscred
newspaper
node-fetch
nuclei
nukescan
okhttp
oncrawl
orbbot
pagefreezer
panscient.com
petalbot
pingdom.com
pinterestbot
piplbot
python-requests
qwantify
re-re studio
riddler
rocketvalidator
rogerbot
rustbot
safeassign
scrapy
screaming frog
seobilitybot
search365bot
searchblox
searchmetricsbot
searchunify
seekport
semanticscholarbot
semrushbot
seokicks
seoscanners
serpstatbot
sessioncam
seznambot
site24x7
siteauditbot
siteimprove
siteimprove
sitelockspider
sitesucker
skyperoom
slackbot
slurp
sogou web spider
special_archiver
spiderling
statuscake
swiftbot
synack
turnitin
trendictionbot
trendkite-akashic-crawler
ucbrowser
uptime
uptimerobot
usasearch
ut-dorkbot
weborama-fetcher
whitehat security
widenwebhookclient
wget
wtwbot
www.loc.gov
xenu link sleuth
vagabondo
velenpublicwebcrawler
yeti
veracode security scan
yandexbot
yandeximages
yisouspider
y!j
zabbix
zoominfobot
zoomspider
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /core/ |
| Disallow | /profiles/ |
| Disallow | /README.txt |
| Disallow | /web.config |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips |
| Disallow | /node/add/ |
| Disallow | /search |
| Disallow | /search/ |
| Disallow | /signup |
| Disallow | /sign-up-for-our-emails |
| Disallow | /user/register/ |
| Disallow | /user/password/ |
| Disallow | /user/login/ |
| Disallow | /user/logout/ |
| Disallow | /drupal10/deprecation_status/errors/ |
| Disallow | /drupal9/deprecation_status/errors/ |
| Disallow | /index.php/admin/ |
| Disallow | /index.php/comment/reply/ |
| Disallow | /index.php/filter/tips |
| Disallow | /index.php/node/add/ |
| Disallow | /index.php/search/ |
| Disallow | /index.php/user/password/ |
| Disallow | /index.php/user/register/ |
| Disallow | /index.php/user/login/ |
| Disallow | /index.php/user/logout/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.ndcs.org.uk/sitemap.xml |
Warnings
- 1 invalid line.