hosting.uk
robots.txt
Robots Exclusion Standard data for hosting.uk
Resource Scan
Scan Details
| Site Domain | hosting.uk |
| Base Domain | hosting.uk |
| Scan Status | Ok |
| Last Scan | 2024-06-08T14:50:48+00:00 |
| Next Scan | 2024-07-08T14:50:48+00:00 |
Last Scan
| Scanned | 2024-06-08T14:50:48+00:00 |
| URL | https://hosting.uk/robots.txt |
| Redirect | https://www.hosting.co.uk/robots.txt |
| Redirect Domain | www.hosting.co.uk |
| Redirect Base | hosting.co.uk |
| Domain IPs | 104.21.39.236, 172.67.171.215, 2606:4700:3030::6815:27ec, 2606:4700:3036::ac43:abd7 |
| Redirect IPs | 104.21.88.84, 172.67.174.81, 2606:4700:3036::ac43:ae51, 2606:4700:3037::6815:5854 |
| Response IP | 104.21.88.84 |
| Found | Yes |
| Hash | 365a2a562453145fcd1d5067db7165df28a2ed2a3d9b181a1388c8a0193f9113 |
| SimHash | 76db12858f66 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /clientarea/ |
| Disallow | /free-trial |
| Disallow | /refer/samples_tests/ |
| Disallow | /wp/ |
| Disallow | /wp-admin/ |
| Disallow | /maintenance.html |
| Disallow | /tsync |
b2w/0.1
backdoorbot
backdoorbot/1.0
baiduspider
becomebot
blowfish
blowfish/1.0
bookmark search tool
botalot
browsershots
bruinbot
btbot
builtbottough
bullseye
bullseye/1.0
bunnyslippers
| Rule | Path |
|---|---|
| Disallow | / |
cfetch
cheesebot
cherrypicker
cherrypickerelite/1.0
cherrypickerse/1.0
coolbot
copyrightcheck
cosmos
cowbot
crescent
crescent internet toolpak http ole control v.1.0
cydralspider
| Rule | Path |
|---|---|
| Disallow | / |
gaisbot
generic
georgios
getright
getright/4.2
gigabot
goforit
gonzo
gridbot
grub
grub-client
| Rule | Path |
|---|---|
| Disallow | / |
harvest
harvest/1.5
haste
henrythemiragorobot
heritrix
hloader
hoowwwer
httplib
httrack
humanlinks
| Rule | Path |
|---|---|
| Disallow | / |
lachesis
larbin
lexibot
libweb/clshttp
libwww
linkextractorpro
linko
linkscan
linkscan/8.1a unix
linkwalker
lnspiderguy
localcombot
looksmart
lwp-trivial
lwp-trivial/1.34
| Rule | Path |
|---|---|
| Disallow | / |
mata hari
mentormate spider
microsoft url control
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
miixpc
miixpc/4.2
mirago
mister pix
mmcrawler
moget
moget*
moget/2.1
molbsy
moni
mozilla/4.0 (compatible; bullseye; windows 95)
mozilla/4.0 (compatible; netcraft web server survey)
msiecrawler
muscat ferret
myengines-bot
| Rule | Path |
|---|---|
| Disallow | / |
naverbot
net attache
netants
netmechanic
netresearchserver
nicerspro
nimblecrawler
npbot
npt
nutch
| Rule | Path |
|---|---|
| Disallow | / |
objectssearch
offline explorer
openbot
openfind
openfind data gathere
oracle ultra search
| Rule | Path |
|---|---|
| Disallow | / |
penthesilea*
perman
phpdig*
propowerbot
propowerbot/2.14
prowebwalker
psbot
python-urllib
| Rule | Path |
|---|---|
| Disallow | / |
radiation retriever
radiation retriever 1.1
repomonkey
repomonkey bait & tackle/v1.01
repomonkey*
rma
rpt-httpclient
| Rule | Path |
|---|---|
| Disallow | / |
sbider
searchpreview
searchspider
shim-crawler
sitecheck.internetseer.com
sitesnagger
sna
sohu-search
spankbot
spanner
speedy
spider_ monkey
spiderjack
spinne
stalker
steeler
superget
suzuran
szukacz
szukacz/1.4
| Rule | Path |
|---|---|
| Disallow | / |
tamu_cs_irl_crawler
teleport
telesoft
the intraformant
thenomad
thesubot
thumbshots-de-bot
tocrawl/urldispatcher
true_robot
true_robot/1.0
turingos
tutorgig
twiceler
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.hosting.co.uk/sitemap_index.xml |