ukmodelshops.co.uk
robots.txt
Robots Exclusion Standard data for ukmodelshops.co.uk
Resource Scan
Scan Details
| Site Domain | ukmodelshops.co.uk |
| Base Domain | ukmodelshops.co.uk |
| Scan Status | Ok |
| Last Scan | 2024-09-29T15:18:43+00:00 |
| Next Scan | 2024-10-29T15:18:43+00:00 |
Last Scan
| Scanned | 2024-09-29T15:18:43+00:00 |
| URL | https://ukmodelshops.co.uk/robots.txt |
| Domain IPs | 91.207.50.20 |
| Response IP | 91.207.50.20 |
| Found | Yes |
| Hash | a4f6090b4460d17700e80c5ef710679eb289220716445010fcf3a71962dbea33 |
| SimHash | f3d75353ccb2 |
Groups
claudebot
magellan
sirdatabot
peer39_crawler
bytespider
ttd-content
ips-agent
googlebot-image
queryseekerspider
magpie-crawler
komodiabot
mauibot
criteobot
lcc
maxpointcrawler/nutch-1.17
ltx71
gpcsupbot
testbot
panscient.com
domainstatsbot
tennysoncrawler
ccbot
velenpublicwebcrawler
sputnikbot
aspiegelbot
admantx
ioncrawl
aihitbot
serpstatbot
proximic
clickagy intelligence bot v2
seekportbot
seekport crawler
qwantify
obot
blexbot
sogou spider
linkpadbot
dispatch/0.14.0-snapshot
nimbostratus-bot/v1.3.2
ioncrawl
zoominfobot
adsbot/3.1
garlikcrawler
buck
buck/2.2
screaming frog seo spider
istellabot
changedetection
tweetmemebot
moatbot
seznambot
sentibot
scrapy
petalbot
linespider
crawler4j
mail.ru_bot
mail.ru
mail.ru_bot
megaindex.ru
buck
coccocbot-image
coccocbot-web
daum
sidetrade indexer bot
yandexbot
yandeximages
yandex
yandexaccessibilitybot
semrushbot
semrushbot-sa
semrushbot-ba
semrushbot-si
semrushbot-swa
semrushbot-ct
semrushbot-bm
spbot
ahrefsbot
wesee
vagabondo
bilbo
dotbot
huaweisymantecspider
baiduspider
baiduspider-video
baiduspider-image
baidu
sogou spider
youdaobot
twiceler
yanga worldsearch bot
naverbot
yeti
yisouspider
mojeekbot
ias-or
ias-ir
ias-va
axios
embedly
wellknownbot
orbbot
df bot
pinterestbot
pandalytics
ninjbot
pubmatic crawler bot
the knowledge ai
internet security survey bot
iab atq team adform robot
grapeshotcrawler
wfuzz
builtwith
stormcrawler
crowdtanglebot
pandalytics
ltx71 - (http://ltx71.com/)
ultimate_sitemap_parser/0.5
neevabot
netcraftsurveyagent
dataforseobot
seochecker
seokicks
seo scanner
seocompany
geedobot
uipbot
scraperbot
leikibot
matchorysearch
vsusearchspider
cincraw/1.0
entferbot
netpeakcheckerbot
crawlson
semanticbot
morealtbot
trendsmapresolver
adstxtcrawlertp
httrack
dataprovider
ahc
swspider
bitsightbot
embarcadero uri client/1.0
lightspeedsystemscrawler
zombiebot
vuhuvbot
birdcrawlerbot
internetmeasurement
webcorp
mastodon
monsidobot
turnitin
paperlibot
nicecrawler
netestate ne crawler
brightedge crawler
uribot
youbot
quantcastbot
smtbot
haosouspider
spark
webwikibot
wikido
diffbot
snappreviewbot
insytfulbot
impendoom-bot
senutobot
seolyt
yadirectfetcher
echoboxbot
pagepeeker
| Rule | Path |
|---|---|
| Disallow | / |
facebookexternalhit
nutch
node-fetch
bitlybot
ubicrawler
sproose
sbider
bloodhound/nutch-0.9
wwwster/1.4
mqbot/nutch-0.9-dev
botrighthere
webzip
larbin
larbin_2.6.3
libwww
zyborg
download ninja
heritrix
sitecheck.internetseer.com
microsoft.url.control
b2w/0.1
copernic
psbot
netmechanic
npbot
url_spider_pro
cherrypicker
emailcollector
emailsiphon
webbandit
emailwolf
extractorpro
copyrightcheck
crescent
sitesnagger
prowebwalker
cheesebot
lnspiderguy
alexibot
teleport
teleportpro
miixpc
telesoft
grub-client
metauri
genieo
omgili
omgilibot
gigablastopensource
ntentbot
doc
zao
zealbot
msiecrawler
fetch
linko
website quester
webzip
moget
ichiro
webzip/4.0
webstripper
websauger
webcopier
webreaper
netants
mister pix
webauto
thenomad
www-collector-e
rma
libweb/clshttp
asterias
httplib
turingos
spanner
infonavirobot
harvest/1.5
bullseye/1.0
bullseye
crescent internet toolpak http ole control v.1.0
cherrypickerse/1.0
cherrypickerelite/1.0
webbandit/3.50
nicerspro
microsoft url control - 5.01.4511
dittospyder
foobot
spankbot
botalot
lwp-trivial/1.34
lwp-trivial
bunnyslippers
microsoft url control - 6.00.8169
linkwalker
cosmos
moget
hloader
humanlinks
linkextractorpro
offline explorer
mata hari
lexibot
web image collector
the intraformant
true_robot/1.0
true_robot
blowfish/1.0
jennybot
miixpc/4.2
urly warning
builtbottough
propowerbot/2.14
backdoorbot/1.0
tocrawl/urldispatcher
suzuran
tighttwatbot
vci webviewer vci webviewer win32
vci
szukacz/1.4
openfind data gatherer
openfind
xenu's link sleuth 1.1c
xenu
zeus
repomonkey bait & tackle/v1.01
repomonkey
microsoft url control
openbot
url control
zeus link scout
zeus 32297 webster pro v2.9 win32
webster pro
erocrawler
linkscan/8.1a unix
keyword density/0.9
kenjin spider
iron33/1.0.2
bookmark search tool
getright/4.2
fairad client
gaisbot
aqua_products
radiation retriever 1.1
flaming attackbot
jyxobot
mj12bot
ezooms
amznkassocbot
exabot
scoutjet
yahooseeker
speedy spider
gigabot
innovantagebot
grapeshot
wikido
newspaper/0.2.8
facebot
twitterbot
bidswitchbot
discordbot
surdotlybot
clark-crawler2
startmebot
woorankreview
webprosbot
cloudinary
indeedbot
ev-crawler
bubing
inetdex-bot/1.5
trendictionbot
apis-google
viberbot
experiancrawluk
serendeputybot
mediatoolkitbot
femtosearchbot
bnf.fr_bot
t3versionsbot
zaldamosearchbot
spideraf-bot
gnowitnewsbot
certytags
archivebot
archive.org_bot
riddler
amazonbot
gozlebot
facebookbot
gptbot
chatgpt-user
anthropic-ai
google-extended
my-tiny-bot
fidget-spinner-bot
weborama-fetcher
ows.eu/
barkrowler
owler
genai
imagesiftbot
wirereaderbot
timpibot
wpbot
friendlycrawler
amazonadbot
ips-agent
gabanzabot
geedoproductsearch
kakaotalk-scrap
storebot-google
checkmarknetwork
hawaiibot
iaskbot
adbeat_bot
makemerrybot
perplexitybot
redekenbot
stractbot
qwantbot
meta-externalagent
meta-externalfetcher
activecomply
oai-searchbot
cyberfindcrawler
wepchsearchengine
wget/1.6
wget/1.5.3
wget
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /hpt/outdoorslocomotive.php |
| Disallow | /hpt/spider.php |
| Disallow | /hpt/scripts.php |
| Disallow | /other/newsletterMC.php |
| Disallow | /other/jsnone.html |
| Disallow | /other/jsrecover.html |
| Disallow | /newsletter_subscribe.html |
| Disallow | /club/accept1 |
| Disallow | /club/validate/ |
| Disallow | /club/check2/ |
| Disallow | /counties/ |
| Disallow | /hpt/ |
| Disallow | /portal/ |
| Disallow | /includes/ |
| Disallow | /images/ |
| Disallow | /graphics/ |
| Disallow | /pictures/ |
| Disallow | /amendment/ |
| Disallow | /update/ |
| Disallow | /script/ |
| Disallow | /forms/ |
| Disallow | /form/ |
| Disallow | /supplierAmend/ |
| Disallow | /shopAmend/ |
| Disallow | /shopRPT/ |
| Disallow | /supplierRPT/ |
| Disallow | /banner/ |
| Disallow | /form/banner/ |
| Disallow | /shopreview/ |
| Disallow | /form/shopreview/ |
| Disallow | /layout/ |
| Disallow | /Connections/ |
| Disallow | /mscripts/ |
| Disallow | /preview/ |
| Disallow | /jscripts/ |
| Disallow | /Scripts/ |
| Disallow | /preview2/ |
| Disallow | /previewu/ |
| Disallow | /previewr/ |
| Disallow | /reply/ |
| Disallow | /replyr/ |
| Disallow | /shop/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.ukmodelshops.co.uk/sitemap.xml |
Warnings
- 3 invalid lines.