prod.medo.pitchleads.com
robots.txt

Robots Exclusion Standard data for prod.medo.pitchleads.com

Resource Scan

Scan Details

Site Domain prod.medo.pitchleads.com
Base Domain pitchleads.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-04-25T06:43:21+00:00
Next Scan 2024-07-24T06:43:21+00:00

Last Successful Scan

Scanned2023-06-08T00:38:49+00:00
URL http://prod.medo.pitchleads.com/robots.txt
Redirect https://www.macmillandictionary.com/robots.txt
Redirect Domain www.macmillandictionary.com
Redirect Base macmillandictionary.com
Domain IPs 54.179.136.179
Redirect IPs 54.179.136.179
Response IP 54.77.8.77
Found Yes
Hash b347147b8a28e22c71e3524d66efff85bd55dc776be338fa9e978cefbe1eba29
SimHash f71252684d87

Groups

alexibot
alvinetspider
antenne hatena
apocalxexplorerbot
asterias
backdoorbot/1.0
bizinformation
black hole
blowfish/1.0
botalot
builtbottough
bullseye/1.0
bunnyslippers
cegbfeieh
cheesebot
cherrypicker
cherrypickerelite/1.0
cherrypickerse/1.0
copyrightcheck
cosmos
crescent
crescent internet toolpak http ole control v.1.0
disco pump 3.1
dittospyder
dotbot
emailcollector
emailsiphon
emailwolf
erocrawler
extractorpro
flamingo_searchengine
foobot
harvest/1.5
hloader
httplib
httrack
httrack 3.0
humanlinks
igentia
infonavirobot
jennybot
jikespider
kenjin spider
lexibot
libweb/clshttp
linkextractorpro
linkscan/8.1a unix
linkwalker
lwp-trivial
lwp-trivial/1.34
mata hari
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
miixpc
miixpc/4.2
mister pix
mlbot
moget
moget/2.1
ms search 4.0 robot
ms search 5.0 robot
naverbot
netants
netattache
netmechanic
nicerspro
offline explorer
openfind
openindexspider
propowerbot/2.14
prowebwalker
psbot
quepasacreep
queryn metasearch
repomonkey
rma
semrushbot
sightupbot
sitebot
sitesnagger
sitesucker
sogou web spider
sosospider
spankbot
spanner
speedy
suggybot
superbot
superbot/2.6
suzuran
szukacz/1.4
teleport
telesoft
the intraformant
thenomad
tighttwatbot
titan
tocrawl/urldispatcher
toscrawler
true_robot
true_robot/1.0
turingos
turnitinbot
urlpouls
urly warning
vci
web image collector
webauto
webbandit
webbandit/3.50
webcopier
webcopy
webenhancer
webmasterworldforumbot
webmirror
webreaper
websauger
website extractor
website quester
webster pro
webstripper
webstripper/2.02
webzip
wget
wikiofeedbot
winhttrack
www-collector-e
xenu link sleuth/1.3.8
yacy
yrspider
zeus
zookabot

Rule Path
Disallow /

*

Rule Path
Disallow /search
Disallow /search/
Disallow /spellcheck
Disallow /spellcheck/
Disallow /info
Disallow /info/
Disallow /autocomplete
Disallow /autocomplete/
Disallow /us/search
Disallow /us/search/
Disallow /us/spellcheck
Disallow /us/spellcheck/
Disallow /us/info
Disallow /us/info/
Disallow /us/autocomplete
Disallow /us/autocomplete/

Other Records

Field Value
sitemap https://www.macmillandictionary.com/sitemap.xml

Warnings

  • 1 invalid line.