dipslm.com
robots.txt

Robots Exclusion Standard data for dipslm.com

Resource Scan

Scan Details

Site Domain dipslm.com
Base Domain dipslm.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-30T00:23:28+00:00
Next Scan 2024-12-29T00:23:28+00:00

Last Successful Scan

Scanned2023-08-13T21:29:51+00:00
URL https://dipslm.com/robots.txt
Redirect https://www.dipslm.com/robots.txt
Redirect Domain www.dipslm.com
Redirect Base dipslm.com
Domain IPs 65.9.112.123, 65.9.112.37, 65.9.112.53, 65.9.112.66
Redirect IPs 143.204.9.10, 143.204.9.109, 143.204.9.2, 143.204.9.68
Response IP 18.66.122.107
Found Yes
Hash 4afa7dff06ab36124a068dabe5de40643c8e1b3d2b891e5f8485b531c4c623a4
SimHash f41cfc80dab4

Groups

*

Rule Path
Disallow /leads/
Disallow *.pdf$
Disallow /inventory.aspx*
Disallow /inventory-*.html

bingbot
msnbot
semrushbot
semrushbot-sa
scoutjet
siteimprove.com
match by siteimprove.com
linkcheck by siteimprove.com
sitecheck-sitecrawl by siteimprove.com

Rule Path
Disallow /leads/
Disallow *.pdf$
Disallow /inventory.aspx*
Disallow /inventory-*.html

Other Records

Field Value
crawl-delay 6

mj12bot
omniexplorer_bot
wells search ii 0.0
heritrix/1.10.0
shopwiki
scanalert
copernic
psbot
python-urllib
baiduspider
yandex
ahrefsbot
trovitbot
blexbot
seokicks-robot
cliqzbot
mauibot
bubing
qwantify
tweetmemebot
autobot
seokicks
petalbot
barkrowler
zoominfobot
sogou spider
tinytestbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.dipslm.com/sitemap.xml
sitemap https://www.dipslm.com/sitemap-video.xml
sitemap https://www.dipslm.com/sitemap-geo.xml
sitemap https://www.dipslm.com/sitemap-images.xml

Warnings

  • 2 invalid lines.