1stoplighting.com
robots.txt

Robots Exclusion Standard data for 1stoplighting.com

Resource Scan

Scan Details

Site Domain 1stoplighting.com
Base Domain 1stoplighting.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-14T11:57:50+00:00
Next Scan 2024-12-13T11:57:50+00:00

Last Successful Scan

Scanned2023-04-29T02:31:10+00:00
URL https://www.1stoplighting.com/robots.txt
Domain IPs 104.103.145.154
Response IP 104.69.151.151
Found Yes
Hash 338a3b7e53b4f5a5322d106ec3ad8f81b59e7feed97daff22a4a8d098e3df2a1
SimHash 2316f0169112

Groups

*

Rule Path
Allow /

*

Rule Path
Disallow /ProductSheet.aspx
Disallow /secure/
Disallow /product.aspx
Disallow /more-info.aspx
Disallow /cart.aspx
Disallow /search.aspx
Disallow /*/product.aspx
Disallow /Secure/*
Disallow /Project/*
Disallow /Product/*
Disallow /Search/*
Disallow /Cart/*
Disallow /Error/*
Disallow /User/*
Disallow /Media/*
Disallow /Survey/*
Disallow /Supplier/*
Disallow /User/*
Disallow /*.php$
Disallow /*first_answer%3D

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1.0

megaindex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5.0

redcarpet
shopwiki
baiduspider
voyager
yandex
sogou spider
converacrawler
ocelli
scoutjet
camontspider
ahrefsbot
discobot
twiceler
fatbot
becomebot
speedy
openbot
dealozbot
naverbot
exabot
yanga
psbot
mj12bot
ubicrawler
semanticdiscovery
turnitinbot
twengabot
sitebot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.1stoplighting.com/sitemap1stoplighting.xml

Comments

  • Global Directives

Warnings

  • 1 invalid line.