bcwsupplies.com
robots.txt

Robots Exclusion Standard data for bcwsupplies.com

Resource Scan

Scan Details

Site Domain bcwsupplies.com
Base Domain bcwsupplies.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-20T15:13:19+00:00
Next Scan 2025-01-18T15:13:19+00:00

Last Successful Scan

Scanned2024-05-31T15:11:26+00:00
URL https://bcwsupplies.com/robots.txt
Redirect https://www.bcwsupplies.com/robots.txt
Redirect Domain www.bcwsupplies.com
Redirect Base bcwsupplies.com
Domain IPs 172.66.40.233, 172.66.43.23
Redirect IPs 172.66.40.233, 172.66.43.23
Response IP 172.66.43.23
Found Yes
Hash 66a31bbb9cfa5cabfc38facd17fa90df009913d1c49b14687ea73a420e772185
SimHash 632e9c4beab2

Groups

*

Rule Path
Disallow /index.php/
Disallow /checkout/
Disallow /app/
Disallow /lib/
Disallow /*.php$
Disallow /pkginfo/
Disallow /report/
Disallow /var/
Disallow /catalog/
Disallow /customer/
Disallow /sendfriend/
Disallow /review/
Disallow /*SID%3D
Disallow /blog/category*
Disallow /wp*

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

pinterestbot/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

mozilla/5.0 (compatible; pinterestbot/1.0; +https://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

Other Records

Field Value
sitemap https://www.bcwsupplies.com/sitemap.xml