Log In Sign Up

shop.bigtenstore.com
robots.txt

Robots Exclusion Standard data for shop.bigtenstore.com

Resource Scan

Scan Details

Site Domain shop.bigtenstore.com
Base Domain bigtenstore.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-07-28T18:27:34+00:00
Next Scan 2024-10-26T18:27:34+00:00

Last Successful Scan

Scanned2024-03-24T18:25:15+00:00
URL https://shop.bigtenstore.com/robots.txt
Domain IPs 23.59.168.147, 23.59.168.170
Response IP 23.59.168.170
Found Yes
Hash b63bede1f8a2b7cf769f32caf83576d4983f21c8cf5b6369bce6e49472cfbda3
SimHash 4cbc5946e67c

Groups

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

aspiegelbot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

adsbot-google
adidxbot

Rule Path
Allow /*?_s=gppc*
Allow /*%2Bf-*
Allow /*/browse/featuredproduct/*

googlebot
*

Rule Path
Allow /
Disallow /*_r%3D*
Disallow /*query*
Disallow /account
Disallow /track-order
Disallow /cart
Disallow /shipping*
Disallow /payment*
Disallow /thank-you/*
Disallow *?vap=1
Back to top

Comments

  • Block MegaIndex.ru
Back to top