Log In Sign Up

gbcrew.com
robots.txt

Robots Exclusion Standard data for gbcrew.com

Resource Scan

Scan Details

Site Domain gbcrew.com
Base Domain gbcrew.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-12-31T04:39:34+00:00
Next Scan 2026-03-01T04:39:34+00:00

Last Successful Scan

Scanned2025-09-17T01:21:38+00:00
URL https://gbcrew.com/robots.txt
Domain IPs 104.26.10.34, 104.26.11.34, 172.67.74.131, 2606:4700:20::681a:a22, 2606:4700:20::681a:b22, 2606:4700:20::ac43:4a83
Response IP 104.26.11.34
Found Yes
Hash 0fcd33fd2e94de0dcf8d9209a2a4575d335ba6946d9265a7105394609f66bbdb
SimHash 6914d840c182

Groups

*

Rule Path
Disallow /index.php/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /wishlist/
Disallow /admin/
Disallow /catalogsearch/
Disallow /review/product/
Disallow /sendfriend/
Disallow /enable-cookies/
Disallow /LICENSE.txt
Disallow /LICENSE.html
Disallow /directory/
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /tag/
Disallow /review/
Disallow /shop/

amazonbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

applebot-extended

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bytespider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

claudebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

google-extended

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

gptbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

meta-externalagent

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.gbcrew.com/sitemap.xml
Back to top

Comments

  • Crawler Setup
  • Disallow: /skin/
  • Disallow: /js/
  • XML Sitemap
Back to top