Log In Sign Up

thebike.co.il
robots.txt

Robots Exclusion Standard data for thebike.co.il

Resource Scan

Scan Details

Site Domain thebike.co.il
Base Domain thebike.co.il
Scan Status Ok
Last Scan2024-09-20T00:34:48+00:00
Next Scan 2024-09-27T00:34:48+00:00

Last Scan

Scanned2024-09-20T00:34:48+00:00
URL https://thebike.co.il/robots.txt
Redirect https://www.thebike.co.il/robots.txt
Redirect Domain www.thebike.co.il
Redirect Base thebike.co.il
Domain IPs 104.21.71.124, 172.67.145.40, 2606:4700:3030::ac43:9128, 2606:4700:3035::6815:477c
Redirect IPs 104.21.71.124, 172.67.145.40, 2606:4700:3030::ac43:9128, 2606:4700:3035::6815:477c
Response IP 104.21.71.124
Found Yes
Hash e15a0309b53fa32c35c2dbfa4f225ae106716db8795948598e84764ba5b3c625
SimHash a6c65640c023

Groups

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduspider+(+http://www.baidu.com/search/spider.htm)

Rule Path
Disallow /

baiduspider/2.0;+http://www.baidu.com/search/spider.html

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

mozilla/5.0(compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Rule Path
Disallow /

spider

Rule Path
Disallow /

sosospider+

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

mozilla/5.0 (compatible; ezooms/1.0; ezooms.bot[at]gmail[dot]com)

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /
Back to top

Comments

  • Baiduspider
  • spider
  • Sosospider
  • Ezooms
  • Sogou
Back to top

Warnings

  • 6 invalid lines.
Back to top