magaseek.com
robots.txt

Robots Exclusion Standard data for magaseek.com

Resource Scan

Scan Details

Site Domain magaseek.com
Base Domain magaseek.com
Scan Status Ok
Last Scan2024-06-13T09:14:26+00:00
Next Scan 2024-06-20T09:14:26+00:00

Last Scan

Scanned2024-06-13T09:14:26+00:00
URL https://magaseek.com/robots.txt
Redirect https://www.magaseek.com:443/robots.txt
Redirect Domain www.magaseek.com
Redirect Base magaseek.com
Domain IPs 13.35.18.22, 13.35.18.47, 13.35.18.51, 13.35.18.62
Redirect IPs 13.35.18.22, 13.35.18.47, 13.35.18.51, 13.35.18.62
Response IP 13.35.18.51
Found Yes
Hash f151d1f09a0b038f3c16dd77a9721f96bb04dfb5569f5df5594ac01316e425ab
SimHash 6545dc156753

Groups

*

Rule Path
Disallow /cart/
Disallow /stcklist/
Disallow /other/size/
Disallow /other/henpin/
Disallow /shop/top/sh_-tr_s
Disallow /oauth/authenticate

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://www.magaseek.com/sitemap_index.xml

Comments

  • User-agent: Slurp
  • Crawl-Delay: 2