yorksu.org
robots.txt

Robots Exclusion Standard data for yorksu.org

Resource Scan

Scan Details

Site Domain yorksu.org
Base Domain yorksu.org
Scan Status Ok
Last Scan2025-10-01T18:32:49+00:00
Next Scan 2025-10-15T18:32:49+00:00

Last Scan

Scanned2025-10-01T18:32:49+00:00
URL https://yorksu.org/robots.txt
Domain IPs 104.21.45.65, 172.67.210.189, 2606:4700:3030::ac43:d2bd, 2606:4700:3033::6815:2d41
Response IP 104.21.45.65
Found Yes
Hash f9865b8b7cd0601ea0b7d35994156bf16b966f388af9a3b91662c4ba577dff8a
SimHash 4016d3f54557

Groups

*

Rule Path
Disallow /

googlebot

Rule Path
Allow /
Disallow /shop/*
Allow /shop/product/*

bingbot

Rule Path
Allow /
Disallow /shop/*
Allow /shop/product/*

facebookexternalhit

Rule Path
Allow /
Disallow /shop/*
Allow /shop/product/*

slurp

Rule Path
Allow /
Disallow /shop/*
Allow /shop/product/*

baiduspider

Rule Path
Allow /
Disallow /shop/*
Allow /shop/product/*

applebot

Rule Path
Allow /
Disallow /shop/*
Allow /shop/product/*

ibm-appconnect/v1

Rule Path
Allow /
Disallow /shop/*
Allow /shop/product/*