Log In Sign Up

petsware.co.uk
robots.txt

Robots Exclusion Standard data for petsware.co.uk

Resource Scan

Scan Details

Site Domain petsware.co.uk
Base Domain petsware.co.uk
Scan Status Ok
Last Scan2026-01-20T19:56:10+00:00
Next Scan 2026-02-19T19:56:10+00:00

Last Scan

Scanned2026-01-20T19:56:10+00:00
URL https://petsware.co.uk/robots.txt
Domain IPs 104.26.10.129, 104.26.11.129, 172.67.72.36, 2606:4700:20::681a:a81, 2606:4700:20::681a:b81, 2606:4700:20::ac43:4824
Response IP 104.26.11.129
Found Yes
Hash 24be955214a16498ac90fb3bf648e7f1c497a2d39e3409572581783476264371
SimHash e53919934ba1

Groups

*

Rule Path
Allow /
Allow /shop/
Allow /shop/dog-food/
Allow /shop/dog-beds/
Allow /shop/dog-treats/
Allow /shop/clearance/
Allow /contact-us
Allow /faqs
Allow /privacy-policy
Allow /refund-returns
Allow /terms-of-service
Allow /auth/login
Allow /auth/register
Allow /auth/forgot-password
Allow /auth/reset-password
Disallow /account/
Disallow /auth/logout
Disallow /cart
Disallow /checkout/
Disallow /_app/
Disallow /*?search=
Disallow /*?filter=
Disallow /*?sort=
Disallow /*?page=
Disallow /*?limit=

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://petsware.co.uk/sitemap.xml
Back to top

Comments

  • Sitemap location
  • Allow all product pages
  • Allow important public pages
  • Allow public auth pages (but not logout)
  • Disallow private and sensitive areas
  • Disallow search and filter parameters to prevent duplicate content
  • Allow specific crawlers full access
Back to top