catalogueoffers.co.uk
robots.txt

Robots Exclusion Standard data for catalogueoffers.co.uk

Resource Scan

Scan Details

Site Domain catalogueoffers.co.uk
Base Domain catalogueoffers.co.uk
Scan Status Ok
Last Scan2024-11-09T09:24:16+00:00
Next Scan 2024-11-16T09:24:16+00:00

Last Scan

Scanned2024-11-09T09:24:16+00:00
URL https://catalogueoffers.co.uk/robots.txt
Redirect https://www.catalogueoffers.co.uk/robots.txt
Redirect Domain www.catalogueoffers.co.uk
Redirect Base catalogueoffers.co.uk
Domain IPs 104.21.34.163, 172.67.163.18, 2606:4700:3030::ac43:a312, 2606:4700:3033::6815:22a3
Redirect IPs 13.227.254.118, 13.227.254.72, 13.227.254.82, 13.227.254.89, 2600:9000:200a:1e00:e:11d3:21c0:93a1, 2600:9000:200a:4800:e:11d3:21c0:93a1, 2600:9000:200a:5c00:e:11d3:21c0:93a1, 2600:9000:200a:6c00:e:11d3:21c0:93a1, 2600:9000:200a:7400:e:11d3:21c0:93a1, 2600:9000:200a:8800:e:11d3:21c0:93a1, 2600:9000:200a:a400:e:11d3:21c0:93a1, 2600:9000:200a:c400:e:11d3:21c0:93a1
Response IP 13.227.254.118
Found Yes
Hash fbebbdb8029af47f129b626335ed2e5722fb9f5b30ee12aadf37b7579989aa22
SimHash ecf7ef01d110

Groups

*

Rule Path
Allow /
Disallow /search/*
Disallow /shopping-list/
Disallow /click-out/
Disallow /index.php/click-out/
Disallow /admin/
Disallow /cdn-cgi/
Allow /stores/*/leaflets-offers?*utm_*
Allow /latest-leaflets?*utm_*
Allow /almost-expired-leaflets?*utm_*
Allow /popular-leaflets?*utm_*
Allow /recommended-leaflets?*utm_*
Allow /stores/*/offers/*-offer-*/?*utm_*
Disallow /offers/*?sort=relevance&page=*&*
Allow /offers/*?sort=relevance&page=*
Disallow /offers/*?page=*&*
Allow /offers/*?page=*
Disallow /offers/*?*
Disallow /brands/*?page=*&*
Allow /brands/*?page=*
Disallow /brands/*?*
Disallow /stores/*?page=*&*
Allow /stores/*?page=*
Disallow /stores/*?*
Disallow /cities/*?page=*&*
Allow /cities/*?page=*
Disallow /cities/*?*
Disallow /latest-leaflets?page=*&*
Allow /latest-leaflets?page=*
Disallow /latest-leaflets?*
Disallow /almost-expired-leaflets?page=*&*
Allow /almost-expired-leaflets?page=*
Disallow /almost-expired-leaflets?*
Disallow /popular-leaflets?page=*&*
Allow /popular-leaflets?page=*
Disallow /popular-leaflets?*
Disallow /recommended-leaflets?page=*&*
Allow /recommended-leaflets?page=*
Disallow /recommended-leaflets?*
Disallow /latest-offers?page=*&*
Allow /latest-offers?page=*
Disallow /latest-offers?*
Disallow /almost-expired-offers?page=*&*
Allow /almost-expired-offers?page=*
Disallow /almost-expired-offers?*
Disallow /popular-offers?page=*&*
Allow /popular-offers?page=*
Disallow /popular-offers?*
Disallow /recommended-offers?page=*&*
Allow /recommended-offers?page=*
Disallow /recommended-offers?*

adsbot-google

Rule Path
Disallow /click-out/
Disallow /admin/

*
adsbot-google

Rule Path
Disallow /cdn-cgi/bm/cv/
Disallow /cdn-cgi/challenge-platform/
Disallow /cdn-cgi/images/trace/
Disallow /cdn-cgi/rum
Disallow /cdn-cgi/scripts/
Disallow /cdn-cgi/styles/
Disallow /cdn-cgi/zaraz/

nuclei
wikido
riddler
petalbot
zoominfobot
go-http-client
node/simplecrawler
cazoodlebot
dotbot/1.0
gigabot
barkrowler
blexbot
magpie-crawler
mj12bot
ahrefsbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.catalogueoffers.co.uk/sitemap.xml