Log In Sign Up

luton.cataloxy.co.uk
robots.txt

Robots Exclusion Standard data for luton.cataloxy.co.uk

Resource Scan

Scan Details

Site Domain luton.cataloxy.co.uk
Base Domain cataloxy.co.uk
Scan Status Ok
Last Scan2025-09-28T09:12:38+00:00
Next Scan 2025-10-28T09:12:38+00:00

Last Scan

Scanned2025-09-28T09:12:38+00:00
URL https://luton.cataloxy.co.uk/robots.txt
Domain IPs 104.21.91.17, 172.67.207.234, 2606:4700:3031::6815:5b11, 2606:4700:3032::ac43:cfea
Response IP 104.21.91.17
Found Yes
Hash cd9f1ad6610c2064faa830559526b1fa03f4fff903dce7052d127c309da398c0
SimHash 4a44dc63f150

Groups

yandexdirect
mediapartners-google
yadirectfetcher

Rule Path
Allow /

yandex

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*
Disallow

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*
Disallow

Other Records

Field Value
crawl-delay 1

oai-searchbot

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

Other Records

Field Value
crawl-delay 3

gptbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

chatgpt-user

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

Other Records

Field Value
crawl-delay 1

yahoo

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

bingbot

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

msnbot

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

mail.ru

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

Other Records

Field Value
crawl-delay 15

*

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

Other Records

Field Value
crawl-delay 3
Back to top

Other Records

Field Value
sitemap https://luton.cataloxy.co.uk/sitemaps/co.uk/luton/gen_sitemap_index.xml
Back to top

Warnings

  • `clean-param` is not a known field.
Back to top