Log In Sign Up

bedford.cataloxy.co.uk
robots.txt

Robots Exclusion Standard data for bedford.cataloxy.co.uk

Resource Scan

Scan Details

Site Domain bedford.cataloxy.co.uk
Base Domain cataloxy.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2025-10-26T06:00:47+00:00
Next Scan 2025-11-09T06:00:47+00:00

Last Successful Scan

Scanned2025-09-18T03:01:16+00:00
URL https://bedford.cataloxy.co.uk/robots.txt
Domain IPs 104.21.91.17, 172.67.207.234, 2606:4700:3031::6815:5b11, 2606:4700:3032::ac43:cfea
Response IP 172.67.207.234
Found Yes
Hash 8926a874c91372618b2c502d0a8b480f1cc9b7b936299e6157ae3c9dff66332e
SimHash 4a44dc63f150

Groups

yandexdirect
mediapartners-google
yadirectfetcher

Rule Path
Allow /

yandex

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*
Disallow

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*
Disallow

Other Records

Field Value
crawl-delay 1

oai-searchbot

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

Other Records

Field Value
crawl-delay 3

gptbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

chatgpt-user

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

Other Records

Field Value
crawl-delay 1

yahoo

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

bingbot

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

msnbot

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

mail.ru

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

Other Records

Field Value
crawl-delay 15

*

Rule Path
Disallow /*comments.htm?*aa
Disallow /*comments.htm*comment_id%3D
Disallow /*comments.htm*del_comment%3D
Disallow /*comments.htm*confirm_send%3D
Disallow /*agreement.htm?o
Disallow /*?logout=
Disallow /*regions*?cr=
Disallow /setsubscribe
Disallow *confirmedok%3D
Disallow /board*?extend=
Disallow /*?print=
Disallow /*?*%40
Disallow /firms_rand_gold.htm*
Disallow /services_b.php*
Disallow /*%26df%5B
Disallow /*%26df%5B
Disallow /*?df%5B
Disallow /*?df%5B
Disallow /cabinet/*
Disallow /informers/?
Disallow /*?*payment
Disallow /*?change_city
Disallow /*?add=
Disallow /*%26add%3D
Disallow /end.php
Disallow /decode-encoded
Disallow /*?cataloxy
Disallow /*%26cataloxy
Disallow /feedback/?qT
Disallow *setlist%5Bdelivery%5D*
Disallow *setlist%5Bdelivery*
Disallow *setlist%5Bpriority%5D*
Disallow *setlist%5Bpriority*
Disallow /favorite.htm$
Disallow /rabota/*?pr=
Disallow /*keyword%3D
Disallow *%26txt
Disallow *?txt
Disallow /url/
Disallow /robox_notification.htm*

Other Records

Field Value
crawl-delay 3
Back to top

Other Records

Field Value
sitemap https://bedford.cataloxy.co.uk/sitemaps/co.uk/bedford/gen_sitemap_index.xml
Back to top

Warnings

  • `clean-param` is not a known field.
Back to top