bbc.com
robots.txt
Robots Exclusion Standard data for bbc.com
Resource Scan
Scan Details
| Site Domain | bbc.com |
| Base Domain | bbc.com |
| Scan Status | Ok |
| Last Scan | 2024-05-03T06:47:17+00:00 |
| Next Scan | 2024-05-10T06:47:17+00:00 |
Last Scan
| Scanned | 2024-05-03T06:47:17+00:00 |
| URL | https://bbc.com/robots.txt |
| Redirect | https://www.bbc.com/robots.txt |
| Redirect Domain | www.bbc.com |
| Redirect Base | bbc.com |
| Domain IPs | 151.101.0.81, 151.101.128.81, 151.101.192.81, 151.101.64.81, 2a04:4e42:200::81, 2a04:4e42:400::81, 2a04:4e42:600::81, 2a04:4e42::81 |
| Redirect IPs | 151.101.0.81, 151.101.128.81, 151.101.192.81, 151.101.64.81 |
| Response IP | 199.232.44.81 |
| Found | Yes |
| Hash | be2d5ebdbcae2308977575ff330f41663d09f67ccc92f513bdfa2b08d8bddcfe |
| SimHash | dd2ead8cb885 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /bitesize/search$ |
| Disallow | /bitesize/search/ |
| Disallow | /bitesize/search? |
| Disallow | /cbbc/search/ |
| Disallow | /cbbc/search$ |
| Disallow | /cbbc/search? |
| Disallow | /cbeebies/search/ |
| Disallow | /cbeebies/search$ |
| Disallow | /cbeebies/search? |
| Disallow | /chwilio/ |
| Disallow | /chwilio$ |
| Disallow | /chwilio? |
| Disallow | /education/blocks$ |
| Disallow | /education/blocks/ |
| Disallow | /newsround |
| Disallow | /search/ |
| Disallow | /search$ |
| Disallow | /search? |
| Disallow | /food/favourites |
| Disallow | /food/search*?* |
| Disallow | /food/recipes/search*?* |
| Disallow | /education/my$ |
| Disallow | /education/my/ |
| Disallow | /bitesize/my$ |
| Disallow | /bitesize/my/ |
| Disallow | /food/recipes/*/shopping-list |
| Disallow | /food/menus/*/shopping-list |
| Disallow | /news/0 |
| Disallow | /sport/alpha/ |
| Disallow | /ugc$ |
| Disallow | /ugc/ |
| Disallow | /ugcsupport$ |
| Disallow | /ugcsupport/ |
| Disallow | /userinfo/ |
| Disallow | /userinfo |
| Disallow | /u5llnop$ |
| Disallow | /u5llnop/ |
| Disallow | /sounds/search$ |
| Disallow | /sounds/search/ |
| Disallow | /sounds/search? |
| Disallow | /ws/includes |
| Disallow | /radio/imda |
| Disallow | /storyworks/preview/* |
| Disallow | /rd/search$ |
| Disallow | /rd/search/ |
| Disallow | /rd/search? |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.bbc.com/sitemaps/https-index-com-archive.xml |
| sitemap | https://www.bbc.com/sitemaps/https-index-com-news.xml |
| sitemap | https://www.bbc.com/sitemaps/https-index-com-archive_video.xml |
| sitemap | https://www.bbc.com/sitemaps/https-index-com-video.xml |
| sitemap | https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml |
| sitemap | https://www.bbc.com/sport/sitemap.xml |
| sitemap | https://www.bbc.com/sitemaps/sitemap-com-ws-topics.xml |
| sitemap | https://www.bbc.com/afrique/sitemap.xml |
| sitemap | https://www.bbc.com/arabic/sitemap.xml |
| sitemap | https://www.bbc.com/bengali/sitemap.xml |
| sitemap | https://www.bbc.com/burmese/sitemap.xml |
| sitemap | https://www.bbc.com/gahuza/sitemap.xml |
| sitemap | https://www.bbc.com/hausa/sitemap.xml |
| sitemap | https://www.bbc.com/hindi/sitemap.xml |
| sitemap | https://www.bbc.com/indonesia/sitemap.xml |
| sitemap | https://www.bbc.com/mundo/sitemap.xml |
| sitemap | https://www.bbc.com/pashto/sitemap.xml |
| sitemap | https://www.bbc.com/persian/sitemap.xml |
| sitemap | https://www.bbc.com/portuguese/sitemap.xml |
| sitemap | https://www.bbc.com/russian/sitemap.xml |
| sitemap | https://www.bbc.com/swahili/sitemap.xml |
| sitemap | https://www.bbc.com/tajik/sitemap.xml |
| sitemap | https://www.bbc.com/turkce/sitemap.xml |
| sitemap | https://www.bbc.com/ukchina/simp/sitemap.xml |
| sitemap | https://www.bbc.com/ukrainian/sitemap.xml |
| sitemap | https://www.bbc.com/urdu/sitemap.xml |
| sitemap | https://www.bbc.com/uzbek/sitemap.xml |
| sitemap | https://www.bbc.com/vietnamese/sitemap.xml |
| sitemap | https://www.bbc.com/zhongwen/simp/sitemap.xml |
| sitemap | https://www.bbc.com/zhongwen/trad/sitemap.xml |
| sitemap | https://www.bbc.com/bbcx/index_sitemap.xml |
Comments