telegraph.co.uk
robots.txt

Robots Exclusion Standard data for telegraph.co.uk

Resource Scan

Scan Details

Site Domain telegraph.co.uk
Base Domain telegraph.co.uk
Scan Status Ok
Last Scan2024-09-27T12:33:57+00:00
Next Scan 2024-10-04T12:33:57+00:00

Last Scan

Scanned2024-09-27T12:33:57+00:00
URL https://telegraph.co.uk/robots.txt
Redirect https://www.telegraph.co.uk/robots.txt
Redirect Domain www.telegraph.co.uk
Redirect Base telegraph.co.uk
Domain IPs 3.164.182.36, 3.164.182.48, 3.164.182.50, 3.164.182.60
Redirect IPs 23.36.50.134
Response IP 23.54.58.156
Found Yes
Hash 27cd969d1eccd03d0a8fd453d447bc6ef9eae7aa0673b1da0cb1c672d32b4780
SimHash 231e0ca057a0

Groups

*

Rule Path
Disallow /*?mobile=true
Disallow /*?mobile=basic
Disallow /*?ModPagespeed=noscript
Disallow /*_jcr_content*
Disallow /*?source=rss
Disallow /search/*
Disallow /searchbeta/*
Disallow /news/main.jhtml
Disallow /p/*/embed/
Disallow /secure/login/*
Disallow /content/telegraph/
Disallow /customer/secure/checkout/tesco/
Disallow /customer/secure/reset-password/
Disallow /telegraph/*
Disallow /news-app/*
Disallow /amp$
Disallow */application/*
Disallow */ixale/
Disallow /core/Content/
Disallow /promotions/emails/
Disallow /r/
Disallow /sponsored/travel/msc-cruises/
Disallow /travel/8711559/The-Telegraph-Travel-Awards-2011.html
Disallow /travel/hotel/e/*
Disallow /sponsored/staging/
Disallow /sponsored/business/lloyds-tsb-enterprise-awards/
Disallow /sponsored/earth/statoil/
Disallow /sponsored/motoring/alfa-romeo-cars/
Disallow /sponsored/motoring/vw-up/
Disallow /sponsored/property/all-saints-eastbourne/
Disallow /sponsored/supplement-portfolio/
Disallow /sponsored/travel/cunard-cruises/
Disallow /sponsored/travel/cruise-holidays/
Disallow /sponsored/travel/macau/macaumap/
Disallow /sponsored/travel/telegraph-cottages/
Disallow /sponsored/finance/spread-betting/
Disallow /sponsored/finance/retirement-annuity/
Disallow /sponsored/travel/hidden-britain/
Disallow /sponsored/business/sme-business-essentials/
Disallow /sponsored/in-the-know/london-cultural-attractions
Disallow /sponsored/in-the-know/london-dining
Disallow /sponsored/in-the-know/london-entertainment
Disallow /sponsored/in-the-know/london-lifestyle
Disallow /sponsored/in-the-know/london-nightlife
Disallow /sponsored/in-the-know/london-shopping
Disallow /sponsored/in-the-know/london-sport-activities
Disallow /sponsored/in-the-know/london-transport-accommodation
Disallow /sponsored/in-the-know/london-video-guides
Disallow /sponsored/motoring/suzuki-motorbikes/
Disallow /sponsored/technology/cool-list/
Disallow /travel/hotels/hotel-finder/
Disallow /podcasts-more/
Allow /travel/hotels/hotel-finder/$

endeca

Rule Path
Disallow /archive/
Disallow /search/*

google-extended

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.telegraph.co.uk/sitemap.xml
sitemap https://www.telegraph.co.uk/custom/authors/sitemap.xml
sitemap https://www.telegraph.co.uk/custom/main-pages/sitemap.xml
sitemap https://www.telegraph.co.uk/custom/daily-news/sitemap.xml
sitemap https://www.telegraph.co.uk/vouchercodes/sitemap.xml

Comments

  • Production Robots.txt file
  • Special parameters
  • Internal Search
  • Special areas
  • Bots which make unnecessary bot traffic
  • Sitemap URLs