Log In Sign Up

ft.dk
robots.txt

Robots Exclusion Standard data for ft.dk

Resource Scan

Scan Details

Site Domain ft.dk
Base Domain ft.dk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-06-23T20:51:28+00:00
Next Scan 2024-07-07T20:51:28+00:00

Last Successful Scan

Scanned2024-05-16T20:50:58+00:00
URL https://ft.dk/robots.txt
Redirect https://www.ft.dk/robots.txt
Redirect Domain www.ft.dk
Redirect Base ft.dk
Domain IPs 104.22.18.97, 104.22.19.97, 172.67.4.156, 2606:4700:10::6816:1261, 2606:4700:10::6816:1361, 2606:4700:10::ac43:49c
Redirect IPs 104.22.18.97, 104.22.19.97, 172.67.4.156, 2606:4700:10::6816:1261, 2606:4700:10::6816:1361, 2606:4700:10::ac43:49c
Response IP 104.22.19.97
Found Yes
Hash 8154b37dce3239d53b0d5d4ee74acc27357f011db0253975f9d1cd1cf401755d
SimHash 2d50fe3e6e72

Groups

*

Rule Path
Disallow /Minkkommissionen/
Disallow /sitecore/content/
Disallow /sitecore/indhold/
Disallow /samling/20091/almdel/SUU/bilag/459/index.htm
Disallow /samling/20091/almdel/SUU/bilag/459/881338/index.htm
Disallow /samling/20091/almdel/SUU/bilag/459/881338.pdf
Disallow /samling/20061/almdel/REU/Bilag/32/311790/index.HTM
Disallow /samling/20061/almdel/REU/Bilag/32/311790.PDF
Disallow /samling/20061/almdel/REU/Bilag/32/311794/index.HTM
Disallow /samling/20061/almdel/REU/Bilag/32/311794.pdf
Disallow /samling/20061/almdel/UVT/Bilag/52/326779/index.htm
Disallow /samling/20061/almdel/UVT/Bilag/52/326779.pdf
Disallow /samling/20061/almdel/tru/bilag/299/355354.pdf
Disallow /samling/20061/almdel/tru/bilag/299/355354/index.htm
Disallow /samling/20061/almdel/tru/bilag/299/
Disallow /samling/20061/almdel/tru/bilag/299
Disallow /samling/20061/almdel/URU/Bilag/259/398353.pdf
Disallow /samling/20061/almdel/URU/Bilag/259/398353/index.htm
Disallow /samling/20061/almdel/URU/Bilag/259/398355.pdf
Disallow /samling/20042/lovforslag/L27/bilag/7/index.htm
Disallow /samling/20042/Lovforslag/L27/Bilag/7/162719.PDF
Disallow /samling/20072/almdel/REU/spm/754/svar/endeligt/20080627/576028.PDF
Disallow /samling/20072/almdel/REU/spm/754/svar/endeligt/20080627/576028/index.HTM
Disallow /samling/20072/almdel/REU/spm/754/svar/endeligt/20080627/index.htm
Disallow /samling/20061/almdel/UVT/Bilag/233/392371.PDF
Disallow /Samling/20081/almdel/sou/bilag/358/index.htm
Disallow /Samling/20081/almdel/sou/bilag/358/696594/index.htm
Disallow /Samling/20081/almdel/sou/bilag/358/696594.PDF
Disallow /Samling/20081/almdel/sou/bilag/360/index.htm
Disallow /Samling/20081/almdel/sou/bilag/360/696623.PDF
Disallow /Samling/20081/almdel/sou/bilag/361/index.htm
Disallow /Samling/20081/almdel/sou/bilag/361/696675.PDF
Disallow /Samling/20072/almdel/EPU/bilag/265/index.htm
Disallow /samling/20072/almdel/amu/bilag/273/589504.pdf
Disallow /samling/20072/almdel/amu/bilag/273/589504/index.htm
Disallow /Samling/20081/almdel/SOU/bilag/238/index.htm
Disallow /Samling/20081/almdel/SOU/bilag/375/index.htm
Disallow /Samling/20061/almdel/SOU/bilag/407/index.htm
Disallow /samling/20081/almdel/sou/spm/237/index.htm
Disallow /samling/20081/almdel/sou/bilag/403/index.htm
Disallow /samling/20081/lovforslag/L15/spm/3/index.htm
Disallow /samling/20081/lovforslag/L15/spm/34/index.htm
Disallow /samling/20081/lovforslag/L15/bilag/6/index.htm
Disallow /samling/20081/lovforslag/L15/bilag/34/index.htm
Disallow /samling/20081/MENU/dok_SOU_L15_ALLE.htm
Disallow /samling/20051/almdel/SOU/Bilag/261/261970.pdf
Disallow /samling/20051/almdel/SOU/Bilag/261/261970/index.htm
Disallow /samling/20091/almdel/udu/bilag/342/890739.pdf
Disallow /samling/20091/almdel/SUU/bilag/472/index.htm
Disallow /samling/20091/almdel/suu/bilag/472/884093/index.htm
Disallow /samling/20091/almdel/udu/bilag/342/index.htm
Disallow /samling/20091/almdel/UDU/spm/508/index.htm
Disallow /samling/20091/almdel/udu/spm/508/890747.pdf
Disallow /samling/20091/almdel/udu/spm/508/890747/index.htm
Disallow /samling/20091/almdel/udu/spm/508/895177/index.htm
Disallow /samling/20072/almdel/TRU/Bilag/241/538897.PDF
Disallow /samling/20072/almdel/TRU/Bilag/241/538897.pdf
Disallow /samling/20072/almdel/TRU/Bilag/155/526987.PDF
Disallow /samling/20072/almdel/udu/spm/233/index.htm
Disallow /samling/20072/almdel/udu/spm/233/svar/560853/577398.pdf
Disallow /samling/20072/almdel/UDU/spm/233/index.htm
Disallow /samling/20072/almdel/UDU/spm/233/svar/560853/577398.pdf
Disallow /samling/20072/almdel/udu/spm/233/svar/560853/577398/index.htm
Disallow /samling/20061/almdel/sou/bilag/220/352688.pdf
Disallow /samling/20061/almdel/sou/bilag/220/352688/index.htm
Disallow /samling/20061/almdel/SOU/bilag/220/index.htm
Disallow /samling/20061/almdel/sou/spm/119/index.htm
Disallow /samling/20061/almdel/SOU/spm/119/index.htm
Disallow /samling/20072/almdel/bou/spm/74/index.htm
Disallow /samling/20072/almdel/bou/spm/74/svar/551126/561762.pdf
Disallow /samling/20072/almdel/bou/spm/74/svar/551126/561762/index.htm
Disallow /samling/20072/almdel/BOU/bilag/78/555383.PDF
Disallow /samling/20051/almdel/udu/bilag/137/232296/index.htm
Disallow /samling/20042/almdel/sau/bilag/51/157201.pdf
Disallow /samling/20042/almdel/sau/bilag/51/index.htm
Disallow /samling/20042/almdel/sau/bilag/51/157201/index.htm
Disallow /samling/20072/almdel/udu/spm/9/515062.pdf
Disallow /samling/20072/almdel/udu/bilag/12/index.htm
Disallow /samling/20072/almdel/udu/spm/9/index.htm
Disallow /samling/20072/almdel/udu/bilag/12/409374.pdf
Disallow /samling/20072/almdel/udu/bilag/12/409374/index.htm
Disallow /samling/20072/almdel/udu/spm/9/515062/index.htm
Disallow /samling/20101/lovforslag/l46/bilag/47/924894.pdf
Disallow /samling/20101/lovforslag/l46/bilag/47/924894/index.htm
Disallow /Samling/20021/MENU/00742943.htm
Disallow /Samling/20012/MENU/00644779.htm
Disallow /Samling/20021/MENU/00753984.htm
Disallow /Samling/20012/MENU/00649964.htm
Disallow /Samling/20012/MENU/00631066.htm
Disallow /Samling/20021/udvbilag/SOU/Almdel_bilag653.htm
Disallow /Samling/20021/udvbilag/SOU/B57_bilag7.htm
Disallow /samling/20081/almdel/epu/bilag/124/index.htm
Disallow /samling/20081/almdel/epu/bilag/124/633705.pdf
Disallow /samling/20081/raadsmoede/517792/bilag/124/index.htm
Disallow /samling/20091/almdel/bou/bilag/55/index.htm
Disallow /samling/20091/almdel/bou/bilag/55/838820.pdf
Disallow /samling/20091/almdel/bou/bilag/55/838820/index.htm
Disallow /samling/20091/almdel/bou/bilag/56/index.htm
Disallow /samling/20091/almdel/bou/bilag/56/834218/index.htm
Disallow /samling/20091/almdel/bou/spm/292/index.htm
Disallow /samling/20091/almdel/BOU/spm/292/svar/718942/index.htm
Disallow /samling/20091/almdel/bou/spm/292/svar/718942/851564/index.htm
Disallow /samling/20091/almdel/bou/spm/292/svar/723207/859081.pdf
Disallow /samling/20091/almdel/BOU/bilag/56/834218.pdf
Disallow /samling/20091/almdel/bou/spm/292/834234.pdf
Disallow /samling/20091/almdel/bou/spm/292/svar/723207/859081.pdf?title=Endelig_esvarelse_af_BOU_alm.del_spm._292&size=&pages=
Disallow /samling/20051/almdel/reu/spm/549/svar/278138/309067.pdf
Disallow /samling/20051/almdel/reu/spm/549/svar/278138/309067/index.htm
Disallow /samling/20081/almdel/tru/bilag/215/index.htm
Disallow /samling/20081/almdel/tru/bilag/215/633743.pdf
Disallow /samling/20081/almdel/tru/bilag/215/633731.pdf
Disallow /samling/20081/almdel/tru/bilag/215/633743/index.htm
Disallow /samling/20081/almdel/tru/bilag/215/633731/index.htm
Disallow /samling/20081/almdel/p%C3%B8u/bilag/46/index.htm
Disallow /samling/20081/almdel/p%C3%B8u/bilag/46/633201.pdf
Disallow /samling/20081/almdel/p%C3%B8u/bilag/46/633201/index.htm
Disallow /samling/20081/almdel/sau/bilag/102/index.htm
Disallow /samling/20081/almdel/sau/bilag/102/633607.pdf
Disallow /samling/20081/almdel/sau/bilag/102/633607/index.htm
Disallow /samling/20061/almdel/sou/bilag/220/index.htm
Disallow /samling/20061/almdel/sou/spm/119/svar/319110/356226/index.htm
Disallow /samling/20072/almdel/bou/bilag/78/index.htm
Disallow /samling/20072/almdel/BOU/spm/74/index.htm
Disallow /samling/20081/almdel/mpu/bilag/265/633922.pdf
Disallow /samling/20061/almdel/sou/spm/119/svar/319110/356226.pdf
Disallow /samling/20081/almdel/mpu/bilag/265/633922/index.htm
Disallow /samling/20081/almdel/eru/bilag/113/633922/index.htm
Disallow /samling/20101/almdel/fou/bilag/2/index.htm
Disallow /samling/20101/almdel/fou/bilag/2/897664/index.htm
Disallow /samling/20101/almdel/fou/spm/20/index.htm
Disallow /samling/20101/almdel/fou/spm/20/897673/index.htm
Disallow /samling/20101/almdel/fou/spm/20/svar/749980/908970/index.htm
Disallow /samling/20101/almdel/fou/spm/20/svar/749980/908969/index.htm
Disallow /samling/20101/almdel/fou/bilag/2/897664/index.htm?
Disallow /samling/20101/almdel/fou/spm/20/svar/749980/908970.pdf
Disallow /samling/20101/almdel/fou/spm/20/897673.pdf
Disallow /samling/20121/almdel/%C2%A771/bilag/49/1213278.pdf
Disallow /Dokumenter/Publikationer/Folketinget/Saa_meget_tjener.aspx
Disallow /Demokrati/~/media/Pdf_materiale/Pdf_publikationer/Informationsark/Folketingets_medlemmer/saa_meget_tjener%20pdf.ashx
Disallow /samling/20121/almdel/uui/bilag/69/index.htm
Disallow /samling/20121/almdel/beu/bilag/124/index.htm
Disallow /samling/20072/almdel/flf/bilag/135/index.htm
Disallow /Samling/20081/almdel/reu/bilag/698/index.htm
Disallow /samling/20121/almdel/UUI/bilag/69/index.htm
Disallow /samling/20121/almdel/BEU/bilag/124/index.htm
Disallow /samling/20072/almdel/FLF/bilag/135/index.htm
Disallow /Samling/20081/almdel/REU/bilag/698/index.htm
Disallow */baggrund.htm
Disallow /Dokumenter/Vis_efter_udvalg/UDU/Udvalgsbilag.aspx?pCommittee=UDU&pBilagCaseType=1&numberOfRecords=50&pageNr=1
Disallow /Dokumenter/Vis_efter_udvalg/UDU/Spoergsmaal.aspx?pCommittee=UDU&pQuestionType=-1
Disallow /Dokumenter/Vis_efter_udvalg/SUU/Udvalgsbilag.aspx?session=&committee=SUU&bilagCaseType=-1&startDate=20100801&endDate=20100901&sortColumn=&sortOrder=&startRecord=30&totalNumberOfRecords=&pCommittee=SUU&numberOfRecords=&pageNr=1
Disallow /Dokumenter/Vis_efter_udvalg/UDU/Udvalgsbilag.aspx?session=&committee=UDU&bilagCaseType=-1&startDate=20100901&endDate=20101001&sortColumn=&sortOrder=&startRecord=0&totalNumberOfRecords=&pCommittee=UDU&numberOfRecords=25&pageNr=1
Disallow /SendToFriendForm.aspx
Disallow /Dokumentkurv.aspx
Disallow /layouts/actionform.aspx
Disallow /search
Disallow /Search
Disallow /search.aspx
Disallow /Search.aspx
Disallow *master-*

googlebot

Rule Path
Disallow */baggrund.htm
Disallow /~search.aspx
Disallow /~Search.aspx
Disallow /search
Disallow /Search
Disallow *master-*

baiduspider
exabot/3.0
repparser/0.1+(python)
mozilla/4.0+(compatible;+msie+8.0;+windows+nt+6.0)+java/1.6.0_21
mozilla/5.0+(compatible;+mj12bot/v1.4.3;+http://www.majestic12.co.uk/bot.php?+)
psbot/0.1+(+http://www.picsearch.com/bot.html)
mozilla/5.0+(windows+nt+5.1;+rv:6.0.2)+gecko/20100101+firefox/6.0.2
mozilla/5.0+(compatible;+ezooms/1.0;+ezooms.bot@gmail.com)
cybeye.com/2.0+(compatible;+msie+9.0;+windows+nt+5.1;+trident/4.0;+gtb6.4)
mozilla/5.0+(compatible;+exabot/3.0;++http://www.exabot.com/go/robot)
ccbot/2.0
mozilla/5.0+(compatible;+008/0.83;+http://www.80legs.com/webcrawler.html;)+gecko/2008032620
addthis.com+robot+tech.support@clearspring.com
mozilla/5.0+(windows;+u;+windows+nt+5.1;+zh-cn;+rv:1.8.0.11)++firefox/1.5.0.11;+360spider
toscrawler/nutch-1.6+(http://www.toshiba.co.jp/rdc/about/crawl_info_en.htm;+'rdc-crawler+at+ml+dot+toshiba+dot+co+dot+jp')
mozilla/5.0+(windows+nt+6.1;+wow64)+applewebkit/537.36+(khtml,+like+gecko)+chrome/28.0.1500.72+safari/537.36
sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm
dotbot/1.1
mozilla/5.0+(compatible;+ahrefsbot/5.2;++http://ahrefs.com/robot/)
sitesucker+for+os+x/2.7.2
mozilla/5.0+(compatible;+semrushbot/2~bl;++http://www.semrush.com/bot.html)
mozilla/5.0+(macintosh;+intel+mac+os+x+10_13_6;++feeder.co)

Product Comment
sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm 07)
Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.ft.dk/sitemap.xml
sitemap https://sitemap.ft.dk/documentsitemap.aspx?type=dokument&session=all&site=www.ft.dk
sitemap https://sitemap.ft.dk/documentsitemap.aspx?type=raadsmoeder&session=all&site=www.ft.dk
sitemap https://sitemap.ft.dk/documentsitemap.aspx?type=kommissionsforslag&session=all&site=www.ft.dk
sitemap https://sitemap.ft.dk/documentsitemap.aspx?type=fil&session=all&site=www.ft.dk
sitemap https://sitemap.ft.dk/documentsitemap.aspx?type=eumoede&session=all&site=www.ft.dk
Back to top

Comments

  • Disallow crawler
Back to top