tdg.ch
robots.txt

Robots Exclusion Standard data for tdg.ch

Resource Scan

Scan Details

Site Domain tdg.ch
Base Domain tdg.ch
Scan Status Ok
Last Scan2024-11-13T14:34:13+00:00
Next Scan 2024-11-20T14:34:13+00:00

Last Scan

Scanned2024-11-13T14:34:13+00:00
URL https://tdg.ch/robots.txt
Redirect https://www.tdg.ch/robots.txt
Redirect Domain www.tdg.ch
Redirect Base tdg.ch
Domain IPs 13.226.2.114, 13.226.2.43, 13.226.2.46, 13.226.2.82, 2600:9000:21f8:2000:e:5a66:ac0:93a1, 2600:9000:21f8:4200:e:5a66:ac0:93a1, 2600:9000:21f8:5200:e:5a66:ac0:93a1, 2600:9000:21f8:6c00:e:5a66:ac0:93a1, 2600:9000:21f8:7e00:e:5a66:ac0:93a1, 2600:9000:21f8:ea00:e:5a66:ac0:93a1, 2600:9000:21f8:f600:e:5a66:ac0:93a1, 2600:9000:21f8:fa00:e:5a66:ac0:93a1
Redirect IPs 2600:9000:2795:0:e:5a66:ac0:93a1, 2600:9000:2795:2200:e:5a66:ac0:93a1, 2600:9000:2795:2400:e:5a66:ac0:93a1, 2600:9000:2795:2600:e:5a66:ac0:93a1, 2600:9000:2795:5000:e:5a66:ac0:93a1, 2600:9000:2795:ce00:e:5a66:ac0:93a1, 2600:9000:2795:d000:e:5a66:ac0:93a1, 2600:9000:2795:f800:e:5a66:ac0:93a1, 3.164.85.11, 3.164.85.59, 3.164.85.64, 3.164.85.95
Response IP 52.85.49.111
Found Yes
Hash 03ce605111b7628e02dd7d1c13ca50fbde668d8a35b941ef9393b3dc4087d4b2
SimHash 50168348db33

Groups

psbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

proximic

Rule Path
Disallow /

admantx

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

applebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

*

Rule Path
Allow /

Other Records

Field Value
sitemap https://www.tdg.ch/sitemaps/sitemapindex.xml
sitemap https://www.tdg.ch/sitemaps/news.xml

Comments

  • Disallow commercial bots to prevent ad fraud, see DISC-2117
  • Allow crawling for other bots

Warnings

  • 2 invalid lines.