gold.de
robots.txt

Robots Exclusion Standard data for gold.de

Resource Scan

Scan Details

Site Domain gold.de
Base Domain gold.de
Scan Status Ok
Last Scan2024-10-04T05:52:56+00:00
Next Scan 2024-10-11T05:52:56+00:00

Last Scan

Scanned2024-10-04T05:52:56+00:00
URL https://gold.de/robots.txt
Redirect https://www.gold.de/robots.txt
Redirect Domain www.gold.de
Redirect Base gold.de
Domain IPs 172.66.40.97, 172.66.43.159, 2606:4700:3108::ac42:2861, 2606:4700:3108::ac42:2b9f
Redirect IPs 172.66.40.97, 172.66.43.159, 2606:4700:3108::ac42:2861, 2606:4700:3108::ac42:2b9f
Response IP 172.66.43.159
Found Yes
Hash e303e8931ddda70c1c54c057acf6d8b35cd71c45634c16eb22497c6e3936b775
SimHash 33182144ebd1

Groups

*

Rule Path
Disallow /data/
Disallow /ajax/
Disallow /cronjob/
Disallow /goto/
Disallow /cdn/min/
Disallow /tools/rechner/

magpie-crawler

Rule Path
Disallow /

exabot

Rule Path
Disallow /

thesubot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yetibot

Rule Path
Disallow /

yeti/0.01

Rule Path
Disallow /

nhn/1noon

Rule Path
Disallow /

nhn

Rule Path
Disallow /

discobot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

cityreview

Rule Path
Disallow /

mj12bot

Rule Path
Disallow

suggybot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

iisbot

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

custo 3.0 (www.netwu.com)

Rule Path
Disallow /

custo 3 (netwu.com)

Rule Path
Disallow /

scrapy*

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

newspointbot/1.0 (http://www.netsprint.eu/)

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

Warnings

  • 4 invalid lines.