thelocal.de
robots.txt

Robots Exclusion Standard data for thelocal.de

Archived Snapshots

Resource Scan

Scan Details

Site Domain	thelocal.de
Base Domain	thelocal.de
Scan Status	Ok
Last Scan	2024-11-03T17:25:17+00:00
Next Scan	2024-11-10T17:25:17+00:00

Last Scan

Scanned	2024-11-03T17:25:17+00:00
URL	https://thelocal.de/robots.txt
Redirect	https://www.thelocal.de/robots.txt
Redirect Domain	www.thelocal.de
Redirect Base	thelocal.de
Domain IPs	104.18.8.180, 104.18.9.180, 2606:4700::6812:8b4, 2606:4700::6812:9b4
Redirect IPs	104.18.8.180, 104.18.9.180, 2606:4700::6812:8b4, 2606:4700::6812:9b4
Response IP	104.18.9.180
Found	Yes
Hash	97db0976384078c5ef42acf2467d9ded6e7c7bd46ef5237c9e734833b9aa0439
SimHash	5d1950d08f80

Groups

*

Rule	Path
Disallow	.js
Disallow	.css
Disallow	/200*
Disallow	/2010*
Disallow	/2011*
Disallow	/2012*
Disallow	/2013*
Disallow	/amp*
Disallow	/fonts*
Disallow	/index.php/
Disallow	cms.
Disallow	medium.
Disallow	medium2.
Disallow	discuss.

Rule

Path

Disallow

*.js*

Disallow

*.css*

Disallow

/200*

Disallow

/2010*

Disallow

/2011*

Disallow

/2012*

Disallow

/2013*

Disallow

/amp*

Disallow

/fonts*

Disallow

/index.php/

Disallow

*cms.*

Disallow

*medium.*

Disallow

*medium2.*

Disallow

*discuss.*

gptbot

Rule	Path
Disallow	/

Rule

Path

Disallow

chatgpt-user

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

archive.org_bot

Rule	Path
Disallow	/

Rule

Path

Disallow

arquivo-web-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

amazonbot

Rule	Path
Disallow	/

Rule

Path

Disallow

anthropic-ai

Rule	Path
Disallow	/

Rule

Path

Disallow

awariorssbot
awariosmartbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bytespider

Rule	Path
Disallow	/

Rule

Path

Disallow

coccocbot-web

Rule	Path
Disallow	/

Rule

Path

Disallow

claudebot

Rule	Path
Disallow	/

Rule

Path

Disallow

claude-web

Rule	Path
Disallow	/

Rule

Path

Disallow

dataforseobot

Rule	Path
Disallow	/

Rule

Path

Disallow

facebookbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ia_archiver

Rule	Path
Disallow	/

Rule

Path

Disallow

jenkersbot

Rule	Path
Disallow	/

Rule

Path

Disallow

mojeek

Rule	Path
Disallow	/

Rule

Path

Disallow

moodlebot

Rule	Path
Disallow	/

Rule

Path

Disallow

magpie-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

newsnow

Rule	Path
Disallow	/

Rule

Path

Disallow

omgili

Rule	Path
Disallow	/

Rule

Path

Disallow

omgilibot

Rule	Path
Disallow	/

Rule

Path

Disallow

peer39_crawler
peer39_crawler/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

perplexitybot

Rule	Path
Disallow	/

Rule

Path

Disallow

petalbot

Rule	Path
Disallow	/

Rule

Path

Disallow

seekr

Rule	Path
Disallow	/

Rule

Path

Disallow

seznambot

Rule	Path
Disallow	/

Rule

Path

Disallow

turnitinbot

Rule	Path
Disallow	/

Rule

Path

Disallow

youbot

Rule	Path
Disallow	/

Rule

Path

Disallow

yacy

Rule	Path
Disallow	/

Rule

Path

Disallow

yandex

Rule	Path
Disallow	/

Rule

Path

Disallow

Other Records

Field	Value
sitemap	https://www.thelocal.de/sitemap/de/news.xml
sitemap	https://www.thelocal.de/sitemap/index.xml

Field

Value

sitemap

https://www.thelocal.de/sitemap/de/news.xml

sitemap

https://www.thelocal.de/sitemap/index.xml

thelocal.derobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

gptbot

chatgpt-user

ccbot

ahrefsbot

archive.org_bot

arquivo-web-crawler

amazonbot

anthropic-ai

awariorssbotawariosmartbot

bytespider

coccocbot-web

claudebot

claude-web

dataforseobot

facebookbot

ia_archiver

jenkersbot

mojeek

moodlebot

magpie-crawler

newsnow

omgili

omgilibot

peer39_crawlerpeer39_crawler/1.0

perplexitybot

petalbot

seekr

seznambot

turnitinbot

youbot

yacy

yandex

Other Records

thelocal.de
robots.txt

awariorssbot
awariosmartbot

peer39_crawler
peer39_crawler/1.0