thelocal.fr
robots.txt

Robots Exclusion Standard data for thelocal.fr

Archived Snapshots

Resource Scan

Scan Details

Site Domain	thelocal.fr
Base Domain	thelocal.fr
Scan Status	Ok
Last Scan	2024-09-27T08:25:04+00:00
Next Scan	2024-10-04T08:25:04+00:00

Last Scan

Scanned	2024-09-27T08:25:04+00:00
URL	https://thelocal.fr/robots.txt
Redirect	https://www.thelocal.fr/robots.txt
Redirect Domain	www.thelocal.fr
Redirect Base	thelocal.fr
Domain IPs	104.18.12.135, 104.18.13.135, 2606:4700::6812:c87, 2606:4700::6812:d87
Redirect IPs	104.18.12.135, 104.18.13.135, 2606:4700::6812:c87, 2606:4700::6812:d87
Response IP	104.18.13.135
Found	Yes
Hash	3da018a4cdbb7db1bfa0c8f0efa477488bace32eec5dbe28404aaf868cc3de8d
SimHash	dd0d40508f80

Groups

*

Rule	Path
Disallow	.js
Disallow	.css
Disallow	/200*
Disallow	/2010*
Disallow	/2011*
Disallow	/2012*
Disallow	/2013*
Disallow	/amp*
Disallow	/fonts*
Disallow	/index.php/
Disallow	cms.
Disallow	medium.
Disallow	medium2.
Disallow	discuss.

Rule

Path

Disallow

*.js*

Disallow

*.css*

Disallow

/200*

Disallow

/2010*

Disallow

/2011*

Disallow

/2012*

Disallow

/2013*

Disallow

/amp*

Disallow

/fonts*

Disallow

/index.php/

Disallow

*cms.*

Disallow

*medium.*

Disallow

*medium2.*

Disallow

*discuss.*

gptbot

Rule	Path
Disallow	/

Rule

Path

Disallow

chatgpt-user

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

archive.org_bot

Rule	Path
Disallow	/

Rule

Path

Disallow

arquivo-web-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

amazonbot

Rule	Path
Disallow	/

Rule

Path

Disallow

anthropic-ai

Rule	Path
Disallow	/

Rule

Path

Disallow

awariorssbot
awariosmartbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bytespider

Rule	Path
Disallow	/

Rule

Path

Disallow

coccocbot-web

Rule	Path
Disallow	/

Rule

Path

Disallow

claudebot

Rule	Path
Disallow	/

Rule

Path

Disallow

claude-web

Rule	Path
Disallow	/

Rule

Path

Disallow

dataforseobot

Rule	Path
Disallow	/

Rule

Path

Disallow

facebookbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ia_archiver

Rule	Path
Disallow	/

Rule

Path

Disallow

jenkersbot

Rule	Path
Disallow	/

Rule

Path

Disallow

mojeek

Rule	Path
Disallow	/

Rule

Path

Disallow

moodlebot

Rule	Path
Disallow	/

Rule

Path

Disallow

magpie-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

newsnow

Rule	Path
Disallow	/

Rule

Path

Disallow

omgili

Rule	Path
Disallow	/

Rule

Path

Disallow

omgilibot

Rule	Path
Disallow	/

Rule

Path

Disallow

peer39_crawler
peer39_crawler/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

perplexitybot

Rule	Path
Disallow	/

Rule

Path

Disallow

petalbot

Rule	Path
Disallow	/

Rule

Path

Disallow

seekr

Rule	Path
Disallow	/

Rule

Path

Disallow

seznambot

Rule	Path
Disallow	/

Rule

Path

Disallow

turnitinbot

Rule	Path
Disallow	/

Rule

Path

Disallow

youbot

Rule	Path
Disallow	/

Rule

Path

Disallow

yacy

Rule	Path
Disallow	/

Rule

Path

Disallow

yandex

Rule	Path
Disallow	/

Rule

Path

Disallow

Other Records

Field	Value
sitemap	https://www.thelocal.fr/sitemap/fr/news.xml
sitemap	https://www.thelocal.fr/sitemap/index.xml

Field

Value

sitemap

https://www.thelocal.fr/sitemap/fr/news.xml

sitemap

https://www.thelocal.fr/sitemap/index.xml

thelocal.frrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

gptbot

chatgpt-user

ccbot

ahrefsbot

archive.org_bot

arquivo-web-crawler

amazonbot

anthropic-ai

awariorssbotawariosmartbot

bytespider

coccocbot-web

claudebot

claude-web

dataforseobot

facebookbot

ia_archiver

jenkersbot

mojeek

moodlebot

magpie-crawler

newsnow

omgili

omgilibot

peer39_crawlerpeer39_crawler/1.0

perplexitybot

petalbot

seekr

seznambot

turnitinbot

youbot

yacy

yandex

Other Records

thelocal.fr
robots.txt

awariorssbot
awariosmartbot

peer39_crawler
peer39_crawler/1.0