Log In Sign Up

nnp.de
robots.txt

Robots Exclusion Standard data for nnp.de

Resource Scan

Scan Details

Site Domain nnp.de
Base Domain nnp.de
Scan Status Ok
Last Scan2024-09-19T20:06:19+00:00
Next Scan 2024-09-26T20:06:19+00:00

Last Scan

Scanned2024-09-19T20:06:19+00:00
URL http://nnp.de/robots.txt
Redirect https://www.mittelhessen.de/robots.txt
Redirect Domain www.mittelhessen.de
Redirect Base mittelhessen.de
Domain IPs 77.235.162.63
Redirect IPs 52.57.212.58, 52.59.99.13
Response IP 52.57.212.58
Found Yes
Hash eb6f04db4d68930125b23e5379038e75d0c0a5972130a606142dcb6568b24d2e
SimHash 281cd3e0cdb8

Groups

semrushbot
ahrefsbot
dataforseobot
uptimerobot
relemindbot
applebot
amazonbot
seekportbot
yandex
claudebot
claude-web
perplexitybot
rytebot
dotbot
dataforseobot
trendictionbot
mj12bot
paqlebot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

*

Rule Path
Disallow /*?egy_cid=*

*

Rule Path
Disallow /api/
Disallow /archive/
Disallow /na/
Disallow /service/profil/
Back to top

Other Records

Field Value
sitemap https://www.mittelhessen.de/index-sitemap.xml
Back to top

Comments

  • block unwanted bots
  • ChatGPT Plugins
  • Common Crawl
  • Bard
  • Meta’s bot that crawls public web pages to improve language models for their speech recognition technology
  • Used for several purposes, apparently also selling crawled data to LLM companies (http://omgili.com/crawler.html)
  • urls with ?egy_cid= are not crawled
Back to top