Log In Sign Up

kreuzlingen24.ch
robots.txt

Robots Exclusion Standard data for kreuzlingen24.ch

Resource Scan

Scan Details

Site Domain kreuzlingen24.ch
Base Domain kreuzlingen24.ch
Scan Status Ok
Last Scan2024-11-09T11:00:55+00:00
Next Scan 2024-11-16T11:00:55+00:00

Last Scan

Scanned2024-11-09T11:00:55+00:00
URL https://kreuzlingen24.ch/robots.txt
Domain IPs 15.197.129.158, 75.2.43.161, 76.223.11.49, 99.83.217.1
Response IP 99.83.217.1
Found Yes
Hash 28cb47eaf9ee5162152306bb996b119049708870a7ae03a0c84f4ccb0c55dcc9
SimHash 7b76507aff73

Groups

ccbot

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

exabot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

crystalsemantics

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

blp_bbot/0.1

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

psbot
petalbot
mail.ru_bot
megaindex
yisouspider
bytespider
sogou web spider
sogou inst spider
proximic
admantx
seekport crawler
blexbot
mj12bot

Rule Path
Disallow /
Back to top

Comments

  • Special Areas of the Page
  • Special File Endings:
  • Disallow commercial bots
Back to top

Warnings

  • 3 invalid lines.
Back to top