levandong.com
robots.txt

Robots Exclusion Standard data for levandong.com

Resource Scan

Scan Details

Site Domain levandong.com
Base Domain levandong.com
Scan Status Ok
Last Scan2024-10-04T05:11:38+00:00
Next Scan 2024-10-11T05:11:38+00:00

Last Scan

Scanned2024-10-04T05:11:38+00:00
URL https://levandong.com/robots.txt
Redirect https://www.levandong.com/robots.txt
Redirect Domain www.levandong.com
Redirect Base levandong.com
Domain IPs 76.76.21.21
Redirect IPs 76.76.21.22, 76.76.21.9
Response IP 76.76.21.164
Found Yes
Hash aa11277ebd975d9ac5b39fb922f082f1a40d7179872ff1b36831f8b2f89ef526
SimHash 514adc402c10

Groups

*

Rule Path
Disallow /images/
Disallow /js/
Disallow /css/
Disallow /*offline/
Disallow /*404.html$
Disallow /*.md$

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

uptimerobot/2.0

Rule Path
Disallow /

ezooms robot

Rule Path
Disallow /

perl lwp

Rule Path
Disallow /

netestate ne crawler (+http://www.website-datenbank.de/)

Rule Path
Disallow /

wiseguys robot

Rule Path
Disallow /

turnitin robot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

pimonster

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

Other Records

Field Value
sitemap http://www.levandong.com/sitemap.xml

Comments

  • Block SISTRIX
  • Block Uptime robot
  • Block Ezooms Robot
  • Block Perl LWP
  • Block netEstate NE Crawler (+http://www.website-datenbank.de/)
  • Block WiseGuys Robot
  • Block Turnitin Robot
  • Block Heritrix
  • Block pricepi
  • OpenAI, ChatGPT
  • Google AI (Bard, etc)
  • Block common crawl
  • Facebook
  • Cohere.ai
  • Perplexity
  • Anthropic
  • Claudebot by Anthropic

Warnings

  • 4 invalid lines.