landbou.com
robots.txt

Robots Exclusion Standard data for landbou.com

Resource Scan

Scan Details

Site Domain landbou.com
Base Domain landbou.com
Scan Status Ok
Last Scan2024-06-26T14:05:25+00:00
Next Scan 2024-07-03T14:05:25+00:00

Last Scan

Scanned2024-06-26T14:05:25+00:00
URL https://landbou.com/robots.txt
Redirect https://www.landbou.com/robots.txt
Redirect Domain www.landbou.com
Redirect Base landbou.com
Domain IPs 104.18.43.248, 172.64.144.8, 2606:4700:4400::6812:2bf8, 2606:4700:4400::ac40:9008
Redirect IPs 104.18.43.248, 172.64.144.8, 2606:4700:4400::6812:2bf8, 2606:4700:4400::ac40:9008
Response IP 104.18.43.248
Found Yes
Hash c9235e042233d3a62f433f149ac439151d1df2d301185c6246ce6cf8759f836d
SimHash 741c5951edb1

Groups

*

Rule Path
Allow /
Allow */cricket/test/
Allow */cricketworldcup2019/Test/
Disallow /toets/
Disallow */toets/*
Disallow /test/
Disallow */_test/*
Disallow */testpolar/*
Disallow */test/*
Disallow /xArchive/Archive/Illegal-liquor-export-20010319
Disallow /.well-known/
Disallow /assetlinks.json

twitterbot

Rule Path
Allow /

ia_archiver

Rule Path
Disallow /BreakingNewsSms

mauibot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

dataprovider-com

Rule Path
Disallow /

dcrawl

Rule Path
Disallow /

httrack

Rule Path
Disallow /

httrack-3-0

Rule Path
Disallow /

metainspector

Rule Path
Disallow /

newspaper

Rule Path
Disallow /

nutch

Rule Path
Disallow /

offline-explorer

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

hypestat

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

screaming-frog-seo-spider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.landbou.com/sitemap

Comments

  • AI Assistants
  • AI Data Scrapers
  • AI Search Crawlers
  • Scrapers
  • SEO Crawlers
  • Undocumented AI Agents

Warnings

  • 2 invalid lines.