Log In Sign Up

lefkada-rentals.com
robots.txt

Robots Exclusion Standard data for lefkada-rentals.com

Resource Scan

Scan Details

Site Domain lefkada-rentals.com
Base Domain lefkada-rentals.com
Scan Status Ok
Last Scan2024-05-28T14:10:55+00:00
Next Scan 2024-06-04T14:10:55+00:00

Last Scan

Scanned2024-05-28T14:10:55+00:00
URL https://lefkada-rentals.com/robots.txt
Domain IPs 104.26.4.127, 104.26.5.127, 172.67.69.157, 2606:4700:20::681a:47f, 2606:4700:20::681a:57f, 2606:4700:20::ac43:459d
Response IP 172.67.69.157
Found Yes
Hash f2cd9f72f29bbbbdfeb22d593dc46668787224250113f0cc7ed55cc955dad6fd
SimHash 22055981c550

Groups

googlebot

Rule Path
Disallow

mediapartners-google

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

robozilla

Rule Path
Disallow

slurp

Rule Path
Disallow

bingbot

Rule Path
Disallow

sistrix

Rule Path
Disallow

twitterbot

Rule Path
Disallow

applebot

Rule Path
Disallow

yandexbot

Rule Path
Disallow

siteliner

Rule Path
Disallow

ubersuggest

Rule Path
Disallow

ubersuggestbot

Rule Path
Disallow

ia_archiver

Rule Path
Disallow

exabot

Rule Path
Disallow

ccbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 2

gptbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 2

google-inspectiontool

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow

semrushbot

Rule Path
Disallow

rogerbot

Rule Path
Disallow

mj12bot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 2

oncrawl

Rule Path
Disallow

Other Records

Field Value
crawl-delay 2

*

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://s3-eu-west-1.amazonaws.com/lefkada-villas/sitemap/sitemap.xml.gz
Back to top

Comments

  • See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • User-Agent: *
  • Disallow: /
  • Allow only Googlebot, Mediapartners, Adsbot, DMOZ, Yahoo, Optimizr, Twitter, Siteliner, neilpatel dublicate content check,
  • Sitemap: #https://www.lefkada-rentals.com/sitemap.xml
Back to top