Log In Sign Up

wildernessdestinations.com
robots.txt

Robots Exclusion Standard data for wildernessdestinations.com

Resource Scan

Scan Details

Site Domain wildernessdestinations.com
Base Domain wildernessdestinations.com
Scan Status Ok
Last Scan2024-09-21T14:57:31+00:00
Next Scan 2024-10-21T14:57:31+00:00

Last Scan

Scanned2024-09-21T14:57:31+00:00
URL https://wildernessdestinations.com/robots.txt
Redirect https://www.wildernessdestinations.com/robots.txt
Redirect Domain www.wildernessdestinations.com
Redirect Base wildernessdestinations.com
Domain IPs 51.140.210.99
Redirect IPs 152.199.39.108, 2606:2800:247:1cb7:261b:1f9c:2074:3c
Response IP 152.199.39.108
Found Yes
Hash 3868febf86a715710e367670422c0de1093c4f6b879a90f677dbb36303dcc5d9
SimHash 6e8ede50c593

Groups

*

Rule Path
Disallow /umbraco
Disallow /Umbraco
Disallow /modules/*
Disallow /media/blog/*
Disallow /search

alexibot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

xenu's

Rule Path
Disallow /

sogou

Rule Path
Disallow /

seokicks-robots

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

uptimerobot/2.0

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

bingbot

Rule Path
Disallow /modules/*
Disallow /media/blog/*

Other Records

Field Value
crawl-delay 10

slurp

Rule Path
Disallow /modules/*
Disallow /media/blog/*

Other Records

Field Value
crawl-delay 50

duckduckbot

Rule Path
Disallow /modules/*
Disallow /media/blog/*

Other Records

Field Value
crawl-delay 20
Back to top

Other Records

Field Value
sitemap https://www.wildernessdestinations.com/sitemap.xml
sitemap https://www.wildernessdestinations.com/images-sitemap.xml
Back to top

Comments

  • Block not wanted search engines
  • Setting up crawl-delay for specific user-agents
Back to top