Log In Sign Up

nordicabode.com
robots.txt

Robots Exclusion Standard data for nordicabode.com

Resource Scan

Scan Details

Site Domain nordicabode.com
Base Domain nordicabode.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-11-07T13:20:12+00:00
Next Scan 2025-11-21T13:20:12+00:00

Last Successful Scan

Scanned2025-10-15T00:21:47+00:00
URL https://nordicabode.com/robots.txt
Domain IPs 76.76.21.21
Response IP 76.76.21.21
Found Yes
Hash 457edbe33b8f22075d61435900e7f03eec93ae0ff63d934f4eaed281adea26ef
SimHash 6c3aca10e781

Groups

googlebot
bingbot
slurp
duckduckbot
baiduspider
yandexbot
facebot
ia_archiver

Rule Path
Allow /

ahrefsbot
semrushbot
dotbot
mj12bot
blexbot
dataforseobot
petalbot
megaindex
seekport
zoominfobot
go-http-client
python-requests
curl
wget

Rule Path
Disallow /

*

Rule Path
Disallow /

googlebot
bingbot
slurp
duckduckbot
baiduspider
yandexbot

Rule Path
Disallow /cart
Disallow /checkout
Disallow /api/
Back to top

Other Records

Field Value
sitemap https://${NEXT_PUBLIC_VERCEL_URL}/sitemap.xml
Back to top

Comments

  • Allow legitimate search engine crawlers
  • Block price scrapers and spam crawlers
  • Block all other bots by default
  • Allow legitimate bots to access everything except sensitive pages
  • Sitemap location
Back to top