thesmartwallet.com
robots.txt

Robots Exclusion Standard data for thesmartwallet.com

Resource Scan

Scan Details

Site Domain thesmartwallet.com
Base Domain thesmartwallet.com
Scan Status Ok
Last Scan2024-07-04T04:48:14+00:00
Next Scan 2024-07-11T04:48:14+00:00

Last Scan

Scanned2024-07-04T04:48:14+00:00
URL https://thesmartwallet.com/robots.txt
Domain IPs 104.26.10.156, 104.26.11.156, 172.67.70.113, 2606:4700:20::681a:a9c, 2606:4700:20::681a:b9c, 2606:4700:20::ac43:4671
Response IP 104.26.11.156
Found Yes
Hash f5a75fa6b6672b6bcbcb30f2b6873731e4a1e8a01ec1e52af04121d97a89c921
SimHash a4f14d7b0ef4

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-content/plugins
Disallow /wp-content/themes
Disallow /tag/*
Disallow /attachment/*
Disallow /*?attachment_id=*
Disallow /advertiser/*
Disallow /text-blocks/*
Disallow /ctm/*
Disallow /offer/*
Disallow /campaign/*
Disallow /stack/*
Allow /*.js$
Allow /*.css$

Comments

  • Sits in the root, does not actually live where it does in the Docker container
  • It's important for SEO that .css and .js files are still crawlable