Log In Sign Up

lifi.mx
robots.txt

Robots Exclusion Standard data for lifi.mx

Resource Scan

Scan Details

Site Domain lifi.mx
Base Domain lifi.mx
Scan Status Ok
Last Scan2026-01-06T02:11:02+00:00
Next Scan 2026-01-13T02:11:02+00:00

Last Scan

Scanned2026-01-06T02:11:02+00:00
URL https://lifi.mx/robots.txt
Domain IPs 137.66.54.119, 2a09:8280:1::a:5715:0
Response IP 137.66.54.119
Found Yes
Hash 5f19480ca4c5616c62fd6a5ef75f4d7e69552731c5db2bb8dd7ced9298e2b636
SimHash a31818841762

Groups

*

Rule Path
Disallow /*?*search=
Allow /listing/
Allow /products/
Allow /discounts
Allow /cuponeras
Back to top

Other Records

Field Value
sitemap https://lifi.mx/sitemap/sitemap.xml.gz
Back to top

Comments

  • See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • Allow crawlers to fetch paginated results so they can follow links;
  • pagination pages still emit meta noindex,follow to avoid index bloat.
  • Disallow search result pages (low value, saves crawl budget)
  • Allow main content
Back to top