lms.bgauss.com
robots.txt

Robots Exclusion Standard data for lms.bgauss.com

Resource Scan

Scan Details

Site Domain lms.bgauss.com
Base Domain bgauss.com
Scan Status Ok
Last Scan2025-08-29T03:50:18+00:00
Next Scan 2025-09-12T03:50:18+00:00

Last Scan

Scanned2025-08-29T03:50:18+00:00
URL https://lms.bgauss.com/robots.txt
Domain IPs 13.35.202.106, 13.35.202.32, 13.35.202.44, 13.35.202.67
Response IP 13.35.202.106
Found Yes
Hash 98eb308a230d256b44c7d0444f377901253b76459c1fbb21997fd1580cb1a999
SimHash 4446890466f1

Groups

*

Rule Path
Allow /$
Disallow /myleads
Disallow /myleads/
Disallow /bin/
Disallow /obj/
Disallow /logs/
Disallow /*.config$
Disallow /*.json$

Other Records

Field Value
crawl-delay 10

Comments

  • Robots.txt for Lead2Retail
  • Generated to mitigate excessive crawler requests to protected endpoints
  • Allow all crawlers access to the landing page only
  • Block access to protected endpoints
  • Block access to sensitive directories
  • Crawl delay to reduce server load