Log In Sign Up

tlsoman.com
robots.txt

Robots Exclusion Standard data for tlsoman.com

Resource Scan

Scan Details

Site Domain tlsoman.com
Base Domain tlsoman.com
Scan Status Ok
Last Scan2025-11-05T04:51:57+00:00
Next Scan 2025-12-05T04:51:57+00:00

Last Scan

Scanned2025-11-05T04:51:57+00:00
URL https://tlsoman.com/robots.txt
Domain IPs 104.21.68.198, 172.67.198.40, 2606:4700:3033::ac43:c628, 2606:4700:3034::6815:44c6
Response IP 104.21.68.198
Found Yes
Hash b109bbf742efb921d70542c377d0a6558aa8f0866ad313dc84a228874742a0db
SimHash 25069f43a7a3

Groups

*

Rule Path
Allow /
Allow /css/
Allow /js/
Allow /images/
Disallow /api/
Disallow /config/
Disallow /logs/
Disallow /scripts/
Disallow /errors/
Disallow /.env
Disallow /.htaccess
Disallow /database_setup.sql
Disallow /SECURITY.md
Disallow /README.md
Disallow *.log$
Disallow *.tmp$
Disallow *.bak$
Disallow *~$
Disallow /.git/
Disallow /.svn/
Disallow /.DS_Store

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 3

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://tls-oman.com/sitemap.xml
Back to top

Comments

  • TLS Engineering - Robots.txt
  • Website: https://tls-oman.com
  • Updated: 2025-09-16
  • Allow all search engines to crawl the site
  • Allow access to main content
  • Disallow access to sensitive/admin areas
  • Disallow common bot traps and unnecessary files
  • Specific search engine instructions
  • Block aggressive crawlers
  • Sitemap location (will be updated when main site launches)
  • Additional directives for better SEO
  • Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_content
  • Host: https://tls-oman.com
Back to top