Log In Sign Up

toolref.com
robots.txt

Robots Exclusion Standard data for toolref.com

Resource Scan

Scan Details

Site Domain toolref.com
Base Domain toolref.com
Scan Status Ok
Last Scan2025-10-08T01:45:30+00:00
Next Scan 2025-10-15T01:45:30+00:00

Last Scan

Scanned2025-10-08T01:45:30+00:00
URL https://toolref.com/robots.txt
Redirect https://www.toolref.com/robots.txt
Redirect Domain www.toolref.com
Redirect Base toolref.com
Domain IPs 104.21.46.82, 172.67.136.116, 2606:4700:3030::6815:2e52, 2606:4700:3030::ac43:8874
Redirect IPs 104.21.46.82, 172.67.136.116, 2606:4700:3030::6815:2e52, 2606:4700:3030::ac43:8874
Response IP 172.67.136.116
Found Yes
Hash e940fbe2fc8f74b4eaf8202b9ad9a69b41a0e29bd4ec150fdd7519d65e1c5001
SimHash 65560db36584

Groups

*

Rule Path
Allow /
Disallow /cdn-cgi

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://www.toolref.com/sitemap.xml
Back to top

Comments

  • Allow all robots to access all content
  • Specific crawl-delay for major search engines (optional - remove if not needed)
  • This helps prevent server overload while ensuring good crawling
  • Sitemap location (update this URL when you create your sitemap)
  • Additional sitemaps if you have category-specific ones
  • Sitemap: https://www.toolref.com/tools-sitemap.xml
  • Sitemap: https://www.toolref.com/calculators-sitemap.xml
Back to top