Log In Sign Up

toolskitpro.net
robots.txt

Robots Exclusion Standard data for toolskitpro.net

Resource Scan

Scan Details

Site Domain toolskitpro.net
Base Domain toolskitpro.net
Scan Status Ok
Last Scan2025-12-21T06:08:25+00:00
Next Scan 2025-12-28T06:08:25+00:00

Last Scan

Scanned2025-12-21T06:08:25+00:00
URL https://toolskitpro.net/robots.txt
Domain IPs 179.61.189.124, 191.101.228.137, 2a02:4780:15:57f2:592b:b266:5ae8:2eb, 2a02:4780:16:dc53:5665:f465:d080:28a2
Response IP 77.37.115.125
Found Yes
Hash b5d7ccb5f19cb552cd0f3aefeefe43151f21325e821713085496cd6e549c071d
SimHash 385e21536cb6

Groups

*

Rule Path Comment
Allow / Allow crawling of entire site except below disallowed folders/files
Disallow /components/ This folder contains components - block access
Disallow /admin/ Admin panel blocked
Disallow /cgi-bin/ Server CGI scripts blocked
Disallow /includes/ Includes folder blocked
Disallow /tmp/ Temporary files blocked
Disallow /private/ Private data blocked
Disallow /config/ Configuration files blocked
Disallow /node_modules/ Node.js modules folder blocked
Disallow /vendor/ Vendor libraries blocked
Disallow /*.inc$ Block all files ending with .inc
Disallow /*.pdf$ Block all PDF files

gptbot

Rule Path Comment
Disallow / Completely block GPTBot

chatgpt-user

Rule Path Comment
Disallow / Completely block ChatGPT-User

ccbot

Rule Path Comment
Disallow / Completely block CCBot

mj12bot

Rule Path Comment
Disallow / Completely block MJ12bot

dotbot

Rule Path Comment
Disallow / Completely block dotbot

semrushbot

Product Comment
semrushbot Semrush bot is allowed only limited access
Rule Path
Disallow /components/
Disallow /admin/
Disallow /cgi-bin/
Disallow /includes/
Disallow /tmp/
Disallow /private/
Disallow /config/
Disallow /node_modules/
Disallow /vendor/
Disallow /*.inc$
Disallow /*.pdf$

ahrefsbot

Product Comment
ahrefsbot Ahrefs bot is allowed only limited access
Rule Path
Disallow /components/
Disallow /admin/
Disallow /cgi-bin/
Disallow /includes/
Disallow /tmp/
Disallow /private/
Disallow /config/
Disallow /node_modules/
Disallow /vendor/
Disallow /*.inc$
Disallow /*.pdf$

googlebot-image

Product Comment
googlebot-image Allow Google image crawler to access images and tools folder
Rule Path Comment
Allow /assets/images/ -
Allow /tools/ Changed from /tools/*.php to /tools/

bingbot

Product Comment
bingbot Allow Bing image crawler to access images and tools folder
Rule Path Comment
Allow /assets/images/ -
Allow /tools/ Changed from /tools/*.php to /tools/
Back to top

Other Records

Field Value
sitemap https://toolskitpro.net/sitemap.xml
Back to top

Comments

  • ======================================================================
  • General rules for all bots
  • ======================================================================
  • Disallow: /assets/js/ # JS assets blocked from crawling
  • ======================================================================
  • Sitemap for all bots
  • ======================================================================
  • ======================================================================
  • Block specific bots completely
  • ======================================================================
  • ======================================================================
  • Semi-restricted bots with partial access
  • ======================================================================
  • Disallow: /assets/js/
  • Disallow: /assets/js/
  • ======================================================================
  • Allow media bots to crawl images and tools folder
  • ======================================================================
Back to top