Log In Sign Up

dargu.co.il
robots.txt

Robots Exclusion Standard data for dargu.co.il

Resource Scan

Scan Details

Site Domain dargu.co.il
Base Domain dargu.co.il
Scan Status Ok
Last Scan2026-02-26T08:59:00+00:00
Next Scan 2026-03-05T08:59:00+00:00

Last Scan

Scanned2026-02-26T08:59:00+00:00
URL https://dargu.co.il/robots.txt
Domain IPs 104.21.40.4, 172.67.172.217, 2606:4700:3034::ac43:acd9, 2606:4700:3037::6815:2804
Response IP 104.21.40.4
Found Yes
Hash 55f519cb7b35b76262475023aa71fcc3d2d2a08b277c1a75e9722a8a3fe2b5ae
SimHash 05045bd1a581

Groups

*

Rule Path
Allow /
Allow /index.html
Allow /pages/
Allow /css/
Allow /js/
Allow /images/
Disallow /admin-reports.html
Disallow /pages/user-profile.html
Disallow /*.env
Disallow /*.config
Disallow /build.sh

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://dargu.co.il/sitemaps/sitemap.xml
Back to top

Comments

  • Robots.txt for dargu.co.il
  • Hebrew Professor Rating Platform
  • Allow all main pages
  • Block admin and sensitive areas
  • Sitemap location
  • Crawl delay (optional - be nice to servers)
  • Popular search engines
Back to top