Log In Sign Up

satudata.jakarta.go.id
robots.txt

Robots Exclusion Standard data for satudata.jakarta.go.id

Resource Scan

Scan Details

Site Domain satudata.jakarta.go.id
Base Domain jakarta.go.id
Scan Status Ok
Last Scan2025-09-21T03:20:42+00:00
Next Scan 2025-10-21T03:20:42+00:00

Last Scan

Scanned2025-09-21T03:20:42+00:00
URL https://satudata.jakarta.go.id/robots.txt
Domain IPs 103.209.7.64
Response IP 103.209.7.64
Found Yes
Hash 47c1919f36ca947cae984c91ea952b3537ed10c05ebf9e562347e5e6fd2486f1
SimHash 4c335ee1289c

Groups

*

Rule Path
Allow /
Disallow /api/
Disallow /apiV2/
Disallow /admin/
Disallow /login
Disallow /logout
Disallow /register
Disallow /password/
Disallow /email/
Disallow /verify/
Disallow /src/
Disallow /node_modules/
Disallow /dist/
Disallow /build/
Disallow /webpack/
Disallow /babel/
Disallow /eslint/
Disallow /jest/
Disallow /cypress/
Disallow /coverage/
Disallow /package.json
Disallow /package-lock.json
Disallow /yarn.lock
Disallow /webpack.config.js
Disallow /vue.config.js
Disallow /babel.config.js
Disallow /eslint.config.js
Disallow /jest.config.js
Disallow /cypress.config.js
Disallow /tsconfig.json
Disallow /vite.config.js
Disallow /tmp/
Disallow /cache/
Disallow /temp/
Disallow /.cache/
Disallow /.temp/
Disallow /hot/
Disallow /__webpack_hmr/
Disallow /%3A8080/
Disallow /%3A3000/
Disallow /%3A5173/
Disallow /%3A4173/
Allow /$
Allow /home
Allow /about
Allow /contact
Allow /help
Allow /faq
Allow /dataset/
Allow /dataset
Allow /open-data/
Allow /open-data
Allow /statistik/
Allow /statistik
Allow /visualisasi/
Allow /visualisasi
Allow /infografis/
Allow /infografis
Allow /artikel/
Allow /artikel
Allow /analisis/
Allow /analisis
Allow /produk-statistik/
Allow /produk-statistik
Allow /rubrik-statistik/
Allow /rubrik-statistik
Allow /search
Allow /search/
Allow /filter
Allow /filter/
Allow /kategori/
Allow /topik/
Allow /topik
Allow /organisasi/
Allow /kategori
Allow /organisasi
Allow /assets/
Allow /assets
Allow /css/
Allow /css
Allow /js/
Allow /js
Allow /img/
Allow /img
Allow /images/
Allow /images
Allow /fonts/
Allow /fonts
Allow /icons/
Allow /icons

googlebot

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /src/

Other Records

Field Value
crawl-delay 1

googlebot-image

Rule Path
Allow /
Disallow /api/
Disallow /admin/

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /src/

Other Records

Field Value
crawl-delay 2

yandex

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /src/

Other Records

Field Value
crawl-delay 2

baiduspider

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /src/

Other Records

Field Value
crawl-delay 3

duckduckbot

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /src/

Other Records

Field Value
crawl-delay 1

facebookexternalhit

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /src/

Other Records

Field Value
crawl-delay 1

twitterbot

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /src/

Other Records

Field Value
crawl-delay 1

linkedinbot

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /src/

Other Records

Field Value
crawl-delay 1

whatsapp

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /src/

Other Records

Field Value
crawl-delay 1

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

wget

Rule Path
Disallow /

curl

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

java

Rule Path
Disallow /

php

Rule Path
Disallow /

postmanruntime

Rule Path
Disallow /

insomnia

Rule Path
Disallow /

thunder client

Rule Path
Disallow /
Disallow /.env
Disallow /.git/
Disallow /.htaccess
Disallow /web.config
Disallow /robots.txt.bak
Disallow /sitemap.xml.bak
Disallow /.env.local
Disallow /.env.production
Disallow /.env.development
Disallow /vue.config.js
Disallow /vite.config.js
Disallow /babel.config.js
Disallow /eslint.config.js
Disallow /jest.config.js
Disallow /cypress.config.js
Disallow /tsconfig.json
Disallow /*.map
Disallow /dev/
Disallow /development/

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://satudata.jakarta.go.id/sitemap.xml
sitemap https://satudata.jakarta.go.id/sitemap-dynamic.xml
sitemap https://satudata.jakarta.go.id/sitemap-index.xml
sitemap https://satudata.jakarta.go.id/sitemap-dataset.xml
sitemap https://satudata.jakarta.go.id/sitemap-artikel.xml
sitemap https://satudata.jakarta.go.id/sitemap-infografis.xml
Back to top

Comments

  • Robots.txt untuk Satudata Jakarta Frontend
  • Production Environment - https://satudata.jakarta.go.id
  • Updated: 13 Desember 2024 - SEO Optimization Phase 3
  • ========================================
  • GLOBAL RULES - Berlaku untuk semua bot
  • ========================================
  • ========================================
  • BLOCKED PATHS - Jangan crawl path ini
  • ========================================
  • API dan Backend
  • Development dan Build Files
  • Configuration Files
  • Temporary dan Cache Files
  • Development Server
  • ========================================
  • ALLOWED PATHS - Prioritaskan crawl ini
  • ========================================
  • Halaman Utama dan Konten
  • Dataset dan Data
  • Search dan Filter
  • Assets yang Diperlukan
  • ========================================
  • BOT-SPECIFIC RULES
  • ========================================
  • Googlebot - Prioritaskan untuk SEO
  • Googlebot-Image - Untuk gambar
  • Bingbot
  • Yandex
  • Baiduspider
  • DuckDuckBot
  • Facebook External Hit
  • Twitter Bot
  • LinkedIn Bot
  • WhatsApp Bot
  • ========================================
  • BLOCKED BOTS - Jangan izinkan bot ini
  • ========================================
  • Scraping dan Bot Berbahaya
  • Development Tools
  • ========================================
  • SITEMAP DAN METADATA
  • ========================================
  • Sitemap utama
  • Sitemap dinamis dengan hreflang
  • Sitemap index untuk multiple sitemap
  • Sitemap untuk dataset
  • Sitemap untuk artikel
  • Sitemap untuk infografis
  • ========================================
  • PERFORMANCE OPTIMIZATION
  • ========================================
  • Rate limiting untuk semua bot
  • Host directive untuk canonical domain
  • ========================================
  • SECURITY HEADERS (via robots.txt)
  • ========================================
  • Mencegah indexing dari file sensitif
  • ========================================
  • VUE.JS SPECIFIC RULES
  • ========================================
  • Block Vue.js development files
  • Block source maps in production
  • Block development assets
  • ========================================
  • FOOTER
  • ========================================
  • Last updated: 2024
  • Contact: admin@satudata.jakarta.go.id
  • Version: 2.0 - Frontend Vue.js
Back to top

Warnings

  • `host` is not a known field.
Back to top