Log In Sign Up

bayarkilat.id
robots.txt

Robots Exclusion Standard data for bayarkilat.id

Resource Scan

Scan Details

Site Domain bayarkilat.id
Base Domain bayarkilat.id
Scan Status Ok
Last Scan2025-11-24T02:02:10+00:00
Next Scan 2025-12-24T02:02:10+00:00

Last Scan

Scanned2025-11-24T02:02:10+00:00
URL https://bayarkilat.id/robots.txt
Domain IPs 104.21.21.211, 172.67.200.90, 2606:4700:3032::6815:15d3, 2606:4700:3037::ac43:c85a
Response IP 172.67.200.90
Found Yes
Hash 4e43406bbb1ceb820c381494223760a91acdf0fc374c82be4d393eb692a8da1c
SimHash 61e6da37c93b

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /tag/
Disallow /category/
Disallow /?s=
Disallow /*?replytocom
Disallow /*/feed/
Disallow /*/trackback/

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

twitterbot

Rule Path
Allow /

facebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /
Disallow /*?utm_source=
Disallow /*?utm_medium=
Disallow /*?utm_campaign=
Disallow /cgi-bin/
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Disallow /wp-includes/
Disallow /readme.html
Disallow /license.txt
Back to top

Other Records

Field Value
sitemap https://www.bayarkilat.id/sitemap.xml
Back to top

Comments

  • Blokir halaman duplicate content atau yang tidak relevan untuk diindeks
  • Sitemap (pastikan path ini sesuai dengan plugin sitemap yang digunakan)
  • Izinkan bot meng-crawl semua konten
  • Tambahan untuk crawler media sosial (Facebook, Twitter, dll.)
  • Optimisasi khusus Googlebot Image
  • Jangan crawl URL parameter
  • Blokir akses ke file privasi atau server
Back to top