Log In Sign Up

sic.co.id
robots.txt

Robots Exclusion Standard data for sic.co.id

Resource Scan

Scan Details

Site Domain sic.co.id
Base Domain sic.co.id
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2025-11-08T18:56:51+00:00
Next Scan 2025-12-08T18:56:51+00:00

Last Successful Scan

Scanned2025-09-16T21:45:54+00:00
URL https://sic.co.id/robots.txt
Domain IPs 103.163.138.117
Response IP 103.163.138.117
Found Yes
Hash 09ffaf259614c43d6bfbee3dc79caa0191398fcbb4e90765acd22b978eabd60a
SimHash 613d42c5e4fa

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /private/
Disallow /includes/
Disallow /tmp/
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /backend/
Disallow /dashboard/
Disallow /wp-content/plugins/
Disallow /wp-includes/
Disallow /search/
Disallow /*?*query=
Disallow /*?*sort=
Disallow /*?*filter=
Disallow /*?*utm_*=
Disallow /error_log
Disallow /config.php
Disallow /.env
Disallow /.env.local
Disallow /*.sql$
Disallow /*.log$
Disallow /*.conf$
Disallow /.git/
Disallow /.htaccess
Disallow /composer.json
Disallow /package.json
Disallow /package-lock.json
Disallow /node_modules/
Disallow /dev/
Disallow /test/
Disallow /staging/

googlebot

Rule Path
Disallow /duplicate-content/

Other Records

Field Value
crawl-delay 5

bingbot

Rule Path
Disallow /duplicate-content/

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

ia_archiver

Rule Path
Disallow /private/
Disallow /admin/
Disallow /account/

googlebot-image

Rule Path
Allow /images/
Allow /assets/images/
Back to top

Other Records

Field Value
sitemap https://sic.co.id/sitemap.xml
Back to top

Comments

  • Global settings
  • Block sensitive files
  • Disallow non-production content
  • Specific instructions for major bots
  • Block aggressive bots
  • Block archive.org bot from archiving private content
  • Media bots
  • SEO resource links
Back to top