Log In Sign Up

datascripmall.id
robots.txt

Robots Exclusion Standard data for datascripmall.id

Resource Scan

Scan Details

Site Domain datascripmall.id
Base Domain datascripmall.id
Scan Status Ok
Last Scan2025-11-21T21:48:40+00:00
Next Scan 2025-12-21T21:48:40+00:00

Last Scan

Scanned2025-11-21T21:48:40+00:00
URL https://datascripmall.id/robots.txt
Domain IPs 104.26.14.196, 104.26.15.196, 172.67.74.70, 2606:4700:20::681a:ec4, 2606:4700:20::681a:fc4, 2606:4700:20::ac43:4a46
Response IP 104.26.14.196
Found Yes
Hash 66b7b30e2d82a4ecec964c7da25d4693ddad1889dd6af0a1c25871a6e6f40c19
SimHash 41164b7662ed

Groups

mj12bot
semrushbot
yandex
yandexbot
zoominfobot
um-ln
mtrobot
safednsbot
bytespider
bitlybot
barkrowle
coccocbot-image
petalbot
dotbot
blexbot
aspiegelbot
baiduspider
trovitbot
mail.ru_bot
seznambot
yisouspider
zh-cn
slackbot
slackbot-linkexpanding
claudebot
amazonbot
fidget-spinner-bot
pinterestbot
vebot
awariobot
geedoproductsearch
skypeuripreview
gptbot
chatgpt-user
meta-externalagent
facebookexternalhit
chatgpt-user

Rule Path
Disallow /

*

Rule Path
Disallow /*/*/*/switch
Disallow /*/*/*/redirect
Disallow /?dir
Disallow /?dir=desc
Disallow /?dir=asc
Disallow /?limit=all
Disallow /?mode*
Disallow /?color*
Disallow /*?color*
Disallow /*%26color*
Disallow /?tax_invoice*
Disallow /*?tax_invoice*
Disallow /*%26tax_invoice*
Disallow /?vendor_region*
Disallow /*?vendor_region*
Disallow /*%26vendor_region*
Disallow /?brand*
Disallow /*?brand*
Disallow /*%26brand*
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /tag/
Disallow /review/
Disallow /*?
Disallow /api*
Disallow /graphql*
Disallow /index.php
Disallow /*/*/*/switch
Disallow /*/*/*/redirect
Disallow /*/elasticsuite
Disallow /*/catalogsearch
Disallow /*/*/*?srsltid*
Disallow /*/*/*/*/___store/*/___from_store
Disallow /*/*/*/switch/?*
Disallow /*/manual-book-download
Disallow /*/search/ajax/suggest/?*
Disallow /*/*/*/*?srsltid*

*

Rule Path
Allow /*?brand=*
Allow /*/*?brand=*
Disallow /api
Disallow /graphql*
Disallow /*?
Disallow /catalog
Disallow /catalogsearch
Disallow /wishlist
Disallow /admin
Disallow /checkout
Disallow /onestepcheckout
Disallow /customer
Disallow /review
Disallow /sendfriend
Disallow /enable-cookies
Disallow /LICENSE.txt
Disallow /LICENSE.html
Disallow /skin
Disallow /js
Disallow /directory
Disallow /404
Disallow /*?dir*
Disallow /*?dir*
Disallow /*?limit*
Disallow /*/*?mode*
Disallow /*/*?dir*
Disallow /*/*?limit*
Disallow /*/*?mode*
Disallow /catalogsearch/result*

*

Rule Path
Disallow /*?utm*
Disallow /*?*&utm*
Disallow /*/*?utm*
Disallow /*/*?*&utm*

*

Rule Path
Disallow /*.csv$
Disallow /*.xls$
Disallow /*.json$
Disallow /*.asa$
Disallow /*.asax$
Disallow /*.ascx$
Disallow /*.axd$
Disallow /*.backup$
Disallow /*.bak$
Disallow /*.bat$
Disallow /*.cdx$
Disallow /*.cer$
Disallow /*.cfg$
Disallow /*.cmd$
Disallow /*.com$
Disallow /*.config$
Disallow /*.conf$
Disallow /*.cs$
Disallow /*.csproj$
Disallow /*.csr$
Disallow /*.dat$
Disallow /*.db$
Disallow /*.dbf$
Disallow /*.dll$
Disallow /*.dos$
Disallow /*.htr$
Disallow /*.htw$
Disallow /*.ida$
Disallow /*.idc$
Disallow /*.idq$
Disallow /*.inc$
Disallow /*.ini$
Disallow /*.key$
Disallow /*.licx$
Disallow /*.lnk$
Disallow /*.log$
Disallow /*.mdb$
Disallow /*.old$
Disallow /*.pass$
Disallow /*.pdb$
Disallow /*.pol$
Disallow /*.printer$
Disallow /*.pwd$
Disallow /*.rdb$
Disallow /*.resources$
Disallow /*.resx$
Disallow /*.sql$
Disallow /*.swp$
Disallow /*.sys$
Disallow /*.vb$
Disallow /*.vbs$
Disallow /*.vbproj$
Disallow /*.vsdisco$
Disallow /*.webinfo$
Disallow /*.xsd$
Disallow /*.xsx$
Disallow /*.md$
Disallow /*.bzr$
Disallow /*.MD$
Disallow /*._darcs$
Disallow /*.git$
Disallow /*.ssh$
Disallow /*.svn$
Disallow /*.class$

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

instagram

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60
Back to top

Other Records

Field Value
sitemap https://datascripmall.id/media/sitemap-1-1.xml
sitemap https://datascripmall.id/media/sitemap-1-2.xml
sitemap https://datascripmall.id/media/sitemap-1-3.xml
Back to top

Comments

  • Block Bad Bot
  • Datascrip Rules
  • Filters
  • New Filters
  • Blocking CMS directories
  • Blocking duplicate content
  • Default Rules
  • Blocking UTM
  • Blocking extension. Hardening security purpose for defend from unknow hack tools
  • Crawl Delay For Decrease High Load
Back to top

Warnings

  • `host` is not a known field.
Back to top