mithaly.sa
robots.txt

Robots Exclusion Standard data for mithaly.sa

Resource Scan

Scan Details

Site Domain mithaly.sa
Base Domain mithaly.sa
Scan Status Ok
Last Scan2025-09-27T05:09:38+00:00
Next Scan 2025-10-11T05:09:38+00:00

Last Scan

Scanned2025-09-27T05:09:38+00:00
URL https://mithaly.sa/robots.txt
Domain IPs 104.21.91.46, 172.67.166.185, 2606:4700:3035::6815:5b2e, 2606:4700:3035::ac43:a6b9
Response IP 172.67.166.185
Found Yes
Hash 5bc045c47f6ffdf9090bd18bd33ad4a41cddb320efb11bc09b6277c797e0edd7
SimHash 21515990e6b3

Groups

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

googlebot-mobile

Rule Path
Disallow

msnbot

Rule Path
Disallow

slurp

Rule Path
Disallow

teoma

Rule Path
Disallow

gigabot

Rule Path
Disallow

robozilla

Rule Path
Disallow

nutch

Rule Path
Disallow

ia_archiver

Rule Path
Disallow

baiduspider

Rule Path
Disallow

yahoo-mmcrawler

Rule Path
Disallow

psbot

Rule Path
Disallow

yahoo-blogs/v3.9

Rule Path
Disallow

adsbot-google

Rule Path
Disallow /*/cellucor*

adsbot-google-mobile

Rule Path
Disallow /*/cellucor*

*

Rule Path
Disallow /*/app/
Disallow /*/bootstrap/
Disallow /*/config/
Disallow /*/database/
Disallow /*/modules/
Disallow /*/node_modules/
Disallow /*/resources/
Disallow /*/routes/
Disallow /*/themes/
Disallow /*/vendor/
Disallow /*/index.php
Disallow /*/account
Disallow /*/account/*
Disallow /*/checkout/*
Disallow /cgi-bin/
Disallow /debugbar*
Disallow /suggestions
Disallow /*/.php$
Disallow /*/?SID=*
Disallow /*/?___SID=*
Disallow /*/*height%3D
Disallow /*/products?query=*

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://drnutrition.com/sitemap.xml
sitemap https://drnutrition.com/images-sitemap.xml

Comments

  • Sitemaps
  • Seacrch Engine
  • GoogleAds - block C4 pages
  • GoogleAds-Mobile - block C4 pages
  • Directories
  • Paths (Clean URLs)
  • Clear Category Sort
  • Block crawling software

Warnings

  • `host` is not a known field.