dotnetmastery.com
robots.txt

Robots Exclusion Standard data for dotnetmastery.com

Resource Scan

Scan Details

Site Domain dotnetmastery.com
Base Domain dotnetmastery.com
Scan Status Ok
Last Scan2026-01-18T08:23:39+00:00
Next Scan 2026-02-17T08:23:39+00:00

Last Scan

Scanned2026-01-18T08:23:39+00:00
URL https://dotnetmastery.com/robots.txt
Domain IPs 20.118.48.3
Response IP 20.118.48.3
Found Yes
Hash 4fe349106bef5693ea4fba9ff37df813c7af991cafe0df5756d44c37b6d5ca08
SimHash 61859ac6f706

Groups

*

Rule Path
Allow /
Allow /Home/Index
Allow /Home/Details
Allow /Blog/
Allow /Course/
Disallow /Identity/
Disallow /Admin/
Disallow /api/
Disallow /lib/
Disallow /css/
Disallow /js/
Disallow /*.json$
Allow /images/logo.png
Allow /images/favicon*.png
Allow /images/head2.png
Allow /images/courses/

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

googlebot-image

Rule Path
Allow /images/

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

Other Records

Field Value
sitemap https://www.dotnetmastery.com/sitemap.xml

Comments

  • DotNetMastery Robots.txt
  • Updated: 2024
  • Important pages for crawlers
  • Disallow admin areas
  • Disallow unnecessary resources
  • Allow important images for Google Image Search
  • Crawl-delay (be polite)
  • Sitemaps
  • Specific instructions for major search engines