Log In Sign Up

defkey.com
robots.txt

Robots Exclusion Standard data for defkey.com

Resource Scan

Scan Details

Site Domain defkey.com
Base Domain defkey.com
Scan Status Ok
Last Scan2024-09-29T02:50:10+00:00
Next Scan 2024-10-06T02:50:10+00:00

Last Scan

Scanned2024-09-29T02:50:10+00:00
URL https://defkey.com/robots.txt
Domain IPs 104.21.9.179, 172.67.160.243, 2606:4700:3030::ac43:a0f3, 2606:4700:3034::6815:9b3
Response IP 172.67.160.243
Found Yes
Hash 5bb1a05602497361887f2981a3f29b1cc4547023f93e81e20050d9dcd2f886c9
SimHash e916951043d8

Groups

*

Rule Path
Allow /tr
Allow /de/tastaturtester
Disallow /en/
Disallow */Account/Login?
Disallow */Account/Register?
Disallow /xhr/
Disallow /search?s=*
Disallow /tr/arama?s=*
Disallow /fr/recherche?s=*
Disallow /embed/*
Disallow /tr/shortcutarticle/incrementviewcount
Disallow /shortcutarticle/incrementviewcount
Disallow /emaillist/addtolistjson
Disallow /partial/
Disallow /partial/getprogramcomment
Disallow /tr/emailList/addToListJson
Disallow /emailList/addToListJson
Disallow /tr/newsArticle/incrementNewsArticleViewCount
Disallow /newsArticle/incrementNewsArticleViewCount
Disallow /newsArticle/*
Disallow /*?CurrentProgram=*
Disallow /*?sortBy=*
Disallow /ShortcutActions/
Disallow */shortcutactions/popularactionspartial*
Disallow /tools/randomprogram/*
Disallow /visualizeshortcut/
Disallow /*?plainText=True*
Disallow /*?pdf=True*
Back to top

Comments

  • Uncomment this after Shortcut Actions with category Ids are removed from search engine indexes.
  • Disallow: /*?CategoryId=*
  • Note: Putting a file into robots.txt will not prevent Google from indexing the page. It will only prevent Googlebot from re-crawling the page.
  • If you want Google to remove the page from the index you should allow Googlebot to crawl the page and see the "noindex" in the meta robots tag.
  • Check IIS logs every now and then to block unwanted URLs. Bots can discover weird URLs (such as directly calling controller) to keep site very busy.
  • Test after updating here: https://www.google.com/webmasters/tools/robots-testing-tool?siteUrl=https://defkey.com/
Back to top