turnkeymsp.net
robots.txt

Robots Exclusion Standard data for turnkeymsp.net

Resource Scan

Scan Details

Site Domain turnkeymsp.net
Base Domain turnkeymsp.net
Scan Status Ok
Last Scan2025-11-22T21:18:17+00:00
Next Scan 2025-11-29T21:18:17+00:00

Last Scan

Scanned2025-11-22T21:18:17+00:00
URL https://turnkeymsp.net/robots.txt
Redirect https://www.turnkeymsp.net/robots.txt
Redirect Domain www.turnkeymsp.net
Redirect Base turnkeymsp.net
Domain IPs 104.21.68.228, 172.67.199.54, 2606:4700:3033::6815:44e4, 2606:4700:3035::ac43:c736
Redirect IPs 104.21.68.228, 172.67.199.54, 2606:4700:3033::6815:44e4, 2606:4700:3035::ac43:c736
Response IP 104.21.68.228
Found Yes
Hash ad86e86883d329515e79e7f8a5a003e683250dea60513bb1a7f7c8fc37c94ad0
SimHash 4189c84366b3

Groups

termlybot

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /private-uploads/
Disallow /wp-content/uploads/private/
Disallow /wp-content/uploads/wc-logs/
Disallow /wp-content/uploads/woocommerce_transient_files/
Disallow /wp-content/uploads/woocommerce_uploads/
Disallow /*blackhole
Disallow /?blackhole
Disallow /checkout/
Disallow /cart/
Disallow /my-account/
Disallow /wp-login.php
Disallow /network-admin/
Disallow /wp-includes/

Other Records

Field Value
crawl-delay 5

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

bingbot

Rule Path
Allow /

claritybot

Rule Path
Allow /

yandexbot

Rule Path
Allow /

aspiegelbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

googlebot-image

Rule Path
Allow /wp-content/uploads/
Disallow /wp-content/uploads/private/

Other Records

Field Value
sitemap https://www.turnkeymsp.net/sitemap_index.xml
sitemap https://www.turnkeymsp.net/post-sitemap.xml
sitemap https://www.turnkeymsp.net/page-sitemap.xml
sitemap https://www.turnkeymsp.net/category-sitemap.xml
sitemap https://www.turnkeymsp.net/local-sitemap.xml

Comments

  • Termly scanner
  • General settings for all bots
  • Allow specific bots
  • Block Huawei Aspiegel/PetalBot
  • Allow Googlebot-Image for image indexing, but not private uploads
  • Sitemap references