Log In Sign Up

codeby.info
robots.txt

Robots Exclusion Standard data for codeby.info

Resource Scan

Scan Details

Site Domain codeby.info
Base Domain codeby.info
Scan Status Ok
Last Scan2025-08-30T11:09:46+00:00
Next Scan 2025-09-06T11:09:46+00:00

Last Scan

Scanned2025-08-30T11:09:46+00:00
URL https://codeby.info/robots.txt
Domain IPs 185.137.235.142
Response IP 185.137.235.142
Found Yes
Hash fed61574a295a3c98abd550598804b89a67bf2d6bedc9aea692c1d9c3223cbf9
SimHash 1420014625f7

Groups

yandex

Rule Path Comment
Disallow /admin.php -
Disallow /install/ -
Disallow /src/ -
Disallow /internal_data/ -
Disallow /job.php -
Disallow /blogs/wp-admin/ -
Disallow /blogs/wp-includes/ -
Disallow /blogs/wp-content/plugins/ -
Disallow /blogs/wp-content/cache/ -
Disallow /blogs/wp-content/themes/ -
Disallow /blogs/xmlrpc.php -
Disallow /blogs/wp-json/ -
Disallow /account/ -
Disallow /conversations/ -
Disallow /login/ -
Disallow /logout/ -
Disallow /lost-password/ -
Disallow /register/ -
Disallow /two-step/ -
Disallow /search/ -
Disallow /goto/ -
Disallow /misc/ -
Disallow /online/ -
Disallow /report/ -
Disallow /whats-new/ -
Disallow /find-new/ -
Disallow /recent-activity/ -
Disallow /blogs/feed/ -
Disallow /blogs/trackback/ -
Disallow /blogs/*/feed/ -
Disallow /blogs/*/trackback/ -
Disallow /blogs/wp-login.php -
Disallow /*.json$ API responses in JSON format
Disallow /data/avatars/ User avatars (no SEO value)
Disallow /data/attachments/thumbnails/ Block only thumbnails
Disallow /members/* Block all user profiles
Disallow /threads/*/reply -
Disallow /threads/*/add-reply -
Disallow /threads/*/watch -
Disallow /threads/*/unwatch -
Disallow /forums/*/watch -
Disallow /forums/-/mark-read -
Disallow /blogs/?s= WordPress blog search
Disallow /blogs/?p= WordPress post/page param
Disallow /blogs/?preview= WP previews
Disallow /blogs/?replytocom= WP comment replies
Allow /data/assets/ -
Allow /styles/ -
Allow /blogs/wp-content/uploads/ -
Allow /members/$ Allow only main members list page

googlebot
*

Rule Path Comment
Disallow /admin.php -
Disallow /install/ -
Disallow /src/ -
Disallow /internal_data/ -
Disallow /job.php -
Disallow /blogs/wp-admin/ -
Disallow /blogs/wp-includes/ -
Disallow /blogs/wp-content/plugins/ -
Disallow /blogs/wp-content/cache/ -
Disallow /blogs/wp-content/themes/ -
Disallow /blogs/xmlrpc.php -
Disallow /blogs/wp-json/ -
Disallow /account/ -
Disallow /conversations/ -
Disallow /login/ -
Disallow /logout/ -
Disallow /lost-password/ -
Disallow /register/ -
Disallow /two-step/ -
Disallow /search/ -
Disallow /goto/ -
Disallow /misc/ -
Disallow /online/ -
Disallow /report/ -
Disallow /whats-new/ -
Disallow /find-new/ -
Disallow /recent-activity/ -
Disallow /blogs/feed/ -
Disallow /blogs/trackback/ -
Disallow /blogs/*/feed/ -
Disallow /blogs/*/trackback/ -
Disallow /blogs/wp-login.php -
Disallow /*.json$ -
Disallow /data/avatars/ -
Disallow /data/attachments/thumbnails/ -
Disallow /members/* Block all user profiles
Disallow /threads/*/reply -
Disallow /threads/*/add-reply -
Disallow /threads/*/watch -
Disallow /threads/*/unwatch -
Disallow /forums/*/watch -
Disallow /forums/-/mark-read -
Disallow /blogs/?s= -
Disallow /blogs/?p= -
Disallow /blogs/?preview= -
Disallow /blogs/?replytocom= -
Allow /data/assets/ -
Allow /styles/ -
Allow /blogs/wp-content/uploads/ -
Allow /members/$ Allow only main members list page
Back to top

Other Records

Field Value
sitemap https://codeby.net/sitemap.xml
sitemap https://codeby.net/blogs/sitemap.xml
Back to top

Comments

  • --- [DISALLOW] System paths for XenForo and WordPress ---
  • --- [DISALLOW] Service sections and pages with no SEO value ---
  • --- [DISALLOW] Technical duplicates, filters, and action pages ---
  • --- [ALLOW] Important resources for rendering ---
  • --- [Yandex Clean-param] to prevent duplicate indexing ---
  • Marketing and tracking parameters
  • Analytics and session params
  • WordPress and XenForo Resources
  • AMP pages
  • =====================================================================
  • Section #2: Rules for Googlebot and all other crawlers
  • =====================================================================
  • --- [DISALLOW] System paths for XenForo and WordPress ---
  • --- [DISALLOW] Service sections and pages with no SEO value ---
  • --- [DISALLOW] Technical duplicates, filters, and action pages ---
  • --- [ALLOW] Important resources for rendering ---
  • =====================================================================
  • Section #3: Sitemaps
  • =====================================================================
Back to top

Warnings

  • `clean-param` is not a known field.
Back to top