Log In Sign Up

mlaib.com
robots.txt

Robots Exclusion Standard data for mlaib.com

Resource Scan

Scan Details

Site Domain mlaib.com
Base Domain mlaib.com
Scan Status Ok
Last Scan2025-10-14T00:15:27+00:00
Next Scan 2025-10-21T00:15:27+00:00

Last Scan

Scanned2025-10-14T00:15:27+00:00
URL https://mlaib.com/robots.txt
Redirect https://www.belgoal.com/robots.txt
Redirect Domain www.belgoal.com
Redirect Base belgoal.com
Domain IPs 51.222.10.241
Redirect IPs 51.222.10.241
Response IP 51.222.10.241
Found Yes
Hash e762ea67929d5c5726af8c650b20c00527aba45378ff485e807f05c338b5bc3c
SimHash 4664d33865b3

Groups

*

Rule Path
Allow /wp-admin/admin-ajax.php
Allow /wp-includes/js/
Allow /wp-includes/images/
Allow /wp-content/uploads/

mediapartners-google*

Rule Path
Disallow /wp-admin/
Allow /

googlebot-image

Rule Path
Disallow /wp-admin/
Allow /wp-content/uploads/

twitterbot

Rule Path
Disallow /wp-admin/
Allow /images/
Allow /archives/

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Disallow /?s=
Disallow /*?s=
Disallow /?attachment_id=
Disallow /feed/
Disallow /comments/
Disallow /trackback/
Disallow /cgi-bin/
Disallow /author/
Disallow /wp-json/
Back to top

Other Records

Field Value
sitemap https://www.belgoal.com/sitemap_index.xml
Back to top

Comments

  • Allow AJAX requests
  • Allow specific JS and image resources
  • Allow media uploads to be crawled
  • Google Ads bot
  • Googlebot-Image
  • Twitterbot
  • General user-agent rules for all bots
  • Prevent access to core WordPress admin and includes directories
  • Block plugin and theme files
  • Block low-quality and duplicate content
  • Sitemap location
Back to top