Log In Sign Up

jobster.com
robots.txt

Robots Exclusion Standard data for jobster.com

Resource Scan

Scan Details

Site Domain jobster.com
Base Domain jobster.com
Scan Status Ok
Last Scan2026-03-01T13:16:59+00:00
Next Scan 2026-03-31T13:16:59+00:00

Last Scan

Scanned2026-03-01T13:16:59+00:00
URL https://jobster.com/robots.txt
Domain IPs 185.117.226.79, 2a10:7180:100::79
Response IP 185.117.226.79
Found Yes
Hash d33542599204ecf9c25eeee7f1316b3e7e513e6ef0266a7b3a5869983a79da59
SimHash 420cd9502d90

Groups

*

Rule Path
Allow /
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /wp-*.php
Disallow /xmlrpc.php
Disallow /wp-json/
Disallow /*?*orderby=
Disallow /*?*s=
Disallow /*?*p=*&*
Disallow /search/
Disallow /*?s=
Disallow /author/
Disallow /authors/
Disallow /administrator/
Disallow /?author=*
Disallow /trackback/
Disallow /feed/
Disallow /comments/feed/
Disallow /*.sql
Disallow /*.zip
Disallow /*.gz
Disallow /*.rar
Disallow /*.7z
Disallow /backup/
Disallow /backups/
Disallow /old/
Disallow /temp/
Disallow /tmp/
Allow /feed/$
Allow /*/feed/$
Allow /wp-admin/admin-ajax.php
Disallow /*?utm_*
Disallow /*?fbclid=*
Back to top

Other Records

Field Value
sitemap https://yoursite.com/sitemap.xml
sitemap https://yoursite.com/sitemap_index.xml
Back to top

Comments

  • WordPress Robots.txt - Best Practices Configuration
  • Place this file in your website root directory (same level as wp-config.php)
  • Allow all bots by default
  • Disallow private/sensitive WordPress directories and files
  • Disallow common WordPress files that shouldn't be indexed
  • Disallow search results and private pages
  • Disallow admin pages and user pages
  • Disallow spam-prone pages
  • Disallow temporary and backup files
  • Disallow common backup directories
  • Allow specific RSS feeds (optional - remove if you want to disallow all feeds)
  • Allow specific admin pages needed by search engines
  • Disallow WooCommerce pages (if using WooCommerce)
  • Disallow: /shop/*?*
  • Disallow: /*?product_cat=*
  • Disallow: /*?product-page=*
  • Disallow specific query strings (duplicate content)
  • Crawl delay (in seconds) - optional
  • Crawl-delay: 1
  • Sitemap location
  • Optional: More specific rules for particular bots
  • User-agent: AhrefsBot
  • Disallow: /
  • User-agent: SemrushBot
  • Disallow: /
  • Optional: Block bad bots
  • User-agent: MJ12bot
  • Disallow: /
Back to top