Log In Sign Up

wisdek.com
robots.txt

Robots Exclusion Standard data for wisdek.com

Resource Scan

Scan Details

Site Domain wisdek.com
Base Domain wisdek.com
Scan Status Ok
Last Scan2026-01-28T11:38:22+00:00
Next Scan 2026-02-27T11:38:22+00:00

Last Scan

Scanned2026-01-28T11:38:22+00:00
URL https://wisdek.com/robots.txt
Domain IPs 66.71.220.1, 66.71.220.2
Response IP 66.71.220.2
Found Yes
Hash 0e23dc96c4be15450c1c41cbe1c4bae62c976c29fefca30dd376776299ec3e51
SimHash 747b4b31e5c2

Groups

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

ccbot

Rule Path
Allow /

semrushbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

ahrefsbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

mj12bot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

dotbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

dataforseobot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

*

Rule Path
Allow /
Allow /_next/static/
Disallow /api/
Disallow /admin/
Disallow /private/
Disallow *.php$
Disallow *.cgi$
Disallow *.asp$
Disallow *.aspx$
Disallow /cgi-bin/
Back to top

Other Records

Field Value
sitemap https://wisdek.com/sitemap.xml
Back to top

Comments

  • Wisdek Digital Marketing - Robots.txt
  • Optimized for Google Search Console compliance
  • Last updated: 2026-01-19
  • Cache-busting update: 2026-01-19T21:09:58.722Z
  • Sitemap location (Primary)
  • Major search engine crawlers - full access (no Crawl-delay to avoid GSC warnings)
  • AI-powered search engines - EXPLICITLY ALLOWED for blog content and social sharing
  • SEO audit and analysis tools (rate limited)
  • Additional AI crawlers - Comment these out if you want to allow them
  • User-agent: anthropic-ai
  • Disallow: /
  • User-agent: Claude-Web
  • Disallow: /
  • User-agent: cohere-ai
  • Disallow: /
  • User-agent: Google-Extended
  • Disallow: /
  • User-agent: PerplexityBot
  • Disallow: /
  • User-agent: Omgilibot
  • Disallow: /
  • Block aggressive crawlers that don't respect crawl budgets
  • Default rules for all other bots
  • IMPORTANT: Do NOT block /_next/static/ - Google needs access to CSS, JS, and static resources
  • to properly render and evaluate pages for indexing and ranking
Back to top