Log In Sign Up

unric.org
robots.txt

Robots Exclusion Standard data for unric.org

Resource Scan

Scan Details

Site Domain unric.org
Base Domain unric.org
Scan Status Ok
Last Scan2025-09-03T17:15:55+00:00
Next Scan 2025-10-03T17:15:55+00:00

Last Scan

Scanned2025-09-03T17:15:55+00:00
URL https://unric.org/robots.txt
Domain IPs 104.26.4.221, 104.26.5.221, 172.67.68.66, 2606:4700:20::681a:4dd, 2606:4700:20::681a:5dd, 2606:4700:20::ac43:4442
Response IP 172.67.68.66
Found Yes
Hash 5a92042873d9e333e42cd40b73a20337f7ebc6f32af6d050086e28332fc1b4cc
SimHash 684e0e50a4f1

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /cgi-bin/
Disallow /wp-login.php
Disallow /xmlrpc.php
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Disallow /?s=
Disallow /search/
Disallow /page/*/?s=
Disallow */trackback/
Disallow */feed/
Disallow */comments/
Disallow /*?replytocom
Disallow /*.php$
Disallow /*.cgi$
Disallow /*.xhtml$
Disallow /*.swf$
Disallow /*.inc$
Disallow /*.wmv$
Allow /wp-admin/admin-ajax.php
Allow /*.css$
Allow /*.js$
Allow /*.woff$
Allow /*.woff2$
Allow /*.ttf$
Allow /*.eot$
Allow /*.svg$

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

anthropicai

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://unric.org/sitemap_index.xml
sitemap https://unric.org/en/sitemap_index.xml
sitemap https://unric.org/fr/sitemap_index.xml
sitemap https://unric.org/it/sitemap_index.xml
sitemap https://unric.org/pt/sitemap_index.xml
sitemap https://unric.org/nl/sitemap_index.xml
sitemap https://unric.org/es/sitemap_index.xml
sitemap https://unric.org/de/sitemap_index.xml
sitemap https://unric.org/sv/sitemap_index.xml
sitemap https://unric.org/da/sitemap_index.xml
sitemap https://unric.org/no/sitemap_index.xml
sitemap https://unric.org/fi/sitemap_index.xml
sitemap https://unric.org/is/sitemap_index.xml
sitemap https://unric.org/el/sitemap_index.xml
Back to top

Comments

  • UNRIC Robots.txt - SEO & Performance Optimized - IA Block Agent (PC release - 17 April 2025)
  • Block backend and system directories
  • Block sensitive plugin directory
  • It's crucial to keep wp-content/plugins/ blocked to ensure robots
  • don't access any sensitive files that may be present, but also to reduce the load on the site.
  • If you are using plugins to generate content that search engines need to access,
  • then you need to find a different configuration for your site.
  • Block cache directory
  • Block theme files (review carefully)
  • Disallow: /wp-content/themes/ # Only uncomment if you have very specific reasons
  • Block search pages and duplicate content
  • Block WordPress duplicates and unused endpoints
  • Block file types not meant for indexing
  • Allow AJAX
  • Let Cloudflare, RocketCDN & Fonts load freely
  • BLOCK AI CRAWLERS (April 2025)
  • Sitemaps for Yoast SEO Multisite
Back to top