Log In Sign Up

iwmi.cgiar.org
robots.txt

Robots Exclusion Standard data for iwmi.cgiar.org

Resource Scan

Scan Details

Site Domain iwmi.cgiar.org
Base Domain cgiar.org
Scan Status Ok
Last Scan2024-06-21T04:57:09+00:00
Next Scan 2024-07-21T04:57:09+00:00

Last Scan

Scanned2024-06-21T04:57:09+00:00
URL https://iwmi.cgiar.org/robots.txt
Redirect https://www.iwmi.cgiar.org/robots.txt
Redirect Domain www.iwmi.cgiar.org
Redirect Base cgiar.org
Domain IPs 141.193.213.10, 141.193.213.11
Redirect IPs 141.193.213.10, 141.193.213.11
Response IP 141.193.213.10
Found Yes
Hash 80ca368e69c5dce60cdd22c37005c0d122ee6a5a1108639ee9ecd4b86a6baea8
SimHash 295adb0196f5

Groups

twitterbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 600

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

googlebot-image

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

googlebot-video

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

googlebot-news

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

mediapartners-google

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

bing

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

yahoo-blogs

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

yahoo-mmcrawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

duckduckbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

exabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

facebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

facebookexternalhit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 600

ia_archiver

Rule Path
Allow /Publications/*/
Allow /wp-content/images/*/
Allow /wp-content/uploads/*/
Allow /about/staff-list/*
Allow /wp-includes/*/
Allow /events/*
Disallow /wp-admin/
Disallow /wp-content/mu-plugins/
Disallow /wp-content/upgrade/
Disallow /search-results/
Disallow /search_gcse/
Disallow /wp-login.php
Disallow /wp-cron.php
Disallow /xmlrpc.php

Other Records

Field Value
crawl-delay 600
Back to top

Other Records

Field Value
sitemap https://www.iwmi.cgiar.org/page-sitemap.xml
sitemap https://www.iwmi.cgiar.org/page-sitemap2.xml
sitemap https://www.iwmi.cgiar.org/page-sitemap3.xml
sitemap https://www.iwmi.cgiar.org/post-sitemap.xml
sitemap https://www.iwmi.cgiar.org/post-sitemap2.xml
sitemap https://www.iwmi.cgiar.org/post-sitemap3.xml
sitemap https://www.iwmi.cgiar.org/event-sitemap.xml
sitemap https://www.iwmi.cgiar.org/category-sitemap.xml
sitemap https://www.iwmi.cgiar.org/post_tag-sitemap.xml
sitemap https://www.iwmi.cgiar.org/post_tag2-sitemap.xml
Back to top

Comments

  • allow all
  • alllow the twitter bot
  • set the crawl delay
  • but allow only important bots
  • Index these directories
  • Do not index directories
  • Files
Back to top

Warnings

  • 1 invalid line.
Back to top