Log In Sign Up

ddeclille.org
robots.txt

Robots Exclusion Standard data for ddeclille.org

Resource Scan

Scan Details

Site Domain ddeclille.org
Base Domain ddeclille.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-09-30T19:13:01+00:00
Next Scan 2025-10-30T19:13:01+00:00

Last Successful Scan

Scanned2025-08-09T18:10:43+00:00
URL https://www.ddeclille.org/robots.txt
Domain IPs 217.182.133.158
Response IP 217.182.133.158
Found Yes
Hash 440be1d3cad0897f7673acc18c7878b3a3eafa46ab8e4df9a7e8c14e70b53887
SimHash 6a1e94504531

Groups

googlebot

Rule Path
Disallow /*?*
Disallow /*folder_factories$
Disallow /*send_as_pdf*
Disallow /*download_as_pdf*
Disallow /parametrages/
Disallow /newsletter/
Disallow /abonnez-vous/
Disallow /don-en-ligne/
Disallow /portal_checkouttool/
Disallow /Members/

mediapartners-google

Rule Path
Disallow
Allow /
Disallow /*folder_factories$
Disallow /*send_as_pdf*
Disallow /*download_as_pdf*
Disallow /parametrages/
Disallow /newsletter/
Disallow /abonnez-vous/
Disallow /don-en-ligne/
Disallow /portal_checkouttool/
Disallow /Members/

yahoo! slurp

Rule Path
Disallow /*?*
Disallow /*folder_factories$
Disallow /*send_as_pdf*
Disallow /*download_as_pdf*
Disallow /parametrages/
Disallow /newsletter/
Disallow /abonnez-vous/
Disallow /don-en-ligne/
Disallow /portal_checkouttool/
Disallow /Members/

Other Records

Field Value
crawl-delay 10

bingbot

Rule Path
Disallow /*?*
Disallow /*folder_factories$
Disallow /*send_as_pdf*
Disallow /*download_as_pdf*
Disallow /parametrages/
Disallow /newsletter/
Disallow /abonnez-vous/
Disallow /don-en-ligne/
Disallow /portal_checkouttool/
Disallow /Members/

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Disallow /*?*
Disallow /*folder_factories$
Disallow /*send_as_pdf*
Disallow /*download_as_pdf*
Disallow /parametrages/
Disallow /newsletter/
Disallow /abonnez-vous/
Disallow /don-en-ligne/
Disallow /portal_checkouttool/
Disallow /Members/

Other Records

Field Value
crawl-delay 10

msnbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

discobot

Rule Path
Disallow /

blekkobot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

uptimerobot/2.0

Rule Path
Disallow /

ezooms robot

Rule Path
Disallow /

perl lwp

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

wiseguys robot

Rule Path
Disallow /

turnitin robot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

babya discoverer

Rule Path
Disallow /

*

Rule Path
Disallow /parametrages/
Disallow /newsletter/
Disallow /abonnez-vous/
Disallow /don-en-ligne/
Disallow /portal_checkouttool/
Disallow /Members/

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.ddeclille.org/sitemap.xml.gz
Back to top

Comments

  • Define access-restrictions for robots/spiders
  • http://www.robotstxt.org/wc/norobots.html
  • see http://opensourcehacker.com/2009/08/07/seo-tips-query-strings-multiple-languages-forms-and-other-content-management-system-issues/
  • Googlebot allows regex in its syntax
  • Block all URLs including query strings (? pattern) - contentish objects expose query string only for actions or status reports which
  • might confuse search results.
  • This will also block ?set_language
  • Allow Adsense bot on entire site
  • Block MJ12bot as it is just noise
  • Block Ahrefs
  • Block Sogou
  • Block SEOkicks
  • SEOkicks
  • Dicoveryengine.com
  • Blekkobot
  • Block BlexBot
  • Block SISTRIX
  • Block Uptime robot
  • Block Ezooms Robot
  • Block Perl LWP
  • Block netEstate NE Crawler
  • Block WiseGuys Robot
  • Block Turnitin Robot
  • Exabot
  • Yandex
  • Babya Discoverer
  • Directories
  • Request-rate: defines pages/seconds to be crawled ratio. 1/20 would be 1 page in every 20 second.
  • Crawl-delay: defines howmany seconds to wait after each succesful crawling.
  • Visit-time: you can define between which hours you want your pages to be crawled. Example usage is: 0100-0330 which means that pages will be indexed between 01:00 AM - 03:30 AM GMT.
Back to top

Warnings

  • 2 invalid lines.
  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top