Log In Sign Up

mongodb.org
robots.txt

Robots Exclusion Standard data for mongodb.org

Resource Scan

Scan Details

Site Domain mongodb.org
Base Domain mongodb.org
Scan Status Ok
Last Scan2024-05-08T23:55:26+00:00
Next Scan 2024-05-22T23:55:26+00:00

Last Scan

Scanned2024-05-08T23:55:26+00:00
URL https://mongodb.org/robots.txt
Redirect https://www.mongodb.com/robots.txt
Redirect Domain www.mongodb.com
Redirect Base mongodb.com
Domain IPs 54.175.147.155
Redirect IPs 2600:9000:2003:1800:7:7859:3840:93a1, 2600:9000:2003:8e00:7:7859:3840:93a1, 2600:9000:2003:9000:7:7859:3840:93a1, 2600:9000:2003:9a00:7:7859:3840:93a1, 2600:9000:2003:cc00:7:7859:3840:93a1, 2600:9000:2003:d200:7:7859:3840:93a1, 2600:9000:2003:e800:7:7859:3840:93a1, 2600:9000:2003:ee00:7:7859:3840:93a1, 52.84.229.113, 52.84.229.28, 52.84.229.65, 52.84.229.87
Response IP 52.84.229.65
Found Yes
Hash 984d8b9824cb326227bc1ae42d3958b53e97ecf3f2b493aa765c3de19b14a340
SimHash e285dd1f4dd9

Groups

mauibot

Rule Path
Disallow /community/forums/

semrushbot

Rule Path
Disallow /community/forums/

ahrefsbot

Rule Path
Disallow /community/forums/

blexbot

Rule Path
Disallow /community/forums/

seo spider

Rule Path
Disallow /community/forums/

bingbot

Rule Path
Disallow /community/forums/auth/
Disallow /community/forums/assets/browser-update*.js
Disallow /community/forums/email/
Disallow /community/forums/session
Disallow /community/forums/session/
Disallow /community/forums/admin
Disallow /community/forums/admin/
Disallow /community/forums/user-api-key
Disallow /community/forums/user-api-key/
Disallow /community/forums/*?api_key*
Disallow /community/forums/*?*api_key*

Other Records

Field Value
crawl-delay 60

*

Rule Path
Disallow /misc/
Disallow /profiles/
Disallow /u/
Disallow /community/forums/auth/
Disallow /community/forums/email/
Disallow /community/forums/session
Disallow /community/forums/session/
Disallow /community/forums/admin
Disallow /community/forums/admin/
Disallow /community/forums/user-api-key
Disallow /community/forums/user-api-key/
Disallow /community/forums/g/
Disallow /community/forums/u/
Disallow /community/forums/user_avatar/
Disallow /admin/*
Disallow /data/*
Disallow /comment/reply/
Disallow /filter/tips/
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /lp/p/*
Disallow /lp/s/*
Disallow /lp/d/*
Disallow /thank-you/*
Disallow /link/*
Disallow /community/forums/assets/browser-update*.js
Disallow /docs/*.epub
Disallow /docs/*.pdf
Disallow /*?quicktabs_*
Disallow /*%26quicktabs_*
Disallow /*?qt-view_*
Disallow /*%26qt-view_*
Disallow /field-collection/*
Disallow /tags/*
Disallow /taxonomy/term/*
Disallow /?q=admin%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=filter%2Ftips%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F
Disallow /?q=user%2Flogout%2F
Disallow /commercialsupport/
Disallow /partners/referral-federal
Disallow /customer-evaluation-downloads-development-versions
Disallow /community/forums/*?api_key*
Disallow /community/forums/*?*api_key*
Disallow /docs/meta/
Disallow /docs/search/
Disallow /docs/openapi/
Disallow /docs-qa*
Disallow /devcenter-uat*
Disallow /de-de/developer*
Disallow /es/developer*
Disallow /fr-fr/developer*
Disallow /it-it/developer*
Disallow /ja-jp/developer*
Disallow /ko-kr/developer*
Disallow /pt-br/developer*
Disallow /zh-cn/developer*

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.mongodb.com/sitemap.xml
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • User-agent: *
  • Disallow: /
  • Directories
  • Files
  • No access for quicktabs in the URL
  • No crawling field collections or taxonomy
  • Paths (no clean URLs)
  • Specific pages
  • Prevent exposing future docs
  • Prevent exposing devhub UAT
Back to top