Log In Sign Up

10gen.com
robots.txt

Robots Exclusion Standard data for 10gen.com

Resource Scan

Scan Details

Site Domain 10gen.com
Base Domain 10gen.com
Scan Status Ok
Last Scan2024-05-29T10:50:15+00:00
Next Scan 2024-06-12T10:50:15+00:00

Last Scan

Scanned2024-05-29T10:50:15+00:00
URL https://10gen.com/robots.txt
Redirect https://www.mongodb.com/robots.txt
Redirect Domain www.mongodb.com
Redirect Base mongodb.com
Domain IPs 54.175.147.155
Redirect IPs 2600:9000:2003:2400:7:7859:3840:93a1, 2600:9000:2003:5400:7:7859:3840:93a1, 2600:9000:2003:7a00:7:7859:3840:93a1, 2600:9000:2003:a600:7:7859:3840:93a1, 2600:9000:2003:b800:7:7859:3840:93a1, 2600:9000:2003:e200:7:7859:3840:93a1, 2600:9000:2003:e400:7:7859:3840:93a1, 2600:9000:2003:f600:7:7859:3840:93a1, 52.84.229.113, 52.84.229.28, 52.84.229.65, 52.84.229.87
Response IP 52.84.229.28
Found Yes
Hash 984d8b9824cb326227bc1ae42d3958b53e97ecf3f2b493aa765c3de19b14a340
SimHash e285dd1f4dd9

Groups

mauibot

Rule Path
Disallow /community/forums/

semrushbot

Rule Path
Disallow /community/forums/

ahrefsbot

Rule Path
Disallow /community/forums/

blexbot

Rule Path
Disallow /community/forums/

seo spider

Rule Path
Disallow /community/forums/

bingbot

Rule Path
Disallow /community/forums/auth/
Disallow /community/forums/assets/browser-update*.js
Disallow /community/forums/email/
Disallow /community/forums/session
Disallow /community/forums/session/
Disallow /community/forums/admin
Disallow /community/forums/admin/
Disallow /community/forums/user-api-key
Disallow /community/forums/user-api-key/
Disallow /community/forums/*?api_key*
Disallow /community/forums/*?*api_key*

Other Records

Field Value
crawl-delay 60

*

Rule Path
Disallow /misc/
Disallow /profiles/
Disallow /u/
Disallow /community/forums/auth/
Disallow /community/forums/email/
Disallow /community/forums/session
Disallow /community/forums/session/
Disallow /community/forums/admin
Disallow /community/forums/admin/
Disallow /community/forums/user-api-key
Disallow /community/forums/user-api-key/
Disallow /community/forums/g/
Disallow /community/forums/u/
Disallow /community/forums/user_avatar/
Disallow /admin/*
Disallow /data/*
Disallow /comment/reply/
Disallow /filter/tips/
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /lp/p/*
Disallow /lp/s/*
Disallow /lp/d/*
Disallow /thank-you/*
Disallow /link/*
Disallow /community/forums/assets/browser-update*.js
Disallow /docs/*.epub
Disallow /docs/*.pdf
Disallow /*?quicktabs_*
Disallow /*%26quicktabs_*
Disallow /*?qt-view_*
Disallow /*%26qt-view_*
Disallow /field-collection/*
Disallow /tags/*
Disallow /taxonomy/term/*
Disallow /?q=admin%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=filter%2Ftips%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F
Disallow /?q=user%2Flogout%2F
Disallow /commercialsupport/
Disallow /partners/referral-federal
Disallow /customer-evaluation-downloads-development-versions
Disallow /community/forums/*?api_key*
Disallow /community/forums/*?*api_key*
Disallow /docs/meta/
Disallow /docs/search/
Disallow /docs/openapi/
Disallow /docs-qa*
Disallow /devcenter-uat*
Disallow /de-de/developer*
Disallow /es/developer*
Disallow /fr-fr/developer*
Disallow /it-it/developer*
Disallow /ja-jp/developer*
Disallow /ko-kr/developer*
Disallow /pt-br/developer*
Disallow /zh-cn/developer*

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.mongodb.com/sitemap.xml
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • User-agent: *
  • Disallow: /
  • Directories
  • Files
  • No access for quicktabs in the URL
  • No crawling field collections or taxonomy
  • Paths (no clean URLs)
  • Specific pages
  • Prevent exposing future docs
  • Prevent exposing devhub UAT
Back to top