awslabs/scale-out-computing-on-aws

Optional OpenSearch

Closed this issue · 5 comments

urgent commented

Is your feature request related to a problem? Please describe.
Manual deletion of OpenSearch required post install

Describe the feature you'd like
A flag in default.yml to disable OpenSearch

jsynack commented

Hello @urgent,

Can you describe your use-case of not wanting/needing metrics/analytics/opensearch?

We have been looking at ways to decrease the overall cost of a SOCA deployment - and the OpenSearch cluster is a fair amount of this cost.

We are roadmapping a potential change to OpenSearch serverless to decrease the overall infrastructure cost of SOCA.

Would this meet your objective or would a serverless deployment also not be desired for your use-cases?

Thanks!

tammy-ruby-cherry commented

Hi all,

Two suggestions below:

  • Make OpenSearch optional
  • Change OpenSearch parameters to reduce costs

Make OpenSearch optional
Allow "" as an option for the engine by tweaking cdk_construct.json to handle this:

Replace line 194: self.analytics() #Create Analytics domain
with a conditional:

        if (
            install_props.Config.analytics.engine
        ):
            self.analytics()  # Create Analytics domain

Note: Use "" rather than None as the value of the engine parameter to prevent an annoying issue with the WebUI.

Quick solution to reduce OpenSearch costs:
Change the the analytics parameters in default_config.yml. E.g., the configuration below lowers OpenSearch costs to $60-ish/month. There are fault-tolerance and performance tradeoffs*, but, hey, life is all about choices. These specs are fine for some use cases.

  analytics:     # defaults are set for the smallest possible engine as a placeholder
    engine: "opensearch" # elasticsearch, opensearch. 
    data_node_instance_type: "t3.small.search" # instance type for analytics data nodes
    data_nodes: 1  # number of data nodes for analytics
    ebs_volume_size: 10  # EBS volume size attached to data nodes in GB
    deletion_policy: "NO_RETAIN"  # RETAIN will preserve the ES even if you delete the stack. Any other value will delete OS if you delete the CFN stack

*See the downsides:

  1. https://repost.aws/knowledge-center/opensearch-fault-tolerance)
  2. https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-instance-types.html

Best,
Tammy

  • 👍1
mcrozes commented

Hey Tammy -

We are looking into this, especially with the new support for AWS SSM parameters as mentioned on #120

I think the recommendation here would be to add extend the default_config.yml file and add a enabled flag. This flag will be used to determine if (1) we need to deploy OpenSearch, (2) will be used on all Analytics script and - more importantly - (3) will make sure we can hot-patch a live cluster post-deployment in case we finally want to re-enable OpenSearch at a later time.

Happy to chat / share more details, feel free to email us if you are interesting with the beta build.

mcrozes commented

Hey Tammy -

As discussed previously, OpenSearch will now be optional with the next version of SOCA. We also have rewritten the analytics scripts to handle case where OpenSearch is not there.

mcrozes commented

Closing this ticket as this capability has been added and pushed on Github