dreamer-89/opensearch-ci

Enables continuous integration across OpenSearch, OpenSearch Dashboards, and plugins.

• OpenSearch Continous Integration
• Getting Started
• Deployment
  • Dev Deployment
  • Executing Optional Tasks
    • SSL Configuration
    • setup OpenId Connect (OIDC) via Federate
  • Troubleshooting
    • Main Node
  • Useful commands
  • Architecture Overview
• Contributing
• Getting Help
• Code of Conduct
• Security
• License
• Copyright

OpenSearch Continuous Integration

OpenSearch Continuous Integration is an open source CI system for OpenSearch and its plugins.

Getting Started

• Requires NPM to be installed
• Install project dependencies using npm install from this project directory
• Configure aws credentials
• Deploy stacks with npm run cdk deploy

Deployment

Dev Deployment

1. Setup your local machine to credentials to deploy to the AWS Account

2. Deploy the bootstrap stack by running following command that sets up required resources to create the stacks. More info

   npm run cdk bootstrap -- -c useSsl=false -c runWithOidc=false

3. Deploy the ci-config-stack using the with one of the following (takes ~1 minute to deploy) -

   npm run cdk deploy CI-Config-Dev -- -c useSsl=false -c runWithOidc=false

4. Optional Configure the elements of the config stack for SSL configuration

5. Optional Configure the elements setting up oidc via federate

6. Deploy the ci-stack, takes ~10 minutes to deploy (parameter values depend on step 2 and step 3)

   npm run cdk deploy CI-Dev -- -c useSsl=false -c runWithOidc=false

7. Log onto the AWS Console of the account, navigate to cloud watch, open log groups, looking for JenkinsMainNode/var/log/jenkins/jenkins.log

8. Search the logs for Jenkins initial setup is required. An admin user has been created and a password generated. After that entry the password for the jenkins instance will be in the cloudwatch logs.

9. Go to the CI-Dev.JenkinsExternalLoadBalancerDns url returned by CDK output to access the jenkins host.

10. If you want to destroy the stack make sure you delete the agent nodes manually (via jenkins UI or AWS console) so that shared resources (like vpc, security groups, etc) can be deleted.

Executing Optional Tasks

SSL Configuration

1. Locate the secret manager arns in the ci-config-stack outputs
2. Update the secret value (see docs) for the certContentsSecret with the certificate contents

$aws secretsmanager put-secret-value \
--secret-id MyTestDatabaseSecret_or_ARN \
--secret-string file://mycreds.json_or_value

3. Update the secret value (see docs) for the privateKeySecret with the certificate private key
4. Upload the certificate to IAM see docs
5. Update the secret value for the certificateArnSecret with the certificate arn generated by IAM
6. Run with parameter using one of the following (refer this for value of runWithOidc)
   1. npm run cdk deploy CI-Dev -- -c useSsl=true -c runWithOidc=true or,
   2. cdk deploy CI-Dev -c useSsl=true -c runWithOidc=true
7. Continue with next steps

Setup OpenId Connect (OIDC) via Federate

1. Locate the secret manager arns in the ci-config-stack outputs
2. Update the secret value (see docs) for the OIDCClientIdSecret with the credentials as json as follows:
   1. JSON format

    {
        "clientId": "example_id",
        "clientPassword": "example_password",
        "wellKnownOpenIDConfigurationUrl": "https://www.example.com",
        "tokenServerUrl": "https://example.com/token",
        "authorizationServerUrl": "https://example.com/authorize",
        "userInfoServerUrl": "https://example.com/userinfo"
    }
   

   2. Command Eg: see docs

    $aws secretsmanager put-secret-value \
    --secret-id MyTestDatabaseSecret_or_ARN \
    --secret-string file://mycreds.json_or_value
   

3. Run with parameter with one of the following (refer this for value of useSsL) -
   1. npm run cdk deploy CI-Dev -- -c runWithOidc=false -c useSsl=true or,
   2. cdk deploy CI-Dev -c runWithOidc=false -c useSsl=true
4. Continue with next steps

Troubleshooting

Main Node

Useful links

• Log are found in Cloud Watch Logs
• Need to access the host, ssh via Session Manager in EC2 Console
• Instance instance isn't coming up, get the system log in EC2 Console

Useful commands

• npm run build compile typescript to js, run lint, run tests
• npm run watch watch for changes and compile
• npm run cdk deploy deploy this stack to your default AWS account/region
• npm run cdk diff compare deployed stack with current state
• npm run cdk synth emits the synthesized CloudFormation template

Architecture Overview

Built using AWS Cloud Development Kit the configuration of the CI systems will be available for replication in your own accounts. The Jenkins instance will be hardened and publically visible, connected to GitHub to make build notifications easy for everyone to see.

Contributing

See developer guide and how to contribute to this project.

Getting Help

If you find a bug, or have a feature request, please don't hesitate to open an issue in this repository.

For more information, see project website and documentation. If you need help and are unsure where to open an issue, try forums.

Code of Conduct

This project has adopted the Amazon Open Source Code of Conduct. For more information see the Code of Conduct FAQ, or contact opensource-codeofconduct@amazon.com with any additional questions or comments.

Security

If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page. Please do not create a public GitHub issue.

License

This project is licensed under the Apache v2.0 License.

Copyright

Copyright OpenSearch Contributors. See NOTICE for details.