cloudquery: A Go repository from katagun

The open-source cloud asset inventory powered by SQL.

CloudQuery extracts, transforms, and loads your cloud assets into normalized PostgreSQL tables. CloudQuery enables you to assess, audit, and monitor the configurations of your cloud assets.

CloudQuery key use-cases and features:

Search: Use standard SQL to find any asset based on any configuration or relation to other assets.
Visualize: Connect CloudQuery standard PostgreSQL database to your favorite BI/Visualization tool such as Grafana, QuickSight, etc...
Policy-as-Code: Codify your security & compliance rules with SQL as the query engine.

Supported providers (Actively expanding)

Checkout https://hub.cloudquery.io

If you want us to add a new provider or resource please open an Issue.

See docs for developing new provider.

Download & install

You can download the precompiled binary from releases, or using CLI:

export OS=Darwin # Possible values: Linux,Windows,Darwin
curl -L https://github.com/cloudquery/cloudquery/releases/latest/download/cloudquery_${OS}_x86_64 -o cloudquery
chmod a+x cloudquery
./cloudquery --help

# if you want to download a specific version and not latest use the following endpoint
export VERSION= # specifiy a version
curl -L https://github.com/cloudquery/cloudquery/releases/download/${VERSION}/cloudquery_${OS}_x86_64 -o cloudquery

Homebrew

brew install cloudquery/tap/cloudquery
# After initial install you can upgrade the version via:
brew upgrade cloudquery

Quick Start

Running

First generate a config.hcl file that will describe which resources you want cloudquery to pull, normalize and transform resources to the specified SQL database by running the following command:

cloudquery init aws # choose one or more from: [aws azure gcp okta]
# cloudquery init gcp azure # This will generate a config containing gcp and azure providers
# cloudquery init --help # Show all possible auto generated configs and flags

Once your config.hcl is generated run the following command to fetch the resources:

# you can spawn a local postgresql with docker
# docker run -p 5432:5432 -e POSTGRES_PASSWORD=pass -d postgres
cloudquery fetch --dsn "postgres://postgres:pass@localhost:5432/postgres?sslmode=disable"
# cloudquery fetch --help # Show all possible fetch flags

Using psql -h localhost -p 5432 -U postgres -d postgres

postgres=# \dt
                                    List of relations
 Schema |                            Name                             | Type  |  Owner
--------+-------------------------------------------------------------+-------+----------
 public | aws_autoscaling_launch_configuration_block_device_mapping   | table | postgres
 public | aws_autoscaling_launch_configurations                       | table | postgres

Run the following example queries from psql shell

List ec2_images

SELECT * FROM aws_ec2_images;

Find all public facing AWS load balancers

SELECT * FROM aws_elbv2_load_balancers WHERE scheme = 'internet-facing';

Running CloudQuery Policies

CloudQuery comes with out-of-the-box policies which you can use as is or modify to fit your use-case.

For example, to run AWS CIS policies enter the following commands (make sure you fetched all the resources beforehand by the fetch command):

./cloudquery policy run aws-cis-1.2.0

Full Documentation, resources and SQL schema definitions are available here.

Providers Authentication

See additional documentation for each provider at https://hub.cloudquery.io.

Compile and run

go build .
./cloudquery # --help to see all options

Running on AWS (Lambda, Terraform)

Checkout cloudquery/terraform-aws-cloudquery

License

By contributing to cloudquery you agree that your contributions will be licensed as defined on the LICENSE file.

Hiring

If you are into Go, Backend, Cloud, GCP, AWS - ping us at jobs [at] our domain

Contribution

Feel free to open Pull-Request for small fixes and changes. For bigger changes and new providers please open an issue first to prevent double work and discuss relevant stuff.

katagun/cloudquery

Links