/cloudquery

cloudquery powered by Osquery

Primary LanguageGoOtherNOASSERTION

Build CodeQL Go Report Card FOSSA Status Contributor Covenant

cloudquery powered by Osquery

cloudquery is Osquery extension to fetch cloud telemetry from AWS, GCP, and Azure. It is extensible so that one can add support for new tables easily, and configurable so that one can change the table schema as well.

Contents


Build and test extension

Build

  • Install prerequisites
  • Set environment variable for extension home (it shoud be path-to-repo/cloudquery/extension)
    export CLOUDQUERY_EXT_HOME=/home/user/work/cloudquery/extension
  • Build the extension:
    make

Test

With osqueryi

  • Copy extension configuration sample file:
    cp ${CLOUDQUERY_EXT_HOME}/extension_config.json.sample ${CLOUDQUERY_EXT_HOME}/config/extension_config.json
  • Edit ${CLOUDQUERY_EXT_HOME}/config/extension_config.json with your cloud accounts. You can add multiple accounts for each cloud provider. Change logging path and other parameters to suit your needs. Make sure log path is writable.
  • Start osqueryi
    osqueryi --nodisable_extensions --extension ${CLOUDQUERY_EXT_HOME}/../cloudquery
  • Query data:
    SELECT account_id, region_code, image_id, image_type FROM aws_ec2_image;

With osqueryd

  • Build and install cloudquery:
    make build
    sudo make install
  • Edit (or create) /etc/osquery/extensions.load file and append the following line: /usr/local/bin/cloudquery.ext
  • Edit /opt/cloudquery/config/extension_config.json with your cloud accounts. You can add multiple accounts for each cloud provider. Change logging path and other parameters to suit your needs.
  • Add following flags to /etc/osquery/osquery.flags (your flag file path could be different)
--extensions_autoload=/etc/osquery/extensions.load
--disable_extensions=false
  • Restart osquery service:
    sudo service osqueryd restart

Test with docker

Setup credentials

Setup credentials before proceeding to testing with osqueryi or osqueryd

  • Create a config directory on the host to hold the credentials for your cloud accounts (~/config is an example, but this could be any directory).

  • Make a copy of extension_config.json.sample as extension_config.json in a directory called config (can be anywhere on your machine)

  • Copy cloud credentials to the config directory

    • For AWS: $HOME/.aws/credentials
    • For GCP: your-serviceAccount.json or any JSON file that contains GCP credentials
    • For Azure: my.auth or any file that holds Azure credentials
  • If using AWS cloud, update the following fields in aws section in config/extension_config.json file:

  • If using Google cloud, update keyFile in gcp section in extension_config.json file. It should be changed to /opt/cloudquery/etc/config/your-serviceAccount.json where your-serviceAccount.json is the JSON key file that contains GCP credentials

  • If using Azure, update the following fields in azure section in extension_config.json file:

Run osqueryi inside cloudquery container

docker run --rm -it --name cloudquery \
  -v <absolute path to host config directory>:/opt/cloudquery/etc/config \
  uptycs/cloudquery:latest \
  osqueryi --extension /usr/local/bin/cloudquery.ext

Run osqueryd from cloudquery container

Following files and directories can be mounted from the host:

  • /opt/cloudquery/logs - Directory that contains the logs
  • /opt/cloudquery/etc/osquery.flags - Osquery flags file
  • /opt/cloudquery/etc/osquery.conf - Osquery configuration JSON file
  • /opt/cloudquery/etc/config - Directory that contains Cloud provider credentials and cloudquery configuration JSON

Sample Osquery configuration with scheduled queries that can be overwritten via osquery.conf:

{
  "schedule": {
    "gcp_compute_network": {
      "query": "SELECT * FROM gcp_compute_network;",
      "interval": 120
    },
    "aws_s3_bucket": {
      "query": "SELECT * FROM aws_s3_bucket;",
      "interval": 120
    },
    "azure_compute_vm": {
      "query": "SELECT * FROM azure_compute_vm;",
      "interval": 120
    }
  }
}
docker run --rm -d --name cloudquery \
  -v <absolute path to host config directory>:/opt/cloudquery/etc/config \
  uptycs/cloudquery:latest

Supported tables