The Censys Unified Cloud Connector is a standalone connector that gathers assets from various cloud providers and stores them in Censys ASM. This Connector offers users the ability to supercharge our ASM Platform with total cloud visibility. This connector currently supports the following cloud providers: Azure and GCP. Support for AWS and other cloud providers will be added in the future.
The following platforms and services are supported and will be used to import Seeds (IP Addresses, Domain Names, CIDRs, and ASNs) as well as Cloud Assets (Object Storage Buckets) into the Censys ASM platform.
- Google Cloud Compute
- Google Cloud Containers
- Google Cloud Networking
- Google Cloud Databases
- Google Cloud Storage
It is important to note that this connector is a Python package. This allows
you to run the connector from the command line as well as enables you to run
the connector in as many different environments as you wish. We have provided
a variety of deployment types and configuration options. We recommend that you
install the package locally to take advantage of the configuration command line
interface (censys-cc config
). After you have configured the connector, you
can deploy it to your environment. In the following sections, we will provide
a brief overview of how to deploy the connector to your environment.
# Clone the repository
git clone https://github.com/censys/censys-cloud-connector.git
cd censys-cloud-connector
# Ensure you have poetry installed
pip install --upgrade poetry
# Recommended installation
poetry install -E azure -E gcp # All dependencies (This is recommended)
# Other installations
# poetry install -E azure # Only Azure dependencies
# poetry install -E gcp # Only GCP dependencies
# Copy .env file
cp .env.sample .env
The following environment variables are available for use in the connector:
CENSYS_API_KEY
- Your Censys ASM API key found in the ASM Integrations Page. (Required)PROVIDERS_CONFIG_FILE
- The path to theproviders.yml
file.SECRETS_DIR
- The path to the directory containing the secrets.LOGGING_LEVEL
- The logging level. Valid values areDEBUG
,INFO
,WARN
,ERROR
, andCRITICAL
.DRY_RUN
- If set totrue
, the connector will not write any data to the ASM platform. This is useful for testing.
.env.sample
is a sample file that contains the above environment variables.
Please use this file as a template to create your own .env
file.
To configure the connector, you can use the command line interface. The base
command is censys-cc
. The configuration command is:
poetry run censys-cc config # Configure supported providers
The censys-cc config
command will guide you through the configuration of
supported cloud providers. This command will assist you in generating a
providers.yml
file. This file can contain multiple provider configurations.
You can optionally specify a provider in the command line with the flag
--provider
.
Before configuring the connector, make sure you are logged in to your cloud provider's CLI tool. See our supported providers below for more information.
You have successfully configured your cloud connector if your providers.yml file is populated with your credentials.
Log in to your cloud provider's CLI tool using the following commands:
-
Google's gcloud CLI:
gcloud auth login
-
AWS CLI:
aws configure
-
Azure CLI:
az login
The providers.yml
file contains the configuration for all cloud providers.
The file is a YAML file and is structured as follows:
You will need to have generated your
providers.yml
file using thecensys-cc config
command before you can run the connector.
- provider: aws
account_number: xxxxxxxxxxxx
access_key: xxxxxxxxxxxxxxxxxxxx
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
regions:
- xxxxxxxxx
# The ignore field takes a list of Azure resource types to ignore during scanning.
# ignore:
# - AWS::ApiGateway
# - AWS::ECS
# - AWS::ElasticLoadBalancing
# - AWS::NetworkInterface
# - AWS::RDS
# - AWS::Route53
# - AWS::S3
# It is also possible to define roles to assume for multiple accounts.
# accounts:
# - account_number: xxxxxxxxxxxx
# access_key: xxxxxxxxxxxxxxxxxxxx
# secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# role_name: xxxxxxxxxxxxxxxxxxxx
- provider: azure
tenant_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
client_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
client_secret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
subscription_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
# The subscription_id field takes one or more subscription IDs.
# subscription_id:
# - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
# - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
# The ignore field takes a list of Azure resource types to ignore during scanning.
# ignore:
# - Microsoft.Network/publicIPAddresses
# - Microsoft.ContainerInstance/containerGroups
# - Microsoft.Sql/servers
# - Microsoft.Network/dnszones
# - Microsoft.Storage/storageAccounts
- provider: gcp
organization_id: xxxxxxxx-xxxx-xxxx
service_account_json_file: service_account.json
service_account_email: censys-cloud-connector@project-id.iam.gserviceaccount.com
# The ignore field takes a list of GCP resource types to ignore during scanning.
# ignore:
# - google.compute.Address
# - google.container.Cluster
# - google.cloud.sql.Instance
# - google.cloud.dns.ManagedZone
# - google.cloud.storage.Bucket
To run the connector, you can use the command line interface.
poetry run censys-cc scan # Scan cloud assets
The censys-cc scan
command runs the connector.
- The connector will scan for assets from all providers in
providers.yml
. - The connector will run once.
-
You can specify one or more providers in the command line with the flag
--provider
. The connector will only scan for assets from the specified providers. -
You can set a scheduled interval for the connector to run on with the flag
--daemon
. This option takes in a time interval in hours. If you do not specify an interval, the default will be set to 1 hour.censys-cc scan --daemon # Run every 1 hour censys-cc scan --daemon 1.5 # Run every 1.5 hours
We offer several Terraform deployment options for you to choose from. These options deploy the connector to the serverless environment in your provider's cloud.
This option deploys the connector to GCP as a Google Cloud Function.
Coming Soon!
Coming Soon!
This option deploys the connector to Azure as a Azure Function.
This method assumes you have Docker installed and running on your server.
-
Authenticate to the GitHub Container Registry
-
Pull the Docker image
docker pull gcr.io/censys-io/censys-cloud-connector:latest
-
If your environment does not allow you to pull the Docker image, you can build it from the Dockerfile using the following command. You can then push the image to a Docker registry.
docker build -t gcr.io/censys-io/censys-cloud-connector:latest .
- Run the Docker container
The following command will run the Docker container. You can specify the
environment variables you want to pass to the container using the -e
flag.
The container also requires the providers.yml
file. The -v
flag will
mount the providers.yml
file as a volume. If your providers.yml
references
additional secret files, you can mount it as a volume as well. The -d
flag
is used to run the container in the background. We also include the --rm
flag to ensure the container is removed after it has finished.
# Ensure you have sourced your environmental variables
source .env
# Mount the providers.yml and secrets files as volumes
docker run -d --rm \
-e "CENSYS_API_KEY=$CENSYS_API_KEY" \
-v $(pwd)/providers.yml:/app/providers.yml \
-v $(pwd)/secrets:/app/secrets \
gcr.io/censys-io/censys-cloud-connector:latest \
scan --daemon 4
# Alternatively if you do not need the secrets volume
docker run -d --rm \
-e "CENSYS_API_KEY=$CENSYS_API_KEY" \
-v $(pwd)/providers.yml:/app/providers.yml \
gcr.io/censys-io/censys-cloud-connector:latest \
scan --daemon 4
# Additionally if you only need to scan once
docker run --rm \
-e "CENSYS_API_KEY=$CENSYS_API_KEY" \
-v $(pwd)/providers.yml:/app/providers.yml \
-v $(pwd)/secrets:/app/secrets \
gcr.io/censys-io/censys-cloud-connector:latest
More information about the
--daemon
flag is found here.
This method assumes you have Docker and Docker Compose installed and running on your server.
- Run the Docker Compose file
docker-compose up -d
- [Optional] Run your connector on a scheduled interval
Uncomment the line # command: scan --daemon 4
in
docker-compose.yml.
Details about the --daemon
option can be found here.
This method assumes you have Kubernetes installed and running on your server.
Visit the Seed Data Page and the Storage Buckets Page to confirm that you're seeing seeds and storage buckets from your cloud provider(s).
In the case where the user has just run the censys-cc config
command
for Azure and then promptly runs the censys-cc scan
command, the scan may
fail with a ClientSecretCredential.get_token failed
exception. This is due
to the fact that Azure is in the process of creating the service principal.
Please wait a few minutes and try again.
Example error message:
ClientSecretCredential.get_token failed: Authentication failed: AADSTS7000215:Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'.
It is highly recommended that a Python version shim like pyenv is used. Once installed, Poetry will make a virtualenv using the correct version of Python automatically.
The following permissions are required to scan:
route53:ListHostedZones
,elasticloadbalancing:DescribeLoadBalancers
,route53domains:ListDomains
,ec2:DescribeNetworkInterfaces
,rds:DescribeDBInstances
,route53:ListResourceRecordSets
,ecs:ListContainerInstances
,apigateway:GET
,s3:GetBucketLocation
,s3:ListBucket
,s3:ListAllMyBuckets
,ecs:ListClusters
Ensure the account's Access control (IAM) role has the following permission to create a service principal with a Reader role:
Microsoft.Authorization/roleAssignments/write
over scope/subscriptions/uuid
The following permissions will be used with this service principal:
Microsoft.ContainerInstance/containerGroups/read
Microsoft.Network/dnszones/read
Microsoft.Network/publicIPAddresses/read
Microsoft.Sql/servers/read
Microsoft.Storage/storageAccounts/read
If you see the following error message, check that you are logged into an account with the correct permissions:
The client 'user@example.com' with object id 'uuid' does not have authorization to perform action 'Microsoft.Authorization/roleAssignments/write' over scope '/subscriptions/uuid' or the scope is invalid. If access was recently granted, please refresh your credentials.
If you encounter the following error while configuring your GCP Cloud Connector, a likely cause is that your service account has reached its maximum quota of keys.
Failed to enable service account. ERROR: (gcloud.iam.service-accounts.keys.create) FAILED_PRECONDITION: Precondition check failed.
Go to https://console.cloud.google.com/iam-admin/serviceaccounts to manage your service account keys.
All contributions (no matter how small) are always welcome. See Contributing to the Cloud Connector to change or test the code or for information on the CI/CD pipeline.
This software is licensed under Apache License, Version 2.0
- Copyright (C) 2022 Censys, Inc.