tool-compare

In the world of infrastructure-as-code security there are several tools for users to choose from. The goal of this repository is to help compare the different options so that users can choose the tool that best fits their own needs.

What tools are there?

	Checkov	Cloudrail	Kics	Snyk	Terrascan	Tfsec
Vendor	Bridgecrew	Indeni	Checkmarx	Snyk	Accurics	Aqua Security
License	OSS	Freemium	OSS	Freemium	OSS	OSS
Written in	Python	Python	Rego	Unknown	Rego	Go
Custom Rule Support	Yes	Yes	Yes	No	Yes	Yes
CI/CD-specific Integrations	CircleCI, GitLab, GitHub	CircleCI, GitLab, GitHub	GitHub	None	CircleCI, GitHub	CircleCI, GitHub
Output Formats (for generic CI/CD support)	Text, JSON, JUnit, SARIF	Text, JSON, JUnit, SARIF, GitLab-SAST	Text, JSON, SARIF, HTML	Text, JSON, SARIF, HTML	Text, JSON, JUnit	Text, JSON, JUnit, SARIF
Coverage for live environment	Not in OSS, use paid product	Yes, integrated into scans	No	No	Not in OSS, use paid product	Yes via differnet product

(there are others, anyone can add to this list, sorted A-Z)

For a list of IaC languages supported and the coverage provided by each tool for different CSPs, scroll down to the test case tables.

How does this repo work?

This repository has a set of test-cases and a main script, called run_all_tools.sh which runs the above-listed tools against each of the test-cases. This allows any potential user to see what the tool can do, and how it compares, before even installing it.

Test case catch rate

The tables below list test cases included in this repository. For each case, it shows which tools are able to catch it specifically, and which don't. Most test cases originate from the cloud service provider's (CSP's) own recommendations and best practices, as well as the CIS benchmark for that specific CSP.

Summary

Last update: 2021-08-27

	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
Tested Version	2.0.363	1.3.385	1.4.1	1.683.0	1.9.0	0.58.4
Terraform - AWS	69%	93%	94%	62%	73%	61%
Terraform - Azure	47%	35%	23%	30%	8%	18%
Terraform - Advanced Language Expressions	20%	100%	20%	0%	0%	0%
Total Catch Rate	59%	72%	65%	48%	47%	43%

test-cases/terraform/aws/best-practices

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
alb_drop_http_headers	✅	✅	✅	✅	❌	✅
cloudfront_not_using_waf	✅	✅	✅	✅	✅	✅
cloudtrail_enabled_on_multi_region	✅	✅	✅	✅	✅	✅
config_aggregator_all_regions	✅	✅	✅	✅	✅	✅
deploy_ec2_to_default_vpc	✅	✅	✅	❌	✅	✅
deploy_redshift_in_ec2_classic_mode	✅	✅	✅	❌	❌	✅
dynamodb_without_recovery_enabled	✅	✅	✅	✅	✅	✅
ec2_ebs_not_optimized	✅	❌	✅	❌	✅	❌
ecr_make_tags_immutable	✅	✅	✅	✅	✅	✅
ecr_use_image_scanning	✅	✅	✅	✅	✅	✅
ecs_cluster_container_insights	✅	✅	✅	✅	❌	✅
elasticache_automatic_backup	✅	❌	✅	✅	❌	✅
kms_uses_rotation	✅	✅	✅	✅	✅	✅
rds_retention_period_set	✅	❌	✅	❌	✅	✅
security_group_no_description_for_rules	✅	✅	✅	✅	✅	✅
security_group_no_description_for_securi..	❌	✅	✅	✅	✅	✅
security_group_no_unused	✅	✅	❌	❌	❌	❌
tag_all_items	❌	✅	✅	❌	❌	❌
using_public_amis	❌	✅	❌	❌	❌	❌
Sub-category Catch Rate	84%	84%	89%	63%	63%	79%

test-cases/terraform/aws/encryption/at-rest

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
athena_not_encrypted	✅	✅	✅	✅	✅	✅
cloudtrail_not_encrypted	✅	✅	✅	✅	✅	✅
cloudwatch_groups_not_encrypted	✅	✅	✅	✅	✅	✅
codbuild_using_aws_key	❌	✅	✅	❌	❌	❌
dax_cluster_not_encrypted	✅	✅	✅	✅	✅	✅
docdb_cluster_encrypted_at_rest_using_cm..	❌	✅	✅	❌	❌	✅
docdb_cluster_encrypted_without_kms_key	❌	✅	✅	✅	✅	✅
docdb_clusters_non_encrypted	✅	✅	✅	✅	✅	✅
dynamodb_not_encrypted	✅	✅	✅	❌	✅	✅
ecr_repo_not_encrypted	✅	✅	✅	✅	✅	✅
elasticache_replication_group_not_encryp..	✅	✅	✅	✅	✅	✅
elasticsearch_not_encrypted	✅	✅	✅	✅	✅	✅
kinesis_stream_not_encrypted	✅	✅	✅	✅	✅	✅
neptune_cluster_no_encryption	✅	✅	✅	✅	✅	✅
rds_cluster_encrypt_at_rest_disabled	✅	✅	✅	✅	✅	✅
redshift_not_encrypted	✅	✅	✅	✅	✅	✅
rest_api_cache_non_encrypted	❌	✅	✅	✅	❌	✅
s3_bucket_non_encrypted	✅	✅	✅	✅	✅	✅
s3_bucket_object_non_encrypted	❌	✅	✅	✅	✅	❌
sagemaker_not_encrypted	✅	✅	✅	✅	✅	❌
secretsmanager_secrets_encrypted_at_rest..	✅	✅	✅	✅	✅	✅
secretsmanager_secrets_encrypted_at_rest..	❌	✅	✅	❌	❌	✅
sns_topic_encrypted_at_rest_with_aws_man..	❌	✅	✅	❌	❌	✅
sqs_queue_not_encrypted	✅	✅	✅	✅	✅	✅
workgroups_non_encrypted	✅	✅	✅	✅	❌	✅
workspace_root_volume_not_encrypted_at_r..	✅	✅	✅	✅	✅	✅
workspace_user_volume_not_encrypted_at_r..	✅	✅	✅	✅	✅	✅
Sub-category Catch Rate	74%	100%	100%	81%	78%	89%

test-cases/terraform/aws/encryption/in-transit

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
alb_use_http	✅	✅	✅	❌	✅	✅
cloudfront_distribution_not_encrypted	✅	✅	✅	✅	✅	✅
cloudfront_protocol_version_is_low	✅	✅	✅	✅	✅	✅
ecs_task_definition_not_encrypted_in_tra..	✅	✅	✅	✅	✅	✅
elasticache_replication_group_not_encryp..	✅	✅	✅	✅	✅	✅
elasticsearch_encrypt_node_to_node_disab..	❌	✅	✅	✅	✅	✅
load_balancer_listener_http	✅	✅	✅	✅	✅	✅
vpc_has_only_dynamodb_vpce_gw_connection	❌	✅	❌	❌	❌	❌
Sub-category Catch Rate	75%	100%	88%	75%	88%	88%

test-cases/terraform/aws/iam/iam-entities

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
human_users_defined	✅	✅	✅	❌	❌	❌
iam_user_inline_policy_attach	✅	✅	✅	✅	✅	❌
iam_user_managed_policy_direct_attachmen..	✅	✅	✅	✅	✅	❌
passrole_and_lambda_permissions_cause_pr..	❌	✅	✅	❌	✅	❌
policy-too-broad	❌	✅	❌	❌	❌	❌
policy_missing_principal	❌	✅	✅	❌	✅	❌
public_and_private_ec2_same_role	❌	✅	✅	❌	❌	❌
role_assume_policy_principal_all	✅	✅	✅	✅	❌	❌
Sub-category Catch Rate	50%	100%	88%	38%	50%	0%

test-cases/terraform/aws/iam/resource-authentication

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
rds_without_authentication	✅	❌	✅	✅	✅	❌
rest_api_without_authorization	✅	✅	✅	✅	❌	❌
Sub-category Catch Rate	100%	50%	100%	100%	50%	0%

test-cases/terraform/aws/iam/resource-policies

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
cloudwatch_log_destination_insecure_poli..	❌	✅	✅	❌	❌	❌
ecr_not_secure_policy	❌	✅	✅	❌	✅	❌
efs_not_secure_policy	❌	✅	✅	❌	✅	❌
elasticsearch_domain_not_secure_policy	❌	✅	✅	❌	✅	❌
glacier_vault_not_secure_policy	✅	✅	❌	✅	❌	✅
glue_data_catalog_not_secure_policy	❌	✅	✅	❌	❌	❌
kms_key_not_secure_policy	❌	✅	✅	❌	✅	❌
lambda_not_secure_policy	❌	✅	✅	❌	❌	❌
rest_api_not_secure_policy	❌	✅	✅	❌	✅	❌
s3_bucket_acl_public_all_authenticated_u..	✅	✅	✅	✅	✅	✅
s3_bucket_acl_public_all_users_canned	✅	✅	✅	✅	✅	✅
s3_bucket_acl_public_all_users_canned_wi..	❌	✅	✅	❌	✅	❌
s3_bucket_policy_public_to_all_authentic..	❌	✅	✅	❌	✅	❌
secrets_manager_not_secure_policy	❌	✅	✅	❌	✅	❌
Sub-category Catch Rate	21%	100%	93%	21%	71%	21%

test-cases/terraform/aws/logging

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
api_gateway_no_xray	✅	✅	✅	✅	✅	✅
cloudfront_distribution_without_logging	✅	✅	✅	✅	✅	✅
cloudtrail_file_log_validation_disabled	✅	✅	✅	✅	✅	✅
cloudwatch_log_groups_no_retention	✅	✅	✅	✅	✅	❌
docdb_audit_logs_missing	✅	✅	✅	✅	✅	✅
ec2_without_monitoring	✅	❌	✅	❌	✅	❌
eks_logging_disabled	✅	✅	✅	✅	✅	✅
elasticsearch_domain_logging_disabled	✅	✅	✅	✅	✅	✅
elb_without_access_logs	✅	❌	✅	❌	✅	❌
globalaccelerator_accelerator_no_flow_lo..	✅	✅	✅	✅	✅	❌
lambda_without_explicit_log_group	❌	✅	❌	❌	❌	❌
lambda_without_xray	✅	✅	✅	✅	✅	✅
neptune_cluster_no_logging	✅	✅	✅	❌	✅	✅
rds_without_logging	✅	❌	✅	❌	✅	❌
redshift_without_logging	✅	✅	✅	✅	✅	❌
rest_api_no_access_logging	✅	✅	✅	✅	✅	✅
s3_access_logging_disabled	✅	✅	✅	✅	✅	✅
Sub-category Catch Rate	94%	82%	94%	71%	94%	59%

test-cases/terraform/aws/networking/vpc-endpoints

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
dynamodb-vpce-exist-without-routeassocia..	❌	✅	✅	❌	❌	❌
sqs-vpc-endpoint-without-dns-resolution	❌	✅	✅	❌	❌	❌
Sub-category Catch Rate	0%	100%	100%	0%	0%	0%

test-cases/terraform/azure/best-practices

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
defender_for_app_services_disabled	✅	❌	❌	❌	❌	❌
defender_for_container_registry_not_used	✅	✅	✅	✅	❌	✅
defender_for_keyvault_disabled	✅	❌	✅	✅	❌	✅
defender_for_kubernetes_not_used	✅	✅	✅	✅	❌	✅
defender_for_servers_not_used	✅	✅	✅	✅	❌	✅
defender_for_sql_servers_not_used	✅	✅	✅	✅	❌	✅
defender_for_storage_not_used	✅	✅	✅	✅	❌	✅
email_notifications_for_high_severity_al..	✅	✅	✅	✅	❌	✅
func_app_not_using_http2	✅	✅	❌	❌	❌	❌
func_app_not_using_latest_tls	❌	✅	❌	❌	❌	❌
functionapp_lin_java_isnot_latest	❌	❌	❌	❌	❌	❌
functionapp_python_isnot_latest	❌	❌	❌	❌	❌	❌
functionapp_win_java_isnot_latest	❌	❌	❌	❌	❌	❌
sql_vulnerability_assessment_not_enabled	✅	❌	❌	❌	❌	❌
sql_vulnerability_email_not_set	✅	❌	❌	✅	❌	❌
vm_unmanaged_disks	✅	❌	❌	❌	❌	❌
vmss_unmanaged_disks	❌	❌	❌	❌	❌	❌
vpn_gw_using_basic_sku	❌	✅	❌	❌	❌	❌
webapp_http2_not_enabled	✅	❌	❌	✅	❌	❌
webapp_lin_java_isnot_latest	❌	❌	❌	❌	❌	❌
webapp_php_isnot_latest	❌	❌	❌	❌	❌	❌
webapp_win_java_isnot_latest	❌	❌	❌	❌	❌	❌
Sub-category Catch Rate	59%	41%	32%	41%	0%	32%

test-cases/terraform/azure/encryption/at-rest

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
activitylog_storage_account_encryption_n..	❌	❌	❌	❌	❌	❌
sql_encryption_customer_key_not_set	❌	❌	❌	❌	❌	❌
storacc_encryption_not_enabled	✅	❌	❌	❌	❌	❌
Sub-category Catch Rate	33%	0%	0%	0%	0%	0%

test-cases/terraform/azure/encryption/in-transit

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
app_service_ftps_unused	❌	✅	❌	❌	❌	❌
app_service_use_most_recent_supported_tl..	✅	✅	❌	✅	❌	❌
func_app_ftps_not_required	❌	❌	❌	❌	❌	❌
mysql_not_forcing_ssl	✅	✅	✅	✅	✅	✅
postgresql_not_forcing_ssl	✅	✅	✅	✅	✅	✅
Sub-category Catch Rate	60%	80%	40%	60%	40%	40%

test-cases/terraform/azure/iam

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
app_service_authentication_missing	✅	✅	❌	✅	❌	❌
custom-role-owner-exists	✅	❌	✅	❌	❌	❌
func_app_authentication	✅	✅	❌	❌	❌	❌
func_app_client_cert_optional	❌	✅	❌	❌	❌	❌
functionapp_not_use_managedidentity	❌	❌	❌	❌	❌	❌
sql-server-ad-admin-not-set	❌	❌	❌	❌	❌	❌
storage_account_public_access_disabled	✅	❌	❌	❌	❌	❌
webapp_client_cert_not_enabled	✅	❌	❌	✅	❌	❌
webapp_not_use_managedidentity	✅	❌	❌	❌	❌	❌
Sub-category Catch Rate	67%	33%	11%	22%	0%	0%

test-cases/terraform/azure/logging

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
auto_prov_log_analytics_agent_disabled	❌	✅	❌	❌	❌	❌
batch_diagnostic_disabled	❌	❌	❌	❌	❌	❌
dl_analytics_diagnostic_not_enabled	❌	❌	❌	❌	❌	❌
dl_store_diagnostic_not_enabled	❌	❌	❌	❌	❌	❌
event_hub_diagnostic_not_enabled	❌	❌	❌	❌	❌	❌
iot_hub_diagnostic_not_enabled	❌	❌	❌	❌	❌	❌
logic_app_wf_diagnostic_not_enabled	❌	❌	❌	❌	❌	❌
postgresql_log_connections_not_enabled	✅	❌	✅	✅	✅	❌
postgresql_log_disconnections_not_enable..	❌	❌	✅	✅	✅	❌
postgresql_logcheckpoints_not_enabled	✅	❌	✅	✅	✅	❌
search_diagnostic_not_enabled	❌	❌	❌	❌	❌	❌
servicebus_namespace_not_enabled	❌	❌	❌	❌	❌	❌
sql-server-audit-retention-30	✅	✅	❌	❌	❌	❌
sql_server_audit_not_used	✅	✅	✅	✅	❌	✅
stream_analytics_diagnostic_not_enabled	❌	❌	❌	❌	❌	❌
vmss_win_diagnostic_log_disabled	❌	❌	❌	❌	❌	❌
Sub-category Catch Rate	25%	19%	25%	25%	19%	6%

test-cases/terraform/azure/networking

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
no_unused_nsg	❌	✅	❌	❌	❌	❌
public_access_sql_db	✅	✅	❌	❌	❌	✅
vm_public_rdp_lb_opened	❌	❌	❌	❌	❌	❌
vm_public_rdp_nat_opened	❌	❌	❌	❌	❌	❌
vmss_public_rdp_lb_opened	❌	❌	❌	❌	❌	❌
Sub-category Catch Rate	20%	40%	0%	0%	0%	20%

test-cases/terraform/hcl_language_complexity

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
using_count_and_ternary_expr	❌	✅	❌	❌	❌	❌
using_for_each	❌	✅	❌	❌	❌	❌
using_locals	❌	✅	✅	❌	❌	❌
using_module_multi	✅	✅	❌	❌	❌	❌
using_module_simple	❌	✅	❌	❌	❌	❌
Sub-category Catch Rate	20%	100%	20%	0%	0%	0%

Contributing

Anyone can contribute to this repository. The main areas of contribution are:

Adding an additional tool - simply add the tool to this readme and the run_all_tools.sh script. Then, execute that script and add all of its results as part of your PR. That's it, you're good to go.
Adding test-cases - you can add the test case in the correct spot in the tree under test-cases and run the run_all_tools.sh script against it. Make sure to include all of the tools' results as part of your PR.