Benchmark collection for Analysis
With this it intends to validate several static code analysis software SAST, in order to create a battery of benchmark tests. In the end it is intended to have a substantial set of common languages and updated code, over time, with the purpose of always having new or improved vulnerabilities.
Projects to Vulnerability Benchmark
Languages to validate
| Status | Icon | Description |
|---|---|---|
| Deprecated | 🙅 | The evaluation of results will not be continued for the language or project. |
| ToDo | 😥 | This list to be made the evaluation of results, in the scope of recommendation or initial proposal. |
| Ongoing | ✍ | This is in the process of analysis and development at the moment, so unexpected changes may happen. |
| Done | 🙌 | Analysis and evaluation of results successfully completed. |
Note: You can always suggest languages, projects or changes. For this we recommend that you open an issue please.
| Languages | ASP | Apex | CPP | CSharp | Cobol | Go | Groovy | Java | JavaScript | Kotlin | Objc | PHP | PLSQL | Perl | Python | Ruby | Scala | Swift | Typescript | VB6 | VbNet | VbScript |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Status | 😥 | ✍ | ✍ | 😥 | 🙅 | 😥 | 😥 | ✍ | 😥 | ✍ | 😥 | ✍ | 😥 | 😥 | ✍ | ✍ | ✍ | ✍ | ✍ | 🙅 | 😥 | 🙅 |
| SAST Scan Used | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx | CheckMarx |
All the languages presented above will be mapped by openSource projects in which we will have the validation and analysis of the results.
🕵🏿 Results and Projects
ASP
Apex
CPP
CSharp
Cobol
Go
Groovy
Java or Android
- Java Report
- Android
JavaScript
Kotlin
Objc
PHP
PLSQL
Perl
Python and FrameWorks
- Python
- Django
Scala
Swift
VB6
VbNet
VbScript
Common or Collections
Note: Some languages are together because they have a hundredth of a relationship. Example of this are Typescript projects are together with those of Node, Angular, React, etc..
Scan tools we intend to use to make comparisons of results and evaluate false positives
| Application/Tools | Languages / frameworks | Util Links | Notes |
|---|---|---|---|
| .NET Security Guard | .NET, CSharp, VB.net | .NET Security Guard | Detects various security vulnerability patterns: SQL Injection, Cross-Site Scripting (XSS), Cross-Site Request Forgery (CSRF), XML eXternal Entity Injection (XXE), etc. Basic intraprocedural taint analysis for input data. Analyzes .NET and .NET Core projects in a background (IntelliSense) or during a build. |
| Agnitio | ASP, ASP.NET, CSharp, Java, Javascript, Perl, PHP, Python, Ruby, VB.NET, XML | Agnitio | A tool to help developers and security professionals conduct manual security code reviews in a consistent and repeatable way. Agnitio aims to replace the adhoc nature of manual security code review documentation, create an audit trail and reporting. Features Security code reviews; Security code review metrics and reporting |
| Anchore Engine | All (to Validat in Docker) | anchore-engine | The Anchore Engine is an open-source project that provides a centralized service for inspection, analysis, and certification of container images. The Anchore Engine is provided as a Docker container image that can be run standalone or within an orchestration platform such as Kubernetes, Docker Swarm, Rancher, Amazon ECS, and other container orchestration platforms. |
| APIsecurity.io Security Audit | API | APIsecurity.io Security Audit | Online tool for OpenAPI / Swagger file static security analysis. |
| Bandit | Python | bandit | Bandit is a comprehensive source vulnerability scanner for Python. |
| Brakeman | Ruby on Rails | brakeman | Brakeman is an open source vulnerability scanner specifically designed for Ruby on Rails applications |
| Checkov | All (to Validat) | checkov | Prevent cloud misconfigurations during build-time for Terraform, Cloudformation, Kubernetes, Serverless framework and other infrastructure-as-code-languages with Checkov by Bridgecrew. |
| Clair | Container / Docker | clair | Clair is an open source project for the static analysis of vulnerabilities in application containers (currently including OCI and docker) |
| Codesake Dawn | Ruby | codesake-dawn | Dawn is a security source code scanner for ruby powered code. Starting from January 07, 2015 this gem is renamed to dawnscanner and this version is no longer supported. Please, upgrade your Gemfile. |
| CodeSec | C, C++, CSharp, Java, JavaScript, PHP, Kotlin, Lua, Scala, TypeScript, Android | CodeSec | Open source network security code audit platform(CodeSec)is mainly used in code security audit and quality analysis, supporting mainstream coding specifications, backdocle code detection, distributed engine deployment. CodeSec's research and development team, after years of in-depth analysis and requirements research on AST technology, focuses on the accuracy of results and the ease of use of tools, and is more suitable for DevSecOps scenarios to improve code security. |
| CodeSonar | C, C++, Java | CodeSonar | CodeSonar is a static code analysis tool from GrammaTech. CodeSonar is used to find and fix bugs and security vulnerabilities in source and binary code. It performs whole-program, inter-procedural analysis with abstract interpretation on C, C++, C#, Java, as well as x86 and ARM binary executables and libraries. |
| Coverity | Android, CSharp, C, C++, Java, JavaScript, Node.js, Objective-C, PHP, Python, Ruby, Scala, Swift, VB.NET | Coverity | CodeSonar supports many popular languages, including C/C++, Java, C# and Android, as well as support for native binaries in Intel, Arm and PowerPC instruction set architectures. CodeSonar also supports OASIS SARIF, for exchange of information with other tools in the DevSecOps environment. Documentation |
| Dawnscanner | Ruby | Dawnscanner | Dawnscanner is an open source security source code analyzer for Ruby, supporting major MVC frameworks like Ruby on Rails, Padrino, and Sinatra. It also works on non-web applications written in Ruby. |
| Deep Dive | Jar, War, other | Deep Dive | Byte code analysis tool for discovering vulnerabilities in Java deployments (EAR, WAR, JAR). |
| DevBug | PHP | DevBug | DevBug is a basic PHP Static Code Analysis (SCA) tool written mostly in JavaScript. The idea behind DevBug is to make basic PHP Static Code Analysis accessible online, to raise security awareness and to integrate SCA into the development process. DevBug could be used to quickly test a page of PHP that you think may have some potential vulnerabilities, to run across a piece of code you have found on Google that you are unsure of or to directly write your own code in. |
| ESLint react plugin | React | ESLint react plugin | React specific linting rules for ESLint |
| ESLint security plugin | JavaScript, TypeScript | ESLint security plugin | ESLint rules for Node Security. This project will help identify potential security hotspots, but finds a lot of false positives which need triage by a human. |
| FindSecBugs | Java, Scala, Groovy | Find Security Bugs | A security specific plugin for SpotBugs that significantly improves SpotBugs's ability to find security vulnerabilities in Java programs. Works with the old FindBugs too. |
| Flawfinder | C/C++ | Flawfinder | Flawfinder is a simple program that scans C/C++ source code and reports potential security flaws. It can be a useful tool for examining software for vulnerabilities, and it can also serve as a simple introduction to static source code analysis tools more generally. It is designed to be easy to install and use. Flawfinder supports the Common Weakness Enumeration (CWE) and is officially CWE-Compatible. |
| Fortify | ABAP/BSP, ActionScript/MXML (Flex), ASP.NET, VB.NET, CSharp (.NET), C/C++, Classic ASP (w/VBScript), COBOL, ColdFusion CFML, HTML, Java (including Android), JavaScript/AJAX, JSP, Objective-C, PHP, PL/SQL, Python, T-SQL, Ruby, Swift, Visual Basic, VBScript, XML | Fortify | Automated static code analysis helps developers eliminate vulnerabilities and build secure software. |
| Git Hound | All (to Validat) | git-hound | Git plugin that prevents sensitive data from being committed. |
| Git-Secrets | All (to Validat) | git-secrets | Prevents you from committing secrets and credentials into git repositories. |
| GolangCI-Lint | Go | GolangCI-Lint | A Go Linters aggregator - One of the Linters is (Go Security), which is off by default but can easily be enabled. |
| Google CodeSearchDiggity | All (to Validat) | Google CodeSearchDiggity | Uses Google Code Search to identify vulnerabilities in open source code projects hosted by Google Code, MS CodePlex, SourceForge, Github, and more. The tool comes with over 130 default searches that identify SQL injection, cross-site scripting (XSS), insecure remote and local file includes, hard-coded passwords, and much more. Essentially, Google CodeSearchDiggity provides a source code security analysis of nearly every single open source code project in existence – simultaneously. |
| Gosec | Go | Gosec | Inspects source code for security problems by scanning the Go AST. |
| Graudit | All (to Validat) | graudit | Scans multiple languages for various security flaws. Basically security enhanced code Grep. |
| HCL AppScan CodeSweep | Python, Ruby, JS (Node, Angular, JQuery, etc) , PHP, Perl, COBOL, APEX | HCL AppScan CodeSweep | This is the first Community edition version of AppScan. It is delivered as a VS Code plugin and scans files upon saving them. The results show the location of a finding, type and remediation advice. The tool currently supports Python, Ruby, JS (Node, Angular, JQuery, etc) , PHP, Perl, COBOL, APEX & a few more. |
| HCL AppScan Source | Android, Apex, ASP, C, C++, COBOL, ColdFusion, Go, Java, JavaScript(Client-side JavaScript, NodeJS, and AngularJS), .NET (CSharp, ASP.NET, VB.NET), .NET Core, Perl, PHP, PL/SQL, Python, Ruby, T-SQL, Visual Basic 6 | HCL AppScan Source | Static application security testing (SAST) solution that helps identify vulnerabilities early in the development lifecycle, understand their origin and potential impact and remediate the problem. |
| Horusec | Python(3.x), Ruby, Javascript, GoLang, .NetCore(3.x), Java, Kotlin, Terraform | Horusec | Horusec is an open source tool that improves identification of vulnerabilities in your project with just one command. |
| HuskyCI | Python, Ruby, JavaScript, Golang, and Java | HuskyCI | HuskyCI is an open-source tool that orchestrates security tests inside CI pipelines of multiple projects and centralizes all results into a database for further analysis and metrics. HuskyCI can perform static security analysis in Python (Bandit and Safety), Ruby (Brakeman), JavaScript (Npm Audit and Yarn Audit), Golang (Gosec), and Java(SpotBugs plus Find Sec Bugs) |
| Insider CLI | Java Maven and Android), Kotlin (Android), Swift (iOS), .NET Full Framework, CSharp, and Javascript (Node.js) | Insider CLI | A open source Static Application Security Testing tool (SAST) written in GoLang for Java Maven and Android), Kotlin (Android), Swift (iOS), .NET Full Framework, C#, and Javascript (Node.js). |
| Klocwork | C, C++, CSharp, Java | Klocwork | Klocwork static application security testing (SAST) for C, C++, C#, and Java identifies software security, quality, and reliability issues helping to enforce compliance with standards. |
| Kubesec | Kubernetes manifests, Helm Charts | Kubesec | Security risk analysis for Kubernetes resources |
| LGTM | C/C++, CSharp, Go, Java, JavaScript/TypeScript, Python | LGTM | A free for open source static analysis service that automatically monitors commits to publicly accessible code in Bitbucket Cloud, GitHub, or GitLab. Supports C/C++, C#, Go, Java, JavaScript/TypeScript, Python. |
| Microsoft FxCop | .NET | Microsoft FxCop | FxCop is an application that analyzes managed code assemblies (code that targets the .NET Framework common language runtime) and reports information about the assemblies, such as possible design, localization, performance, and security improvements. |
| Microsoft PREFast | C, C++ | Microsoft PREFast | PREfast is a static analysis tool that identifies defects in C/C++ programs. PREfast enables you to perform quick desktop error detection on small code bases. |
| MobSF | Java (Android), Kotlin (Android), Objective C, Swift, | MobSF (beta) | Mobile Security Framework (MobSF) is an automated, all-in-one mobile application (Android/iOS/Windows) pen-testing, malware analysis and security assessment framework capable of performing static and dynamic analysis. |
| NextGen Static Analysis | CSharp, Go, Java, JavaScript, Python, Scala | NextGen Static Analysis | NextGen Static Analysis (NG SAST) is a modern code analysis solution, purpose-built to support developer workflows. NG SAST has the speed, accuracy, and comprehensiveness to confidently shift code analysis left by eliminating manual bottlenecks and embracing automation. |
| NodeJsScan | Node.js | NodeJsScan | Nodejsscan is a static security code scanner for Node.js applications. |
| OWASP ASST | JavaScript (Node.js framework), PHP, MySQL | OWASP ASST | An Open Source, Source Code Scanning Tool, developed with JavaScript (Node.js framework), Scans for PHP & MySQL Security Vulnerabilities According to OWASP Top 10 and Some other OWASP's famous vulnerabilities, and it teaches developers of how to secure their codes after scan. |
| OWASP Code Crawler | .NET / Java | OWASP Code Crawler | A tool aimed at assisting code review practitioners. It is a static code review tool which searches for key topics within .NET and J2EE/JAVA code. It's a Microsoft .NET 3.5 Windows Form application which supports the OWASP Code Review Project. It provides automatic STRIDE classification a very simple DREAD calculator and few minor utilities. |
| OWASP LAPSE Project | Java | OWASP LAPSE Project | The OWASP Lapse Project is LAPSE+: The Security Scanner for Java EE Applications. OWASP LAPSE Project is an initiative to make available to developers and auditors a tool for detecting vulnerabilities in Java EE Applications. |
| OWASP Orizon Project | Java | OWASP Orizon Project | OWASP Orizon is a source code security scanner designed to spot vulnerabilities in J2EE web applications, Android code and generally speaking in Java written source code. |
| OWASP WAP | PHP | OWASP WAP | WAP is a tool to detect and correct input validation vulnerabilities in web applications written in PHP and predicts false positives. The tool combines source code static analysis and data mining to detect vulnerabilities and predict false positives. |
| ParaSoft | C, C++, Java, .NET | ParaSoft | Deploy Parasoft static analysis, dynamic analysis, unit testing, and code coverage for software testing of embedded systems to ensure they are safe, secure, and reliable. |
| Phpcs Security Audit | PHP | phpcs-security-audit | A set of PHP_CodeSniffer rules to finds flaws or weaknesses related to security in PHP and its popular CMS or frameworks. It currently has core PHP rules as well as Drupal 7 specific rules. |
| PMD | Apex (Salesforce) | PMD | PMD scans Java source code and looks for potential code problems (this is a code quality tool that does not focus on security issues). |
| Polyspace Static Analysis | C, C++, Ada | Polyspace Static Analysis | Static code analysis products use formal methods to prove the absence of critical run-time errors under all possible control flows and data flows. They include checkers for coding rules, security vulnerabilities, code metrics, and hundreds of additional classes of bugs. |
| PreFast | C, C++ | PreFast | PREfast is a static analysis tool that identifies defects in C/C++ programs. Last update 2006. |
| Progpilot | PHP | progpilot | Progpilot is a static analyzer tool for PHP that detects security vulnerabilities such as XSS and SQL Injection. |
| Puma Scan Professional | .NET, CSharp | Puma Scan Professional | The Puma Scan open source project is where it all started. |
| PVS-Studio | C, C++, CSharp | PVS-Studio | PVS-Studio is a tool for detecting bugs and security weaknesses in the source code of programs, written in C, C++, C# and Java. |
| Pyre | Python | Pyre | A performant type-checker for Python 3, that also has data flow analysis capabilities. |
| Security Checker | PHP | security-checker | The SensioLabs Security Checker is a command line tool that checks if your application uses dependencies with known security vulnerabilities. It uses the Security Check Web service and the Security Advisories Database. |
| Security Code Scan | .NET Core, CSharp VB.NET, .NET Framework | Security Code Scan | Static code analyzer for .NET. It will find SQL injections, LDAP injections, XXE, cryptography weakness, XSS and more. |
| Semgrep | C, C++, CSharp | Semgrep | Like Grep, for code. A lightweight static analysis tool with intuitive rule syntax for searching code. Scans source code. No compilation required. Supports Python, JavaScript, Go, Java, C. |
| Semmle | All (to Validat) | Semmle | A code analysis platform for finding zero-days and automating variant analysis. |
| ShiftLeft Scan | All (to Validat) | ShiftLeft Scan | A free open-source DevSecOps platform for detecting security issues in source ode and dependencies. It supports a broad range of languages and CI/CD pipelines by bundling various open source scanners into the pipeline. |
| Sink Tank | Java | Sink Tank | Java byte code static code analyzer for performing source/sink (taint) analysis. |
| Snyk | All (to Validat) | snyk | CLI and build-time tool to find & fix known vulnerabilities in open-source dependencies |
| Sobelow | Elixir (Phoenix) | Sobelow | Sobelow is a security-focused static analysis tool for the Phoenix framework. For security researchers, it is a useful tool for getting a quick view of points-of-interest. For project maintainers, it can be used to prevent the introduction of a number of common vulnerabilities. |
| SonarCloud | ABAP, C, C++, Objective-C, COBOL, CSharp, CSS, Flex, Go, HTML, Java, Javascript, Kotlin, PHP, PL/I, PL/SQL, Python, RPG, Ruby, Swift, T-SQL, TypeScript, VB6, VB, XML | SonarCloud | Eliminate bugs and vulnerabilities. Champion quality code in your projects. |
| SonarQube | Java, JavaScript, CSharp, TypeScript, Kotlin, Ruby, Go, Scala, Flex, Python, PHP, HTML, CSS, XML and VB.NET | SonarQube | Scans source code for 15 languages for Bugs, Vulnerabilities, and Code Smells. SonarQube IDE plugins for Eclipse, Visual Studio, and IntelliJ provided by SonarLint. |
| Splint | C | Splint | Splint is a tool for statically checking C programs for security vulnerabilities and coding mistakes. With minimal effort, Splint can be used as a better lint. If additional effort is invested adding annotations to programs, Splint can perform stronger checking than can be done by any standard lint. |
| SpotBugs | Groovy, Java, Scala | SpotBugs | SpotBugs is a program which uses static analysis to look for bugs in Java code. It is free software, distributed under the terms of the GNU Lesser General Public License. |
| Sqlmap | All (to Validat) | sqlmap | sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. |
| Sslyze | SSL/TLS | sslyze | SSLyze is a fast and powerful SSL/TLS scanning library. It allows you to analyze the SSL/TLS configuration of a server by connecting to it, in order to detect various issues (bad certificate, weak cipher suites, Heartbleed, ROBOT, TLS 1.3 support, etc.). |
| TFSec | Terraform code | tfsec | tfsec uses static analysis of your terraform templates to spot potential security issues. Now with terraform v0.12+ support. |
| Trivy | All (to Validat to Container) | trivy | A Simple and Comprehensive Vulnerability Scanner for Containers, Suitable for CI |
| TruffleHog | All (to Validat) | truffleHog | Searches through git repositories for secrets, digging deep into commit history and branches. This is effective at finding secrets accidentally committed. |
| Veracode | Android, ASP.NET, CSharp, C, C++, Classic ASP, COBOL, ColdFusion/Java, Go, Groovy, iOS, Java, JavaScript, Perl, PhoneGap/Cordova, PHP, Python, React Native, RPG, Ruby on Rails, Scala, Titanium, TypeScript, VB.NET, Visual Basic 6, Xamarin | Veracode | Veracode offers a holistic, scalable way to manage security risk across your entire application portfolio. We provide visibility into application status across all common testing types in a single view. |
| VisualCodeGrepper | C/C++, CSharp, VB, PHP, Java, PL/SQL, and COBOL | VisualCodeGrepper | VCG is an automated code security review tool for C++, C#, VB, PHP, Java, PL/SQL and COBOL, which is intended to speed up the code review process by identifying bad/insecure code. |
| VisualCodeGrepper (VCG) | C/C++, CSharp, VB, PHP, Java, PL/SQL | VisualCodeGrepper (VCG) | Scans C/C++, C#, VB, PHP, Java, PL/SQL, and COBOL for security issues and for comments which may indicate defective code. The config files can be used to carry out additional checks for banned functions or functions which commonly cause security issues. |