awesome-ci

⚠️ DEPRECATION WARNING

This repository is deprecated. Please use the following docker images below instead:

All #awesome-ci Docker images

ansible-lint • ansible • awesome-ci • bandit • black • checkmake • eslint • file-lint • gofmt • goimports • golint • jsonlint • kubeval • linkcheck • mypy • php-cs-fixer • phpcbf • phpcs • phplint • pycodestyle • pydocstyle • pylint • terraform-docs • terragrunt-fmt • terragrunt • yamlfmt • yamllint

Docker images

Save yourself from installing lot's of dependencies and pick a dockerized version of your favourite linter below for reproducible local or remote CI tests:

GitHub DockerHub Type Description

awesome-ci Basic Tools for git, file and static source code analysis

file-lint Basic Baisc source code analysis

linkcheck Basic Search for URLs in files and validate their HTTP status code

ansible Ansible Multiple versions and flavours of Ansible

ansible-lint Ansible Lint Ansible

gofmt Go Format Go source code ^[1]

goimports Go Format Go source code ^[1]

golint Go Lint Go code

eslint Javascript Lint Javascript code

jsonlint JSON Lint JSON files ^[1]

kubeval K8s Lint Kubernetes files

checkmake Make Lint Makefiles

phpcbf PHP PHP Code Beautifier and Fixer

phpcs PHP PHP Code Sniffer

phplint PHP PHP Code Linter ^[1]

php-cs-fixer PHP PHP Coding Standards Fixer

bandit Python A security linter from PyCQA

black Python The uncompromising Python code formatter

mypy Python Static source code analysis

pycodestyle Python Python style guide checker

pydocstyle Python Python docstyle checker

pylint Python Python source code, bug and quality checker

terraform-docs Terraform Terraform doc generator (TF 0.12 ready) ^[1]

terragrunt Terraform Terragrunt and Terraform

terragrunt-fmt Terraform terraform fmt for Terragrunt files ^[1]

yamlfmt Yaml Format Yaml files ^[1]

yamllint Yaml Lint Yaml files

GitHub	Type	Description
awesome-ci	Basic	Tools for git, file and static source code analysis
file-lint	Basic	Baisc source code analysis
linkcheck	Basic	Search for URLs in files and validate their HTTP status code
ansible	Ansible	Multiple versions and flavours of Ansible
ansible-lint	Ansible	Lint Ansible
gofmt	Go	Format Go source code ^[1]
goimports	Go	Format Go source code ^[1]
golint	Go	Lint Go code
eslint	Javascript	Lint Javascript code
jsonlint	JSON	Lint JSON files ^[1]
kubeval	K8s	Lint Kubernetes files
checkmake	Make	Lint Makefiles
phpcbf	PHP	PHP Code Beautifier and Fixer
phpcs	PHP	PHP Code Sniffer
phplint	PHP	PHP Code Linter ^[1]
php-cs-fixer	PHP	PHP Coding Standards Fixer
bandit	Python	A security linter from PyCQA
black	Python	The uncompromising Python code formatter
mypy	Python	Static source code analysis
pycodestyle	Python	Python style guide checker
pydocstyle	Python	Python docstyle checker
pylint	Python	Python source code, bug and quality checker
terraform-docs	Terraform	Terraform doc generator (TF 0.12 ready) ^[1]
terragrunt	Terraform	Terragrunt and Terraform
terragrunt-fmt	Terraform	`terraform fmt` for Terragrunt files ^[1]
yamlfmt	Yaml	Format Yaml files ^[1]
yamllint	Yaml	Lint Yaml files

^[1] Uses a shell wrapper to add enhanced functionality not available by original project.

Runs on

Continuous Integration command line tools for git repositories, file characteristics, syntax errors and static source code analysis.

Awesome-CI is capable of finding various problems in your code repository as well as fixing them automatically.

Table of Contents

Tools
Learn / Validate
Fix
Custom regex contributions
General usage
Installation
Awesome CI Docker image
Documentation
License

Tools

All checks have the option to only check by one or more file extensions, by shebang as well as to exclude one or more folders from the whole search.

Note: Fixable options are currently in testing phase. Please report any bugs.

Type	Tool	Fixable	Description
Git	git-conflicts		Scan files and check if they contain git conflicts.
Git	git-ignored		Scan git directory and see if ignored files are still in git cache.
File	file-cr	✓	Scan files and check if they contain CR (Carriage Return only).
File	file-crlf	✓	Scan files and check if they contain CRLF (Windows Line Feeds).
File	file-empty		Scan files and check if they are empty (0 bytes).
File	file-nullbyte-char	✓	Scan files and check if they contain a null-byte character (\x00).
File	file-trailing-newline	✓	Scan files and check if they contain a trailing newline.
File	file-trailing-single-newline	✓	Scan files and check if they contain exactly one trailing newline.
File	file-trailing-space	✓	Scan files and check if they contain trailing whitespaces.
File	file-utf8	✓	Scan files and check if they have a non UTF-8 encoding.
File	file-utf8-bom	✓	Scan files and check if they contain BOM (Byte Order Mark): `<U+FEFF>`.
Syntax	syntax-bash		Scan shell files for bash syntax errors.
Syntax	syntax-css		Scan CSS files for CSS syntax errors.
Syntax	syntax-js		Scan JS files for JS syntax errors.
Syntax	syntax-json		Scan files for JSON syntax errors.
Syntax	syntax-markdown		Scan files for Markdown syntax errors.
Syntax	syntax-perl		Scan Perl files for Perl syntax errors.
Syntax	syntax-php		Scan files for PHP syntax errors.
Syntax	syntax-python		Scan Python files for Python syntax errors.
Syntax	syntax-ruby		Scan Ruby files for Ruby syntax errors.
Syntax	syntax-scss		Scan SCSS files for SCSS syntax errors.
Syntax	syntax-sh		Scan shell files for /bin/sh syntax errors.
Code Conventions	inline-css		Scan files and check if they contain inline css code.
Code Conventions	inline-js		Scan files and check if they contain inline javascript code.
Regex	regex-grep		`egrep` (`grep -E`) regex version to scan files for an occurance.
Regex	regex-perl		`perl` regex version to scan files for an occurance.

Learn / validate

All of the above scripts offer the --dry option which will only show you the built command without actually executing it:

$ regex-grep --path=. --ignore=".git,.svn" --shebang=sh --size --text \
--custom="if[[:space:]]*\[\[" --dry

find . -type f -not \( -path "./.git*" -o -path "./.svn*" \) ! -size 0 -print0 | \
   xargs -0 -P 8 -n1 grep -Il '' | \
   tr '\n' '\0' | \
   xargs -0 -P 8 -n1 awk '/^#!.*(\/sh|[[:space:]]+sh)/{print FILENAME}' | \
   tr '\n' '\0' | \
   xargs -0 -P 8 -n1  sh -c 'if [ -f "${1}" ]; then grep --color=always -inHE "if[[:space:]]*\[\[" "$1" || true; fi' --

Fix

Some of the above scripts offer the --fix option (see table above), with which you are actually able to fix the problem. You can also combine it with --dry to see how the actual fix command looks like:

$ file-utf8 --path=dump.sql --fix --dry

find dump.sql -type f -print0 | \
   xargs -0 -P 8 -n1  sh -c 'if [ -f "${1}" ]; then isutf8 "$1" >/dev/null || (TERM=vt100 vi -u NONE -n -es -c "set fileencoding=utf8" -c "wq" "$1" > /dev/tty && echo "Fixing: $1" || echo "FAILED: $1"); fi' --

Custom regex contributions

regex-grep and regex-perl have a lot of potential for doing custom project validation.

In order to give you an idea, have a look at the compiled Regex Contributions.

Please use pull requests to add useful checks.

General Usage

All tools share the same pattern (except git-ignored) and can be used with the same command line arguments.
Some tools have an additional option --custom="" to overwrite the command itself (this is explained and shown in detail in each command's --help option).

Options:

# Required:
  --path            # Specify the path where to scan

# Optional pattern (each option is logically and-ed):
  --fix             # Fix the problems for the specified files (not every check)

  --text            # Only scan non-binary files
  --size            # Only scan non-empty files (greater 0 bytes)
  --shebang         # Only scan files (shell scripts) that match a certain shebang
  --extension       # Only scan files with these file extensions
  --ignore          # Ignore files/folders

# Optional misc:
  --config          # Specify configuration file
  --confpre         # Alter configuration directive prefix for this check
  --verbose         # Show files and commands as being processed
  --debug           # Show additional debug messages
  --list            # Only show files that would be processed (no processing)
  --dry             # Show command that would be executed (no processing)

# System
  --help            # Show help
  --info            # Show version of required binaries
  --version         # Show tool version

Installation

Requirements

Awesome-ci requires the following tools to be installed:

dos2unix
eslint
file
git
jsonlint
mdl
perl
php
python
ruby
scss_lint
shellcheck

Install OSX

brew tap cytopia/tap
brew install awesome-ci

Install Linux/BSD

# Install to /usr/bin
./configure
make install

# Instal to /usr/local/bin
./configure --prefix=/usr/local
make install

# Install to /opt/bin
./configure --prefix=/opt
make install

Awesome CI Docker image

Instead of installing awesome-ci and all its required dependencies locally on your computer, you can also use the bundled Docker image cytopia/awesome-ci which has everything pre-installed and is built nightly by travis-ci.

docker run -v ${PWD}:/ac cytopia/awesome-ci file-crlf --path=/ac

The above example is using file-crlf to scan the current directory for files containing Windows newlines:

${PWD} (the current host directory) is mounted into the container's /ac directoy
file-crlf path then points (inside the container) to /ac (which is the current host directory)
/ac can actually be named by whatever name you want

If you use an awesome-ci configuration which is not inside the directory you want to check, you will also have to mount that into the container:

docker run \
    -v /host/path/to/awesome-ci.conf:/etc/awesome-ci.conf \
    -v ${PWD}:/ac cytopia/awesome-ci file-crlf --path=/ac --config=/etc/awesome-ci.conf

Documentation

To find out more about awesome-ci, have a look at the following links.

License

MIT License