filecheck - A Python-native clone of LLVMs FileCheck tool

Note: This project is the successor of mull-project/FileCheck.py.

This tries to be as close a clone of LLVMs FileCheck as possible, without going crazy. It currently passes 1576 out of 1645 (95.8%) of LLVMs MLIR filecheck tests. We are tracking all 69 remaining test failures in GitHub issues.

There are some features that are left out for now (e.g.a pseudo-numeric variables and parts of numeric substitution ).

The codebase is fully type checked by pyright, and automatically formatted using black. We aim to have tests covering everything from normal matching down to error messages.

Install by running pip install filecheck.

Features:

Here's an overview of all FileCheck features and their implementation status.

Checks:
- CHECK
- CHECK-NEXT
- CHECK-NOT (Bug: #10)
- CHECK-LABEL (Bug: #8)
- CHECK-EMPTY
- CHECK-SAME
- CHECK-DAG
- CHECK-COUNT
Flags:
- --check-prefix
- --check-prefixes
- --comment-prefixes
- --allow-unused-prefixes
- --input-file
- --match-full-lines
- --strict-whitespace (Bug: #6)
- --ignore-case
- --implicit-check-not (Tracked: #20)
- --dump-input (only fail and never supported)
- --dump-input-context
- --dump-input-filter
- --enable-var-scope
- -D<VAR=VALUE>
- -D#<FMT>,<NUMVAR>=<NUMERIC EXPRESSION>
- -version
- -v
- -vv
- --allow-deprecated-dag-overlap
- --allow-empty
- --color Colored output is supported and automatically detected. No support for the flag.
Base Features:
- Regex patterns
- Captures and Capture Matches (Bug: #11)
- Numeric Captures
- Numeric Substitutions (jesus christ, wtf man) (Tracked: #21)
- Literal matching (CHECK{LITERAL})
- Weird regex features ([:xdigits:] and friends)
- Correct(?) handling of matching check lines (Bug: #22)
Testing:
- Base cases
- Negative tests
- Error messages (started)
- Lots of edge cases
- MLIR/xDSL integration tests
UX:
- Good error messages: Error messages are on an okay level, not great, but not terrible either.
  - Parse errors
  - Matching errors
  - Print possible intended matches (could be better still)
  - Malformed regexes
  - Wrong/unkown command line arguments
  - Print variables and their origin in error messages
Infrastructure:
- Formatting: black
- Pyright
- pre-commit
- CI for everything

We are open to PRs for bugfixes or any features listed here.

Differences to LLVMs FileCheck:

We want to be as close as possible to the original FileCheck, and document our differences very clearly.

If you encounter a difference that is not documented here, feel free to file a bug report.

Better Regexes:

We use pythons regex library, which is a flavour of a Perl compatible regular expression (PCRE), instead of FileChecks POSIX regex flavour.

Example:

// LLVM filecheck:
// CHECK: %{{[[:alnum:]]+}}, %{{[[:digit:]]+}}

// our fileheck:
// CHECK: %{{[a-zA-Z0-9]+}}, %{{\d+}}

Some effort is made to translate character classes from POSIX to PCRE, although it might be wrong in edge cases.

Relaxed Matching:

We relax some of the matching rules, like:

Allow a file to start with CHECK-NEXT

No Numerical Substitution:

This is used in 2 out of 1645 tests in our benchmark (upstream MLIR tests).

While our filecheck supports numeric capture ([[#%.3x,VAR:]] will capture a three-digit hex number), we don't support arithmetic expressions on these captured values at the moment. We also don't support the "Pseudo Numeric Variable" @LINE.

Special Feature Flags:

This version of filecheck implements some non-standard extensions to LLVMs filecheck. These are disabled by default but can be enabled through the environment variable FILECHECK_FEATURE_ENABLE=.... Avialable extensions are documented here:

MLIR_REGEX_CLS: Add additional special regex matchers to match MLIR/LLVM constructs:
- \V will match any SSA value name

Reject Empty Captures:

We introduce a new flag called reject-empty-vars that throws an error when a capture expression captures an empty string.

AntonLydike/filecheck