Piggy

NB: This project is no longer supported. trpiggy is a new command-line tool for parse/ast tree manipulations, based on expressions using xpath.

Welcome to Piggy. This free and open source software is an AST pattern matcher and template engine for ASTs. It is a powerful source-to-source transformational system that was motivated by the lack of a good pinvoke generator for C#, and tools such as XPath, XSLT, and Antlr. It uses the same basic algorithm of DFS traversal of the AST commonly used, e.g., as in ClangSharp and CppSharp, but turns the visitor and listener patterns and code of those programs "inside-out" using templates.

In Piggy, a template is a combination of tree regular expressions, C# code and plain text blocks. With an AST and templates as input, a pattern matching engine performs matching of the the patterns in templates with the AST. After matching, output is generated by a DFS in-order tree walk, executing the code blocks and outputing the text blocks in the template. Pattern matching of trees follows the syntax of TreeRegEx, with extensions for node attributes (attr=value), and dynamic string interpolation of values during the match of attributes.

As with ClangSharp and CppSharp, Piggy inputs C++ files, parsed by Clang to get an abstract syntax tree (AST). Piggy reads a specification file that contains passes and templates. Output is generated by a DFS traversal of the AST, matching the tree with patterns. All C# code and text blocks in the templates is collected into a C# class, JIT compiled, then executed during a second traversal.

This tool does not read DLLs for P/Invoke generation, only the headers.

How to run the tool in CSPROJs/MSBuild/Visual Studio/Dotnet

Piggy is outfitted with build rules to run seamlessly within a C# project and build. To do that, first create your C# project, then add a reference to the NuGet Piggy package. Next, create a .PIG file using a copy of the example here, and also create a .CPP file containing the include file of the interface for the DLL. Make sure to set all the properties for the .PIG file (right-click, Properties). Then, "Rebuild".

How to run the tool directly

Note: Piggy only runs on Windows. I'm porting the code to Ubuntu.

Piggy is actually two tools: a wrapper for Clang to output serialized ASTs; a tool that reads ASTs and a .PIG file, which specifies the templates, and outputs code.

Both tools are Net Core apps. So, you can either build the tool and use dotnet or publish the apps to use an stand-alone executable.

Use ClangSerializer to output the AST for your code. To call the program, run _dotnet ClangSerializer/bin/bin/Release/netcoreapp2.2/publish/ClangSerializer.dll -c "..." -f "something.cpp" -o "ast,txt" _, where -c are the Clang options as one argument (use quotes and space delimiting each option, and don't use the minus sign, e.g., -c "Ic:/temp/include"); -f specifies the input file (e.g., -f "cuda-includes.cpp"). ClangSerializer outputs a parenthesized expression tree to standard out, which you should redirect to a file for the next step.
Use Piggy to convert the AST to code. To call the program, run dotnet Piggy/bin/bin/Release/netcoreapp2.2/publish/Piggy.dll -s xxx.pig -a xxx.ast -o xxx.pig.cs, where -s is an option to specify the input .pig file that contains the template rules; where -a is an option to specify the input AST; -o specifies the output C# file. The output of Piggy is assumed to be C# code, and formatted by the tool before outputting. It is also possible to use Piggy as a grep tool to find nodes in the AST. For example, Piggy -a xxx.ast -e "( EnumDecl )" will find all EnumDecl's in the tree.

I recommend you look at the CUDA example in the root directory of Piggy. It contains code for enums, typedefs, and functions for generating a basic CUDA interface. Please understand that Piggy requires a bit of work for you to generate a pinvoke interface. There is no magic here, and you will have to program a bit to get it to work the way you want. I eventually plan to expand the Enums/Structs/Funcs.pig files to have a large library for generating basic interfaces. I also plan to write .targets and .props files to have Piggy generate an interface during a build to use in your program.

Building Piggy

Note: Piggy only runs on Windows. I'm porting the code to Ubuntu.

Piggy requires a few things to build. Once built, a number of the dependencies are not required.

Java SE SDK (for building Piggy)
Antlr4 tool (for building Piggy)
LLVM source (for building Piggy)
Net Core 2 (for building and running Piggy)

Download llvm, clang, and clang extra.

Untar the downloads. Then,

 mkdir clang-llvm
 mv llvm-7.0.0.src clang-llvm/llvm               # rename directory to "llvm"
 mv cfe-7.0.0.src clang-llvm/llvm/tools/clang;   # move and rename directory to "clang" 
 mv clang-tools-extra-7.0.0.src clang-llvm/llvm/tools/clang/tools/extra  # #move and rename

Build using cmake (see instructions).

mkdir build; cd build
cmake -G "Visual Studio 15 2017" -A x64 -Thost=x64 ..\llvm
msbuild LLVM.sln /p:Configuration=Release /p:Platform=x64

Once you have built LLVM and Clang, you can build Piggy. Make sure to map e:/ to the location of clang-llvm/.

Piggy requires Antlr4. Please set up JAVA_HOME for the root of the Java installation, and Antlr4ToolPath to the path of the Antlr4 jar file. Alternatively, place in your CSPROJ file:

<PropertyGroup>
    <JAVA_HOME>C:\Program Files\Java\jdk-11.0.1</JAVA_HOME>
</PropertyGroup>

<PropertyGroup>
    <Antlr4ToolPath>C:\Program Files\Java\javalib\antlr-4.7.2-complete.jar</Antlr4ToolPath>
</PropertyGroup>

Background

Piggy extends the ideas of other pinvoke generators:

SWIG, the original pinvoke generator, which uses a specification file containing type maps.
ClangSharp (Mukul Sabharwal; mjsabby), and CppSharp, which use Clang AST visitors of the Clang-C API.
Lumír Kojecký's CodeProject article for dynamically compiling and executing C# code in the Net Framework.
Clang-query, which was used as a starting point for serializing an AST (the XML serializer no longer exists).
Jared Parsons' PInvoke Interop Assistant, which is another open-source pinvoke generator.
xInterop C++.Net Bridge, by Shawn Liu. Commercial product, no longer available.

Piggy Specification File

Instead of using and extending the unwieldy command-line arguments for input, Piggy uses a specification file. This file specifies teh templates for generating code.

For an example of the Piggy specification file, see the CUDA example in the root directory, starting with the Bash file "cuda-piggy.sh".

Spec file grammar

For the latest Antlr grammar files describing the input into Piggy, see SpecParser.g4 and SpecLexer.g4.

(Note: I highly recommend using my AntlrVSIX plugin for reading and editing Antlr grammars in Visual Studio 2017!)

Using Piggy as a grep

Piggy can be used as a grep to find nodes in the tree with the -e option. Any pattern used in a template should be possible. Here are some examples:

# Get all EnumDecls.
cat .generated_cuda_ast | cat .generated_cuda_ast | dotnet ./Piggy/bin/Debug/netcoreapp2.2/Piggy.dll -e "( EnumDecl )"

# Get all TypedefDecls.
cat .generated_cuda_ast | cat .generated_cuda_ast | dotnet ./Piggy/bin/Debug/netcoreapp2.2/Piggy.dll -e "( TypedefDecl )"

# Get all EnumDecls that start with CU for the name.
cat .generated_cuda_ast | cat .generated_cuda_ast | dotnet ./Piggy/bin/Debug/netcoreapp2.2/Piggy.dll -e '( EnumDecl Name="CU*" )'

# Get all EnumDecls that do not have a Name attribute.
cat .generated_cuda_ast | cat .generated_cuda_ast | dotnet ./Piggy/bin/Debug/netcoreapp2.2/Piggy.dll -e "( EnumDecl !Name )"

Relation to template engines

Piggy is similar to other engines, like CppSharp and ClangSharp, but whereas those engines use hardwired tree walking code to output pinvoke declarations, Piggy uses templates. Like JSP (1), which turned the concept of a servlet inside-out into a template which we now call HTML (2), Piggy turns the tree walking matcher "inside-out" for tree walking.

Piggy does not separate "model/view" as discussed by Parr (2). So, the logic of the translation to pinvoke declarations is interspersed with the AST pattern. In the future, a tree matching template could be formalized in order to refine the concept of a tree matching template. It's probably been done before, but I haven't had time to fully research the idea.

Further notes from my blog

A quick note on Piggy patterns and comparison with XPath

Piggy as a build tool

Refinements to Piggy

This little piggy is not at the market yet

Re-inventing the p/invoke generator

References

(1) JavaServer Pages Technology. https://www.oracle.com/technetwork/java/jsp-138432.html. Accessed Dec 20, 2018.

(2) Parr, Terence John. "Enforcing strict model-view separation in template engines." Proceedings of the 13th international conference on World Wide Web. ACM, 2004.

kaby76/Piggy