Lava

Lava is a project based on the Clang/LLVM toolset aiming to provide a unified C++ based graphics shading language. The ultimate goal is to have all shaders written in a single API-neutral and vendor-neutral language based on ISO-C++ (with certain restrictions due to the nature of the target environment).

It is in a very early state of development and more of an experiment (or proof-of-concept, if you like), worked on by one dude in his spare time. If everything goes according to plan it will be able to output SPIR-V, GLSL, ESSL, HLSL, MetalSL, and C++ headers with compatible data structures etc.

Did you say SPIR-V?

Indeed! Though it is still very early to have anything usable. All C++ control flow statements (except goto) are implemented and progress other stuff is steadily going forward. Getting this done properly and soonish has the highest priority for me.

Getting it

First follow the installation instructions at http://clang.llvm.org/get_started.html, (although I suggest using their Git mirrors instead of SVN) except use http://github.com/mknejp/lava instead of the official Clang repository, but don't run CMake just yet. Make sure all repositories are set to branch release_37 as that is what Lava is currently being developed against.

Checkout http://github.com/mknejp/glslang into the folder llvm/tools/clang/utils/glslang.

Switch the Lava repository to branch lava.

You are now ready to run CMake.

What can I do with it?

To play around build the lava target. Calling it without options prints out the help message. When running it over files expect crashes if the source contains stuff the code generators cannot handle yet. Restrict yourself to functions with basic expressions and control flow. You will need a function marked with vertex or fragment to see anything since the frontend currently only processes symbols reachable from shader entry points. Note that unlike with Clang the -o command specifies a directory to dump output files to since there can be multiple outputs per input, for example separate .vert.glsl and .frag.glsl files as GLSL doesn't support multistage shader sources.

Targets

You must specify a target to generate code for with the -target option. The help message lists all avilable targets. Currently glsl-300-es and spv-100 are available and actively worked on. spv-100 will by default generate binaries but you can additionally specify spv-disassemble-khr to generate a text file with the Khronos-style disassembly instead.

The targets are internally setup in a plugin-like architecture. Once finished users can provide their own target in form of dynamic library plugins in the same fashion Clang plugins work. There is no command line for loading them yet.

Code

All the code generating stuff is in include/Lava and lib/Lava which you can experiment with. I try to make all the pieces as re-usable as possible. There were also some minor adjustments to Clang's Parser, AST and Sema modules to support function entry point attributes, matrices and other minor tweaks. But those adjustments are very incomplete as the primary focus right now is code generation. That's the reason I decided to put everything into a Clang fork instead of separating it. Many parts of the GLSL backend will be re-usable in the other generators since their languages all revolve around a highly similar C-based syntax.

Silly example

In order to see any output you currently require an entry point that calls your example code. Here is what a very silly example would generate from the following input:

vertex int fib(int x)
{
  if(x <= 0)
    return 0;

  auto fn_2 = 1;
  auto fn_1 = 1;
  auto fib = 1;
  int temp;
  for(auto i = 2; i <= x; ++i)
  {
    temp = fib;
    fib = fn_2 + fn_1;
    fn_2 = fn_1;
    fn_1 = temp;
  }
  return fib;
}

This currently produces the following two outputs

// Module Version 99
// Generated by (magic number): bb
// Id's are bound by 35

                              MemoryModel Logical GLSL450
                              Name 4  "_Z3fibi"
                              Name 3  "x"
                              Name 14  "fn_2"
                              Name 15  "fn_1"
                              Name 16  "fib"
                              Name 17  "temp"
                              Name 19  "i"
               1:             TypeInt 32 1
               2:             TypeFunction 1(int) 1(int)
               6:      1(int) Constant 0
               7:             TypeBool
              11:             TypeVoid
              13:      1(int) Constant 1
              18:      1(int) Constant 2
      4(_Z3fibi):      1(int) Function None 2
            3(x):      1(int) FunctionParameter
               5:             Label
               8:     7(bool) SLessThanEqual 3(x) 6
                              SelectionMerge 10 None
                              BranchConditional 8 9 10
               9:               Label
                                ReturnValue 6
              10:             Label
        14(fn_2):      1(int) CopyObject 13
        15(fn_1):      1(int) CopyObject 13
         16(fib):      1(int) CopyObject 13
        17(temp):      1(int) Undef
           19(i):      1(int) CopyObject 18
                              Branch 20
              20:             Label
              33:      1(int) Phi 19(i) 10 28 22
              32:      1(int) Phi 17(temp) 10 24 22
              31:      1(int) Phi 16(fib) 10 25 22
              30:      1(int) Phi 15(fn_1) 10 27 22
              29:      1(int) Phi 14(fn_2) 10 26 22
              23:     7(bool) SLessThanEqual 33 3(x)
                              LoopMerge 21 None
                              BranchConditional 23 22 21
              22:               Label
              24:      1(int)   CopyObject 31
              25:      1(int)   IAdd 29 30
              26:      1(int)   CopyObject 30
              27:      1(int)   CopyObject 24
              28:      1(int)   IAdd 33 13
                                Branch 20
              21:             Label
                              ReturnValue 31
                              FunctionEnd

// int fib(int x)
int _Z3fibi(int x);

// int fib(int x)
int _Z3fibi(int x)
{
  if(x <= 0)
  {
    return 0;
  }
  int fn_2 = 1;
  int fn_1 = 1;
  int fib = 1;
  int temp;
  for(int i = 2; i <= x; ++i)
  {
    temp = fib;
    fib = fn_2 + fn_1;
    fn_2 = fn_1;
    fn_1 = temp;
  }
  return fib;
}

As you can see the GLSL generator tries to preserve the original code structure and statements as closely as possible to make transitioning between the two as straightforward as possible. It would probably make the implementation easier to always generate SPIR-V and then disassemble that into GLSL/HLSL/etc, however imagine you are on a platform that only has debugging support for OpenGL. With the SPIR-V intermediate step you would lose a lot of information about the code, making the generated sources very cryptic, and backporting the fixes from your GLSL debugging session to the original Lava code would be very difficult. Preserving the original code as closely as possible makes this task much easier.

The Language

Lava is based on ISO-C++14 with some restrictions applied that aren't really suitable for a GPU stream processor (exceptions, RTTI, virtual, ...). There are also some limitations that are dictated by the capabilities of the supported target languages (no pointers, reference captures, etc). This all still needs fleshing out as the code generators mature.

This code defines a few vector/matrix types

template<class T, int N>
using vec = T __attribute__((ext_vector_type(N)));

using vec2 = vec<float, 2>;
using vec3 = vec<float, 3>;
using vec4 = vec<float, 4>;

template<class T, int Rows, int Cols>
using mat = T __attribute__((matrix_type(Rows, Cols)));

using mat2x2 = mat<float, 2, 2>;
using mat2 = mat2x2;
using mat3x3 = mat<float, 3, 3>;
using mat3 = mat3x3;
using mat4x4 = mat<float, 4, 4>;
using mat4 = mat4x4;

Don't worry, those ugly attributes are something that can be hidden in a standard library header. And yes, it's row-major matrix notation (not to be confused with layout!) because that is what every damn mathematics textbook in the world is teaching students. And therefore the following

auto m = mat3{1, 2, 3,
              4, 5, 6,
              7, 8, 9}

does exactly what you expect it to! No need to transpose the thing in your head. Let the rest be a problem for the compiler, not the user. There is currently nothing you can do with matrices, they are just there to crete AST nodes so I have input to test code generation with.

To declare an entry point:

int foo() { return 1; }

vertex void transform() { foo(); }
fragment void shade() { foo(); }

Remember that the frontend currently filters out all symbols that are not reachable from entry points. This is intended to be an option in the future.

What next?

Probably a lot of patience (unless you wish to participate), since as already mentioned I am doing this alone in my spare time and this is a very ambitious project. I hope you like what you see so far. Let me know what you think.

Why "Lava"?

Because with Mantle, Vulkan and Metal it seems like geology is a new synonym for computer graphics. And for me lava is the first thing that comes to mind when thinking about volcanos.

Status

This overview represents the capabilities of the frontend and code genrators and is held as up-to-date as my memory and attention span allow.

C++ frontend status

These are the changes planned for the frontend to support all the stuff for graphics programming. I have investigated the Clang code some of them but most are here only for reference.

Code Generation Status

This is a matrix of how far the code generators are and what work is still outstanding. Legend:

❌ Not implemented and not even started work on it yet
✅ Should be feature complete
⚠️ Partially done but not complete yet
💥 Even though code can be generated it is a feature that is dependend on the backend capabilities (like GLSL vs ESSL) or requires an extension to work. This needs addressing once feature-selection is a thing.

Builtin types	SPIR-V	GLSL	HLSL	MetalSL
bool	✅	✅	❌	❌
(u)int16	💥	💥	❌	❌
(u)int32	✅	✅	❌	❌
(u)int64	💥	💥	❌	❌
float16	💥	💥	❌	❌
float32	✅	✅	❌	❌
float64	💥	💥	❌	❌
vector	✅	✅	❌	❌
matrix	✅	✅	❌	❌
arrays	✅	✅	❌	❌

Record types	SPIR-V	GLSL	HLSL	MetalSL
scalar members	✅	✅	❌	❌
array members	✅	✅	❌	❌
record members	✅	✅	❌	❌
base classes	✅	✅	❌	❌
static members	❌	❌	❌	❌
lambda closures	✅	✅	❌	❌

Member functions	SPIR-V	GLSL	HLSL	MetalSL
constructors	❌	❌	❌	❌
destructors	❌	❌	❌	❌
overloaded operators	❌	❌	❌	❌
non-static methods	❌	❌	❌	❌
static methods	❌	❌	❌	❌
lambda invoker	❌	❌	❌	❌

Functions	SPIR-V	GLSL	HLSL	MetalSL
by-value params	✅	✅	❌	❌
by-ref params	⚠️ (no write on return)	✅	❌	❌
by-const-ref params	✅	✅	❌	❌

Statements	SPIR-V	GLSL	HLSL	MetalSL
`break`	✅	✅	❌	❌
`continue`	✅	✅	❌	❌
`discard`	❌	❌	❌	❌
`do`	✅	✅	❌	❌
`for`	✅	✅	❌	❌
`if`/`else`	✅	✅	❌	❌
`return`	✅	✅	❌	❌
`switch`	✅	✅	❌	❌
`while`	✅	✅	❌	❌
initalized local	✅	✅	❌	❌
uninitalized local	✅	✅	❌	❌

Expressions	SPIR-V	GLSL	HLSL	MetalSL
unary operator scalars	✅	✅	❌	❌
unary operator vectors	❌	❌	❌	❌
binary operator scalars	✅	✅	❌	❌
binary operator vectors	❌	❌	❌	❌
binary operator matrices	❌	❌	❌	❌
conditional operator	❌	❌	❌	❌
variable reference	✅	✅	❌	❌
member access	❌	❌	❌	❌
array subscript	❌	❌	❌	❌
swizzle	❌	❌	❌	❌
calls	❌	❌	❌	❌
atomics	❌	❌	❌	❌
casts (see below)	❌	❌	❌	❌

Casts	SPIR-V	GLSL	HLSL	MetalSL
lvalue-to-rvalue	✅	✅	❌	❌
...	❌	❌	❌	❌

Other	SPIR-V	GLSL	HLSL	MetalSL
destructor calls for locals	❌	❌	❌	❌
entry points / main function	❌	❌	❌	❌
shader input / output	❌	❌	❌	❌
export / import functions	❌	❌	❌	❌
export / import variables	❌	❌	❌	❌
static initialization	❌	❌	❌	❌
backend versions	❌	❌	❌	❌
extensions	❌	❌	❌	❌
...	❌	❌	❌	❌