Are the paper's benchmarks in this repo?

[Chillee]

^

[jiazhihao]

The benchmarks are available in cpp_examples. We are working on the Python version of the benchmarks.

[Chillee]

Do the python version of the benchmarks include how the baselines are being run?

[Chillee]

Also, is there example generated code for any of the kernels somewhere?

[jiazhihao]

@Chillee Mirage's generated code can be executed using two backends: (1) a native cutlass-based backend, and (2) a Triton backend. While some of the evaluation results are measured using the cutlass backend (due to some operators not supported by Triton), I think the Triton backend is easier to try. If you are looking for generated code, you can try the docker image and run python demo/demo_group_query_attention_spec_decode.py --checkpoint demo/checkpoint_group_query_attn_spec_decode.json, which will generate all Triton programs discovered by Mirage for group query attention.

[Chillee]

Imo it would help understanding of the paper significantly if you include some of the generated triton programs in the github repo. The paper reports that each benchmark seems to take up to 6 hours, so it's a nontrivial task to run it just to see what kind of kernels it's generating :)

[jiazhihao]

Thanks for the suggesting! We will consider including some Triton examples in the repo to help users understand Mirage and know what to expect from Mirage. One challenge we need to address is that the search procedure itself may be non-deterministic --- users may get different Triton programs from each search procedure. Thanks again for the suggestion.