Some of the benchmarks are single kernels/algorithms rather than applications.

[xorgy]

In some cases I think it makes sense to benchmark a given well-defined algorithm or kernel, but maybe it wouldn't hurt to make some of the benchmarks more realistic.

The nettle-based benchmarks are just running the selected cryptographic algorithms.

Maybe a more interesting benchmark in that realm would be a graft of WireGuard or some similar high performance network + cryptography workload. WireGuard I think is a good candidate since it is fairly small and manageable, and would introduce a very popular stream cipher, ChaCha20 (which is used as the main RNG on Linux, FreeBSD, OpenBSD, and NetBSD), and construction, Poly1305 (which is also one of a couple TLS 1.3 standard AEAD constructions). WireGuard has some cruft in it to link in kernel- and architecture-specific things though, so maybe a bit of work to separate that out and turn it into a self-running benchmark (emulating a connection in a self-contained program); but I think I could give it a shot if that seems like something that would be good.

[jeremybennett]

@xorgy Apologies. This got left behind. Thank you for raising this issue.

Because of the very tight constraints on this first version of Embench, we are inherently going to be using mostly just kernels. We are targeting IoT class devices, with a maximum 64kB for program and 64kB for data. This means that Embench is suitable for 8/16-bit devices as well as 32- and 64-bit devices. We have also kept I/O to a minimum (most IoT devices do not have "printf"), which means the benchmarks must include all their data. We also assume there is no operating system, since this is common for smaller devices.

In the future we plan versions of Embench which will support larger devices - application class embedded systems capable of running a multi-tasking operating system. Such a benchmark suite would certainly be amenable to full applications as you suggest.

The original set of benchmarks had a large selection of crypto kernels. However we wanted to ensure that we were not dominated by crypto, so we reduced to just two (one hash, one encryption) with an aspiration to add a third for eliptic curve DSA. We can't use Nettle ECDSA because it is too large.

We would rather see variation - currently both benchmarks are using the Nettle library. We have a plan that eventually we'll use different crypto libraries, with three benchmarks in total covering hashing, encryption/decryption and eliptic curve DSA. We'd like one from the Nettle library, one from the Wolfcrypt library and a third from the mbed library. The first priority is to ECDSA from either mbed or Wolfcrypt.

Would you be interested in contributing any of this? We rely on the community to put together the benchmark suite?

I'm going to mark this issue closed, since discussions like this are better held on the mailing list. You can join it via www.embench.org or directly at https://lists.librecores.org/listinfo/embench. I look forward to seeing your further comments there.

Best wishes,

Jeremy