A runner for benchmarking compilation speeds across several Scala projects on MacBook Pros I have available at home.
Disclaimer: This is very much non-scientific and I did all this in a weekend afternoon, don't expect everything to be well-executed and accurate :) it's also the first time I prepared anything like this.
nix develop
scala-cli package .
- shutdown the
bloop.Bloop
process (check the PID withjps
) ROUNDS=1 ./app results.csv
(to fully replicate, use ROUNDS=5)
- The M1 MAX outperforms the i9 by a factor of 1.6-2.2, depending on the project.
- The M1 MAX was also much less noisy, although I don't have exact numbers and they were all in one room :) it definitely didn't spin its fans enough to be heard through noise-cancelling headphones, contrary to the Intels.
- The 16" i9's performance seems embarassing compared to the much cheaper 15" i7 model.
MacBook Pro (14-inch, 2021) Chip: Apple M1 MAX Memory: 64 GB
MacBook Pro (16-inch, 2019) Processor: 2.4 GHz 8-Core Intel Core i9 Memory: 64 GB 2667 MHz DDR4
MacBook Pro (15-inch, 2018) Processor: 2,2 GHz 6-Core Intel Core i7 Memory: 32 GB 2400 MHz DDR4
- Projects were chosen in an arbitrary fashion.
- I was aiming to use both libraries and applications, as well as more or less complex builds.
- I didn't want to deal with other build tools so everything builds in sbt.
- The same JVM flags were applied to each compilation: whatever was configured in sbt's defaults,
.sbtopts
or an explicit override (zio-aws got over 16G of memory in total) was used consistently on each machine. - The same version of each JVM release was used between machines, however they're native to each platform (aarch64, x86).
- All dependencies (sbt, JDKs, node/yarn for libraries that cross-compile to Scala.JS) were sourced from a Nix flake (attached in this repository).
- Each project was first "warmed up", which meant executing the compile command, which made sure all dependencies were available for the measured compilations.
- All projects were compiled using sbt (version chosen by each project separately).
- Each project was compiled 5 times one after another, with an
sbt clean
execution before each compile. - All compilations ran sequentially within each machine.
- Every laptop was charging throughout the process.
- The time measured was the total time of executing
sbt
with thecompile
command (on a cold JVM)... - ...although for certain projects there was extra (sub-second) overhead of changing Java versions (entering the Nix shell): these were "work project" (jdk 11) and akka (jdk 8).
- The "work project" was compiled in
IntegrationTest
andTest
configurations (Compile
is implied).
# Java 17
openjdk version "17.0.1" 2021-10-19 LTS
OpenJDK Runtime Environment Zulu17.30+15-CA (build 17.0.1+12-LTS)
OpenJDK 64-Bit Server VM Zulu17.30+15-CA (build 17.0.1+12-LTS, mixed mode, sharing)
# Java 11
openjdk version "11.0.11" 2021-04-20 LTS
OpenJDK Runtime Environment Zulu11.48+21-CA (build 11.0.11+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.48+21-CA (build 11.0.11+9-LTS, mixed mode)
# Java 8 on ARM
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (Zulu 8.54.0.21-CA-macos-aarch64) (build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (Zulu 8.54.0.21-CA-macos-aarch64) (build 25.292-b10, mixed mode)
# Java 8 on Intel
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (Zulu 8.54.0.21-CA-macosx) (build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (Zulu 8.54.0.21-CA-macosx) (build 25.292-b10, mixed mode)
All times are in seconds (floored) - I didn't need more precision.
project | round 1 | round 2 | round 3 | round 4 | round 5 |
---|---|---|---|---|---|
akka | 94 | 94 | 93 | 94 | 96 |
scalatest | 35 | 37 | 36 | 36 | 36 |
cats | 64 | 62 | 60 | 60 | 60 |
cats-effect | 25 | 24 | 25 | 25 | 24 |
doobie | 24 | 24 | 25 | 23 | 24 |
natchez | 10 | 10 | 10 | 10 | 10 |
skunk | 14 | 14 | 14 | 14 | 14 |
fs2 | 31 | 31 | 31 | 31 | 30 |
http4s | 40 | 40 | 39 | 40 | 41 |
metals | 35 | 35 | 34 | 35 | 34 |
steve | 9 | 9 | 9 | 9 | 9 |
scala-steward | 17 | 17 | 17 | 16 | 17 |
smithy4s | 62 | 61 | 61 | 62 | 60 |
weaver-test | 73 | 69 | 74 | 71 | 72 |
trading | 20 | 20 | 20 | 20 | 20 |
zio | 39 | 39 | 40 | 40 | 40 |
work project | 75 | 70 | 69 | 70 | 70 |
scala | 55 | 55 | 54 | 55 | 55 |
dotty | 41 | 43 | 41 | 41 | 41 |
fs2-aws | 12 | 12 | 12 | 12 | 13 |
zio-aws ec2 | 107 | 102 | 102 | 102 | 107 |
project | round 1 | round 2 | round 3 | round 4 | round 5 |
---|---|---|---|---|---|
akka | 149 | 150 | 147 | 147 | 147 |
scalatest | 67 | 67 | 67 | 66 | 66 |
cats | 127 | 129 | 123 | 129 | 133 |
cats-effect | 51 | 55 | 55 | 54 | 51 |
doobie | 50 | 46 | 46 | 46 | 45 |
natchez | 18 | 18 | 18 | 19 | 18 |
skunk | 28 | 31 | 30 | 30 | 30 |
fs2 | 64 | 65 | 63 | 63 | 63 |
http4s | 85 | 77 | 76 | 83 | 78 |
metals | 64 | 63 | 64 | 63 | 62 |
steve | 19 | 18 | 19 | 18 | 19 |
scala-steward | 31 | 31 | 32 | 31 | 31 |
smithy4s | 131 | 125 | 126 | 123 | 126 |
weaver-test | 137 | 142 | 137 | 137 | 140 |
trading | 41 | 41 | 43 | 42 | 42 |
zio | 84 | 86 | 86 | 86 | 86 |
work project | 129 | 124 | 124 | 120 | 122 |
scala | 98 | 96 | 97 | 98 | 95 |
dotty | 73 | 72 | 72 | 73 | 71 |
fs2-aws | 22 | 21 | 21 | 23 | 23 |
zio-aws ec2 | 161 | 158 | 153 | 152 | 155 |
project | round 1 | round 2 | round 3 | round 4 | round 5 |
---|---|---|---|---|---|
akka | 168 | 168 | 168 | 168 | 167 |
scalatest | 75 | 75 | 74 | 75 | 77 |
cats | 133 | 127 | 126 | 127 | 127 |
cats-effect | 50 | 50 | 49 | 49 | 50 |
doobie | 49 | 51 | 51 | 51 | 51 |
natchez | 19 | 19 | 19 | 19 | 19 |
skunk | 29 | 29 | 29 | 28 | 28 |
fs2 | 72 | 67 | 66 | 66 | 66 |
http4s | 85 | 88 | 85 | 87 | 89 |
metals | 68 | 67 | 67 | 69 | 67 |
steve | 18 | 19 | 18 | 18 | 18 |
scala-steward | 34 | 34 | 34 | 34 | 35 |
smithy4s | 138 | 138 | 138 | 137 | 138 |
weaver-test | 153 | 144 | 146 | 144 | 147 |
trading | 42 | 41 | 41 | 41 | 40 |
zio | 87 | 85 | 86 | 86 | 87 |
work project | 139 | 136 | 133 | 135 | 136 |
scala | 113 | 114 | 114 | 113 | 112 |
dotty | 87 | 86 | 86 | 87 | 86 |
fs2-aws | 24 | 24 | 24 | 24 | 23 |
zio-aws ec2 | 179 | 171 | 170 | 174 | 164 |
project | 14" M1 MAX 10-core | i9 | i9/m1 ratio | i7 | i7/m1 ratio |
---|---|---|---|---|---|
akka | 94 | 147 | 1.6 | 168 | 1.8 |
scalatest | 36 | 67 | 1.9 | 75 | 2.1 |
cats | 60 | 129 | 2.2 | 127 | 2.1 |
cats-effect | 25 | 54 | 2.2 | 50 | 2.0 |
doobie | 24 | 46 | 1.9 | 51 | 2.1 |
natchez | 10 | 18 | 1.8 | 19 | 1.9 |
skunk | 14 | 30 | 2.1 | 29 | 2.1 |
fs2 | 31 | 63 | 2.0 | 66 | 2.1 |
http4s | 40 | 78 | 2.0 | 87 | 2.2 |
metals | 35 | 63 | 1.8 | 67 | 1.9 |
steve | 9 | 19 | 2.1 | 18 | 2.0 |
scala-steward | 17 | 31 | 1.8 | 34 | 2.0 |
smithy4s | 61 | 126 | 2.1 | 138 | 2.3 |
weaver-test | 72 | 137 | 1.9 | 146 | 2.0 |
trading | 20 | 42 | 2.1 | 41 | 2.1 |
zio | 40 | 86 | 2.2 | 86 | 2.2 |
work-project | 70 | 124 | 1.8 | 136 | 1.9 |
scala | 55 | 97 | 1.8 | 113 | 2.1 |
dotty | 41 | 72 | 1.8 | 86 | 2.1 |
fs2-aws | 12 | 22 | 1.8 | 24 | 2.0 |
zio-aws ec2 | 102 | 155 | 1.5 | 171 | 1.7 |