Proposal: OCaml language and package builds
Closed this issue · 13 comments
Name, email, company, job title
Name: Anil Madhavapeddy
Email: anil@recoil.org
Company: University of Cambridge
Job: Faculty
Project Title and description
OCaml Build Infrastructure: aarch64 support
OCaml is a general purpose programming language with an emphasis on expressiveness and safety.
Examples of large scale systems implemented in OCaml include Facebook's Hack language, the Infer static analysis tool, the Flow JavaScript inference, the Coq proof assistant, the Compcert certified C compiler, the ReasonML toolchain, and the MirageOS operating system.
OCaml has full support for fast, native code compilation to ARM 32/64. We would like to add in regular ARM CI for both the compiler and the opam package manager. This involves around ~8000 packages being built on a matrix of 8 different compiler versions and variants, all of which are sandboxed in Docker containers.
We would also like to add testing for FreeBSD and OpenBSD on aarch64, but this depends on the respective operating systems booting on Packet.net machines.
It would also be helpful to be able to have access to machines that are capable of 32-bit armhf builds, so that we can generate binary packages for the Raspberry Pi and similar devices.
Which members of the Works on Arm community and/or end-users would benefit from your work?
End users would gain immediate feedback about their submitted OCaml packages working on aarch64 alongside the x86_64 tests. We would also get prompter feedback about aarch64 specific issues (e.g. broken backtraces) due to increased coverage and fuzz testing of the compiler itself.
Is the code that you’re going to run 100% open source? If so, what is the URL or URLs where it is located?
It is all 100% open source:
What infrastructure (computing resources and network access) do you need? (see: https://www.packet.net/bare-metal/)?
- For bulk builds and CI a type 2A machine
- Smaller 32/64 capable machine can also be rotated into the pool
- Sufficient bandwidth to get the logs to the central OCaml build cluster (hosted at the University of Cambridge Computer Laboratory)
Please state your contributions to the open source community and any other relevant initiatives
My personal GitHub at @avsm lists most of the projects I'm involved with.
A note that https://www.worksonarm.com/explore/ocaml should be updated by me with the current state of the project (it's 404 at this moment).
Having ocaml CI sounds incredibly useful!
Where would CI results be published?
Would this help catch and address issues such as:
facebook/hhvm@3c49d4e
@zlim - The hhvm issue was a limits issue. Facebook is a couple of releases back in OCaml,
so CI on the current release would not have caught this issue.
@zlim -- we're working on a big update for https://opam.ocaml.org that will include bulk build, ongoing CI and multiarch. The results of the builds from these machines will be published there. Webpages will be going up soon, but for now the slides of the recent OCaml Workshop 2017 are available: https://speakerdeck.com/avsm/ocaml-platform-2017
I chatted with @avsm today and we're refining the list of needs for this proposal and identifying the variety of organizations that have a stake in this infrastructure. Progress!
This is now coming live and building multiarch images on https://hub.docker.com/r/ocaml/opam2-staging that does arm64 and amd64 currently. Blog post soon as it as stabilises!
OCaml work is ongoing, so this open issue at the cluster can be closed.
And images are being pushed and used on the Docker Hub daily. Thanks for all the help!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
We'll restart the Ampere benchmarking efforts when the time is ripe; closing for now.
@avsm can you update on the Ampere setup? I see a couple of machines in use; would appreciate any news.
I missed the github ping, sorry about that @vielmetti! Here's the update:
We are using the Amperes happily in production to drive the ocaml.org infrastructure CI, all documented at https://github.com/ocurrent/overview. From those, the most ARM-relevant ones are:
- https://base-images.ocamllabs.io -- this pushes to https://hub.docker.com/r/ocaml/opam and it is arm64/arm32 multiarch there (which we can only do on the Amperes and not the ThunderX which is arm64 only)
- running arm64/arm32 builds on a few hundred repositories for the core OCaml ecosystem: https://ci.ocamllabs.io:8100 (click through the + boxes to see it in action)
- the MirageOS (https://mirage.io) CI now uses arm64 by default thanks to these Amperes, and x86 as a secondary option :-) https://ci.mirage.io
We're a bit computationally limited for bulk builds (http://check.ocamllabs.io) due to some of the Amperes having hardware failures, and also need some capacity to run systematic arm64 benchmarking for core OCaml development (e.g. see ocaml/ocaml#10039 for somewhere we are using a Raspberry Pi 4 as its better than the ThunderX boxes).
(We're acknowledging Equinix on our credits site at https://ocaml.org/contributors.html -- please let me know if ARM should also be there)