Clarc: a Clarity compiler for the Ethereum virtual machine
artob opened this issue · 26 comments
Background
What problems do you aim to solve? How does it serve the mission of a user owned internet?
I believe that Clarity is a much-needed evolution towards safer smart contracts, and I wish to see it much more broadly adopted. To that end, I would wish to bring Clarity also to the Ethereum developer community, who are indeed in sore need of it: as is widely understood, poorly-designed Solidity smart contracts have already resulted in hundreds of millions of dollars worth of mayhem.
Broader adoption of Clarity can help mitigate and solve these problems. Indeed, there is a chance that with sufficient and serious backing, Clarity could in time become an important industry standard for smart contracts on a range of blockchains. As it stands, Clarity is already available on Blockstack and soon on Arweave as well, and the Ethereum network would be an obvious--and in terms of current user base, no doubt the most important--next evolution.
Project Overview
What solution are you providing? Who will it serve?
I wish to develop a Clarity compiler to Ethereum virtual machine (EVM) bytecode, enabling the use of Clarity for writing smart contracts for Ethereum as another language choice alongside the current choices of Solidity and Vyper. And I wish to publicize the project widely through blogging and conferences, driving adoption of Clarity.
This will directly serve Ethereum developers in giving them a safer choice for writing their smart contracts, and it will directly serve Ethereum users in giving assurance that the behavior of contracts can be correctly understood. In particular, an audit of a Clarity smart contract will provide a higher level of assurance than can be said for audits of Solidity contracts, and audits can be significantly facilitated by automated tooling due to the decidable nature of Clarity.
The network effects of (much!) broader adoption of Clarity will directly and indirectly serve the Stacks Foundation and the Stacks developer community at least as follows:
- More visibility (hence adoption) for Stacks in the currently most important blockchain developer ecosystem
- More ease in porting contracts from Ethereum to Stacks (hence more adoption), and vice versa
- More cross-transfer of developer skills (hence more Stacks developers)
- More resources for best-of-class tooling for Clarity (hence a better developer experience for everyone)
- An additional implementation and thorough independent review of the Clarity language, nailing down murky corners of the language (of benefit to the Stacks 2.0 mainnet launch)
Scope
What are the components or technical specs of the project? What will the final deliverable look like? How will you measure success?
The project will be delivered as a public-domain GitHub project containing the full source code for the compiler, as well as offered downloads of pre-built standalone executables for Windows, macOS, and Linux. In addition, the executable will have a manual page and there will be further user documentation at https://clarc.dev.
The compiler will be written in OCaml, an excellent programming language for crafting compiler toolchains. Clarc will be a multi-pass compiler consisting of the following stages:
Success will be easier to measure than for most projects: we should be able to compile a substantial subset of existing Clarity contracts, with minimal changes for porting to Ethereum, into usable smart contracts for Ethereum.
Budget and Milestones
What grant amount are you seeking? How long will the project take in hours? If more than 20, please break down the project into milestones, with a clear output (e.g., low-fi mockup, MVP with two features) and include the estimated work hours for each milestone.
The total requested budget to bring the project to fruition and a 1.0 release is $5,000.
I would in fact request more, except for your current $5K limit during the beta grants program. Do kindly note that my actual opportunity cost is a multiple of the proposed budget, but I do consider this a hugely worthwhile project and am passionate to make it happen.
Based on my previous commercial work with Ethereum bytecode generation and analysis, as well as on my recent and ongoing work on a Clarity compiler for SmartWeave, I believe I'm in a good position to estimate this project fairly predictably.
I would be available to work on this part-time from next week and full-time from the middle of October, with project delivery by November. Therefore I suggest the following rough timeline and major milestones:
Release 0.1 (proof of concept) - 20h
- Compiles the trivial
counter.clarexample and tutorial code into EVM bytecode.
Release 0.5 (prototype) - 40h
- Compiles two of the Clarity contracts showcased in friedger/clarity-smart-contracts into EVM bytecode.
Release 1.0 (minimum viable product) - 60h
- Compiles many or most of the Clarity Hackathon contracts (with minimal changes for porting to Ethereum) into usable EVM contracts.
- Published user documentation.
- Released as pre-built binaries for Windows, macOS, and Linux.
Team
Who is building this? What relevant experience do you bring to this project? Are there skills sets you are missing that you are seeking from the community?
I will be building this by myself. I am a prolific open-source author, have 20+ years of professional programming experience, and I have all the relevant deep expertise for toolchains, compilers, and the Ethereum virtual machine. See my GitHub, LinkedIn, and website for more about me.
Risks
What dependencies or obstacles do you anticipate? What contingency plans do you have in place?
The main obstacles on this project are twofold:
-
The Clarity language is as yet underspecified and something of a moving target. This will necessitate some back and forth with Blockstack's Clarity team (@lgalabru et al) to nail down murky corners of the language. This isn't perhaps so much an obstacle as an opportunity to improve the documentation around Clarity and indeed move towards developing a proper formal specification for Clarity.
-
Not all Clarity constructs map directly to the EVM as is. At the least, some Blockstack-specific functions will need to be omitted or emulated, and we will need to discuss and decide how to best handle the difference of Clarity's 128-bit arithmetic given that the EVM is based on a 256-bit machine word size.
Community and Supporting Materials
Do you have previous projects, code commits, or experiences that are relevant to this application? What community feedback or input have you received? How do you plan to share your plan to the community over time and as the final deliverable?
Earlier this month, I was awarded the joint Blockstack and Arweave grant for enabling Clarity on Arweave's smart contract protocol. That work will reach completion (that is, a 1.0 release) by next week, and directly motivates this grant proposal as a follow-up project while everything about the internals of Clarity is still fresh and clear in my mind. Some of the code (in particular, for parsing Clarity code) will be shared between the two projects.
I plan to regularly share my progress in tweets, Discord and forum updates, and educational and motivational blog posts over the course of the project. The blog posts will be targeted at both the Stacks and Ethereum developer communities in order to build interest and begin building up an open-source community of contributors around the project. In addition, I plan to speak at conferences about the construction of this compiler and showcasing how to write contracts in Clarity for Stacks, Arweave, and Ethereum, with the recordings made available on YouTube.
Would love to see this. Thoughts @njordhov ?
I've posted this proposal for discussion on the forum as well: https://forum.blockstack.org/t/clarity-for-ethereum/11213
Hello! We are excited to share that your grant application is approved!
The agreed upon milestones are
October 15th for Proof of Concept ($1000)
- Compiles the trivial counter.clar example and tutorial code into EVM bytecode.
October 22nd for Prototype ($1000)
- Compiles two of the Clarity contracts showcased in friedger/clarity-smart-contracts into EVM bytecode.
November 5th for Minimum Viable Product ($3000)
- Compiles many or most of the Clarity Hackathon contracts (with minimal changes for porting to Ethereum) into usable EVM contracts.
- Published user documentation.
- Released as pre-built binaries for Windows, macOS, and Linux.
The next step will be to send you over a contract so we can disburse payment at the given milestones, please email grants@stacks.org with your contact information.
Thank you!
@ryanarndtcm Thank you for the happy news. I'll be in touch via email.
This is fantastic. Congratulations on your grant.
The grant contract has now been signed, and I've created the project repository at weavery/clarc just now. (If you're interested in the project, go ⭐ it!)
Funded Milestone 1 development: $1000.
Hi @artob we're reopening this issue, where you can post your updates on milestones. We're having a demo day at the end of the month, and we'd love to have you show off what you've built so far on M1 and M2, are you game to join.
@RaffiSapire I discussed this with @ryanarndtcm on Discord, and yes, count me in. I've added it to my calendar.
My end-of-week update: I've got properly started with the project this week, implementing EVM bytecode encoding and decoding. I remain on track to complete M1 (the proof-of-concept) on schedule next week.
@ryanarndtcm @blocks8 @RaffiSapire Ahead of schedule, I have today completed M1 and released Clarc 0.1.0, a proof-of-concept that's able to compile the Clarity counter.clar example contract into usable Ethereum virtual machine (EVM) bytecode.
Here's a demo screencast of Clarc compiling, deploying, and interacting with this contract on an EVM simulator:
Here's the symbolic EVM opcode output from Clarc for this contract:
$ clarc counter.clar -t opcode
PUSH1 0x00 PUSH1 0x00 SSTORE PUSH1 0x64 DUP1 PUSH1 0x10 PUSH1 0x00 CODECOPY
PUSH1 0x00 RETURN PUSH1 0xe0 PUSH1 0x02 EXP PUSH1 0x00 CALLDATALOAD DIV DUP1
PUSH4 0x8ada066e EQ PUSH1 0x28 JUMPI DUP1 PUSH4 0xd09de08a EQ PUSH1 0x36
JUMPI DUP1 PUSH4 0x2baeceb7 EQ PUSH1 0x4d JUMPI STOP JUMPDEST POP PUSH1 0x00
SLOAD PUSH1 0x00 MSTORE PUSH1 0x20 PUSH1 0x00 RETURN STOP JUMPDEST POP
PUSH1 0x01 PUSH1 0x00 SLOAD ADD PUSH1 0x00 SSTORE PUSH1 0x00 SLOAD PUSH1 0x00
MSTORE PUSH1 0x20 PUSH1 0x00 RETURN STOP JUMPDEST POP PUSH1 0x01 PUSH1 0x00
SLOAD SUB PUSH1 0x00 SSTORE PUSH1 0x00 SLOAD PUSH1 0x00 MSTORE PUSH1 0x20
PUSH1 0x00 RETURN STOPHere's the equivalent deployable EVM bytecode output from Clarc for this contract:
$ clarc counter.clar -t bytecode
600060005560648060106000396000f360e060020a6000350480638ada066e146028578063d09de08a1460365780632baeceb714604d57005b5060005460005260206000f3005b5060016000540160005560005460005260206000f3005b5060016000540360005560005460005260206000f300The compiler source code can be found at lib/Clar2EVM/compile.ml and the compiler driver (i.e., the clarc program) at bin/clarc/clarc.ml. The EVM bytecode encoder can be found at lib/EVM/encode.ml and the Clarity language parser is maintained in my standalone Clarity.ml library subproject.
So far no blockers on this project, just a ton of work. I'll share this news on the usual media, and then onwards to M2! 🚀
Hey awesome @artob! We are reviewing it and will disburse M2. Thanks!
Raffi
I am in the process of completing M2, getting close to compiling the two panic.clar and kv-store.clar example contracts.
I'll need another day or so to wrap that up and record demos, though. Expect an end-of-week update to follow.
I have today completed M2 and released Clarc 0.5.0, a prototype that's able to compile two more example contracts, panic.clar and kv-store.clar from the friedger/clarity-smart-contracts repository of Clarity contracts. 🚀
As the EVM simulator I've been using doesn't support all opcodes emitted by the compiler, I'm still in the process of figuring out how to best record a screencast demo of interacting with these contracts. While I figure that out, I'll get at least some web3.js scripts posted here soon.
I have already deployed the compiled bytecode for the three current supported contracts to the public Ropsten testnet, here:
| Contract ID | Contract Code | Contract Bytecode | Contract ABI |
|---|---|---|---|
| 0x8a90b1e93020933295b3bd4ce2317062319351d4 | counter.clar |
counter.bin |
counter.json |
| 0x9a1b29fc432af1e37af03ed2fee00d742ff7372f | panic.clar |
panic.bin |
panic.json |
| 0x2e2487c64b1420111e8d66d751f75f69515c5476 | kv-store.clar |
kv-store.bin |
kv-store.json |
It's possible to interact with these contracts using MyEtherWallet, for example.
The new supported contracts demonstrate several more implemented features of Clarity on EVM; for example:
- Clarity's panics are mapped to the EVM
REVERTopcode - Clarity's
tx-sendermaps to the EVMCALLERopcode - Clarity's
principaltype maps nicely to Ethereum addresses (addressin terms of the Solidity ABI) - Clarity's persistent storage for variables and maps is implemented using the EVM
SLOADandSSTOREopcodes - We are able to pack a tuple containing two 128-bit Clarity integers into a single 256-bit EVM machine word
I'll update the project README soon with a table documenting how we map Clarity constructs to the Ethereum virtual machine.
The delivery of this milestone was a couple of days late. That was not due to any technical blockers (beyond what I mentioned previously regarding the EVM simulator), everything is going as expected and the work done so far is proportionate to my original estimates for the M1 and M2 milestones. However, I did lose several days due to unexpected personal matters. I will aim to catch up with the original schedule over the course of the next week.
Hey Arto! I am confirming your M2 is reviewed and approved!
@ryanarndtcm @jcnelson Thanks for that! I appreciate that everyone has a hectic schedule right now, so a nice turnaround on that.
For those who missed it, my Clarc demo begins at 31m45s in the recording from yesterday's Stacks Grant Demo Day:
Milestone 2 funded for $1000.
As a quick update from my end, I'm working hard on M3 and will release the next version later this week.
Hi Arto! Hope all is well. Just wanted to check in and see if we can be helpful in anyway in getting this to the finish line :) If its ended up being more work than expected, we can also help figure that out.
Hi Arto! Hope all is well. Just wanted to check in and see if we can be helpful in anyway in getting this to the finish line :) If its ended up being more work than expected, we can also help figure that out.
Thanks for checking in, @RaffiSapire! The final milestone has been a slog, but we aren't far off now. I'll have a comprehensive update for you still this week.
I have today released Clarc 0.6.0, which incorporates about a month of work (diff) since the previous 0.5.0 release.
The current state of things is that we do support most Clarity language constructs (with limitations) and we do compile many contracts in friedger/clarity-smart-contracts, but are still--crucially--missing the implementation of Clarity's fungible-token/non-fungible-token support, which is required by a good number of contracts in that collection.
The ft-* and nft-* functions should be mapped onto the ERC-20 and ERC-721 standards, respectively. I tried for a while to (unsuccessfully) follow my original plan of taking existing, known-good Solidity implementations of those standards, compile them to bytecode, and then link that output into Clarity contracts that implement tokens using these functions.
That turns out to not be a very viable path forward, since EVM bytecode from Solidity isn't really suitable for linking in this way. So, that was, plainly, a mistake I made in my original estimation for how much work it would take to implement the ERC-20/721 token support.
I am now in the process of implementing the ERC-20 and ERC-721 standards directly using EVM opcodes (assembly). That's a bit laborious, but it's proceeding nonetheless. I will post another update later this week as I proceed.
Thank you! We're so excited to see all of this happening. I understand this grant may have been a lot more work than anticipated. We're happy to increase the scope of the grant to compensate you for the contributions you are making. If you want to comment here on what is appropriate given the level of work we can discuss on Tuesday or after the holidays, whatever works for you!
As you're working on the last milestone I've funded the remainder of the grant.
As an update, the work on ERC-20 support is nearing usability, I plan to spend some time over the holidays to wrap that up. ERC-721 will need more work still.
Thank you! We're so excited to see all of this happening. I understand this grant may have been a lot more work than anticipated. We're happy to increase the scope of the grant to compensate you for the contributions you are making. If you want to comment here on what is appropriate given the level of work we can discuss on Tuesday or after the holidays, whatever works for you!
Thanks for the kind words, @RaffiSapire! Would you perhaps have a moment to discuss this--that is, the possibilities for restructuring and re-scoping the grant a bit--on a brief call prior to the holidays? I'm available anytime today (Tuesday) and tomorrow. You can reach me on Discord as Arto#1837 as well as via email.
Hello! Thank you for all your hard work this year. We would love to send you a holiday note and some custom stacks stickers. Please email me at Raffi@stacks.org with your names/addresses if you'd like to receive one. We really appreciate all your help and contributions. Stacks is what it is because of you!