AA Benchmarks

A comprehensive benchmark for smart contract accounts that support account abstraction (ERC-4337), built on Hardhat for accurate, transaction-based, fee measurements (see Methodology) and for use of existing TypeScript utilities around fee calculations. This work was inspired by ZeroDev's work on aa-benchmark, which is built on Foundry.

Methodology

This tool seeks to measure the cost of the entire transaction landed on-chain for each action, including associated L1 fees. As smart accounts are expected to proliferate on L2s, it's important to consider L1 fees (measured as a function of the RLP-encoded signed transaction) on top of L2 execution costs. Generally, L1 fees tend to dominate the cost of transactions on L2s, and therefore, optimizing the size of calldata is important.

The L1 fee calculations are done following the formula and constants specified in Optimism Bedrock.

There are two different categories of benchmarks measured in this test: User Operation and Runtime.

User Operation: Measuring the cost of user operations are tricky because each bundler may calculate pre-verification gas differently and the costs may differ based on the number of user operations in the bundle. In this test, the on-chain cost for a bundler to execute the user operation in a bundle of size 1 is measured, to present a lower-bound fee required for the user operation to land in an exclusive bundle. In other words, fees are calculated based on the transaction receipt and the serialized signed transaction (legacy transaction type) for entryPoint.handleUserOp([userOp]). As multi-user-op bundles become more prevalent, we can expect actual fees to undercut the data presented here.
Runtime: Runtime transactions are defined as those performed outside of the user operation flow, with the owner key interacting directly with the account factory or account, akin to the way you might use MetaMask today to interact directly with smart contracts. Support for runtime transactions also enable contracts to interact directly with your smart account. For this flow, fees are calculated based on the transaction's receipt and the serialized signed transaction (legacy transaction type).

Results

These numbers are derived from local simulations with fixed inputs (see Run benchmark below to try different inputs) and actual on-chain numbers may differ.

Run options

Last run: Wed, 21 Feb 2024 05:53:32 GMT

Option	Value
L2 gas price (Gwei)	`0.004`
L1 gas price (Gwei)	`40`
ETH price (USD)	`$2500`

User Operation: Account creation

	Modular Account	Biconomy v2	Kernel v2.1	Safe	Light Account
L2 gas used	`858351`	`329947`	`296301`	`435486`	`310532`
L2 fee (ETH)	`0.000003433`	`0.000001320`	`0.000001185`	`0.000001742`	`0.000001242`
L1 gas used	`8076`	`9256`	`9516`	`11596`	`7832`
L1 fee (ETH)	`0.000220959`	`0.000253244`	`0.000260358`	`0.000317267`	`0.000214284`
Total fee (ETH)	`0.000224393`	`0.000254564`	`0.000261543`	`0.000319009`	`0.000215526`
Total fee (USD)	`$0.56`	`$0.64`	`$0.65`	`$0.80`	`$0.54`

User Operation: Native transfer

	Modular Account	Biconomy v2	Kernel v2.1	Safe	Light Account
L2 gas used	`165100`	`161912`	`163398`	`176479`	`152880`
L2 fee (ETH)	`0.000000660`	`0.000000648`	`0.000000654`	`0.000000706`	`0.000000612`
L1 gas used	`7292`	`7904`	`7408`	`7504`	`7292`
L1 fee (ETH)	`0.000199509`	`0.000216253`	`0.000202683`	`0.000205309`	`0.000199509`
Total fee (ETH)	`0.000200170`	`0.000216901`	`0.000203336`	`0.000206015`	`0.000200121`
Total fee (USD)	`$0.50`	`$0.54`	`$0.51`	`$0.52`	`$0.50`

User Operation: ERC-20 transfer

	Modular Account	Biconomy v2	Kernel v2.1	Safe	Light Account
L2 gas used	`189487`	`185868`	`187609`	`200732`	`177013`
L2 fee (ETH)	`0.000000758`	`0.000000743`	`0.000000750`	`0.000000803`	`0.000000708`
L1 gas used	`7964`	`8576`	`8092`	`8176`	`7940`
L1 fee (ETH)	`0.000217895`	`0.000234639`	`0.000221397`	`0.000223695`	`0.000217238`
Total fee (ETH)	`0.000218653`	`0.000235383`	`0.000222148`	`0.000224498`	`0.000217946`
Total fee (USD)	`$0.55`	`$0.59`	`$0.56`	`$0.56`	`$0.54`

User Operation: Session key creation

	Modular Account	Biconomy v2	Kernel v2.1	Safe	Light Account
L2 gas used	`365368`	`178212`	`172136`	`-`	`-`
L2 fee (ETH)	`0.000001461`	`0.000000713`	`0.000000689`	`-`	`-`
L1 gas used	`11352`	`8484`	`8932`	`-`	`-`
L1 fee (ETH)	`0.000310591`	`0.000232122`	`0.000244380`	`-`	`-`
Total fee (ETH)	`0.000312052`	`0.000232835`	`0.000245068`	`-`	`-`
Total fee (USD)	`$0.78`	`$0.58`	`$0.61`	`Unsupported`	`Unsupported`

User Operation: Session key native transfer

	Modular Account	Biconomy v2	Kernel v2.1	Safe	Light Account
L2 gas used	`163716`	`-`	`127202`	`-`	`-`
L2 fee (ETH)	`0.000000655`	`-`	`0.000000509`	`-`	`-`
L1 gas used	`8116`	`-`	`9872`	`-`	`-`
L1 fee (ETH)	`0.000222054`	`-`	`0.000270098`	`-`	`-`
Total fee (ETH)	`0.000222709`	`-`	`0.000270607`	`-`	`-`
Total fee (USD)	`$0.56`	`Unsupported`	`$0.68`	`Unsupported`	`Unsupported`

User Operation: Session key ERC-20 transfer

	Modular Account	Biconomy v2	Kernel v2.1	Safe	Light Account
L2 gas used	`194942`	`147023`	`153240`	`-`	`-`
L2 fee (ETH)	`0.000000780`	`0.000000588`	`0.000000613`	`-`	`-`
L1 gas used	`8764`	`11404`	`10848`	`-`	`-`
L1 fee (ETH)	`0.000239783`	`0.000312013`	`0.000296801`	`-`	`-`
Total fee (ETH)	`0.000240563`	`0.000312602`	`0.000297414`	`-`	`-`
Total fee (USD)	`$0.60`	`$0.78`	`$0.74`	`Unsupported`	`Unsupported`

Runtime: Account creation

	Modular Account	Biconomy v2	Kernel v2.1	Safe	Light Account
L2 gas used	`725323`	`197473`	`163890`	`289207`	`185267`
L2 fee (ETH)	`0.000002901`	`0.000000790`	`0.000000656`	`0.000001157`	`0.000000741`
L1 gas used	`2600`	`3184`	`3920`	`5940`	`2356`
L1 fee (ETH)	`0.000071136`	`0.000087114`	`0.000107251`	`0.000162518`	`0.000064460`
Total fee (ETH)	`0.000074037`	`0.000087904`	`0.000107907`	`0.000163675`	`0.000065201`
Total fee (USD)	`$0.19`	`$0.22`	`$0.27`	`$0.41`	`$0.16`

Runtime: Native transfer

	Modular Account	Biconomy v2	Kernel v2.1	Safe	Light Account
L2 gas used	`50408`	`-`	`45364`	`75840`	`39396`
L2 fee (ETH)	`0.000000202`	`-`	`0.000000181`	`0.000000303`	`0.000000158`
L1 gas used	`2756`	`-`	`2900`	`5012`	`2744`
L1 fee (ETH)	`0.000075404`	`-`	`0.000079344`	`0.000137128`	`0.000075076`
Total fee (ETH)	`0.000075606`	`-`	`0.000079525`	`0.000137432`	`0.000075233`
Total fee (USD)	`$0.19`	`Unsupported`	`$0.20`	`$0.34`	`$0.19`

Runtime: ERC-20 transfer

	Modular Account	Biconomy v2	Kernel v2.1	Safe	Light Account
L2 gas used	`74693`	`-`	`69230`	`99715`	`63438`
L2 fee (ETH)	`0.000000299`	`-`	`0.000000277`	`0.000000399`	`0.000000254`
L1 gas used	`3420`	`-`	`3576`	`5684`	`3420`
L1 fee (ETH)	`0.000093571`	`-`	`0.000097839`	`0.000155514`	`0.000093571`
Total fee (ETH)	`0.000093870`	`-`	`0.000098116`	`0.000155913`	`0.000093825`
Total fee (USD)	`$0.23`	`Unsupported`	`$0.25`	`$0.39`	`$0.23`

Run benchmark

Copy .env.sample to .env and update the values within .env:

cp .env.sample .env

Test running the benchmarks over the supported accounts and see the results in your terminal:

pnpm benchmark

Run the benchmarks and update this file with the new results:

pnpm benchmark:write

Contributing

To add a new smart account to this repo, follow the steps below.

Fork this repo.
Add the account's artifacts under test/artifacts.
Implement the account's fixture and export an object matching the AccountConfig interface. Add this file under test/accounts.
Add your account under test/accounts.ts to include your account in the benchmarks.
Generate the results and write to the README by running pnpm benchmark:write.
Open a PR against this repo.

huaweigu/aa-benchmarks