Should the initial version of Zebra be able to create blocks?

[hdevalence]

This question started as one about mining code, and morphed into the one in the title.

For context, because we have a small team and high demands on software quality, we'd like to have the minimum code necessary to support any given feature. Supporting CPU mining is clearly not a priority -- the mainnet hashrate is high enough that CPU mining will never be viable, so any actual mining will be done by ASICs or FPGAs, or possibly GPU miners.

In any case, the mining implementations will be external to Zebra. Although supporting external miners is a priority (as long as we continue to rely on proof-of-work mining), miners already use zcashd or something else, so providing an interface for them seems fine to defer past the initial Zebra release.

Deciding that CPU mining isn't a priority doesn't actually help us save code size, because a CPU mining implementation is necessary for testnet implementations to test block creation for private testnets or in CI. (The public testnet is also full of GPU miners).

However, this raises the following question: what is the benefit of having test code that tests functions which will never be executed on mainnet anyways? Instead of maintaining a CPU miner to test block creation functions which will never be used, we could alternately delay block creation functionality to a future version of Zebra. In this case, the initial release of Zebra would provide a full-node implementation that can sync the chain, verify transactions, create transactions, and broadcast them to the network, and we would temporarily make explicit what was previously the de facto implicit reality (that block creation is performed by miners, not by fullnode operators). In a future release, we can provide a hook for external miners and implement block creation.

This poses a slight complication in that our CI and testing infrastructure will, at the beginning, require a zcashd participant. But since we want to test against zcashd anyways, and we have potentially large code savings (upwards of 5% of the total codebase from mining code alone, plus savings from not supporting the block creation logic or message handling), this seems worthwhile.

[str4d]

I think this is fine. No one mines directly with zcashd either anymore (the mining-related text in the metrics UI should probably be disabled on mainnet). zcashd is used to either generate block templates for mining pool software (the getblocktemplate RPC method), or simply as a source of mempool transactions that the mining pool software assembles into blocks by itself.

Something somewhere has to assemble the full candidate block (minus PoW). zcashd provides an API for that, but it's not unreasonable for Zebra to not provide any API, or to have some alternate smaller API on top of which someone could build a block-filling binary.

what is the benefit of having test code that tests functions which will never be executed on mainnet anyways?

zcashd isn't testing the exact same code as is executed on mainnet - difficulty adjustment is ignored in regtest mode, and the maximum target (lowest difficulty) is used. This is primarily so that integration tests complete in faster-than-real-time. In the context of this issue, what matters is testing the block-filling code, and I don't see a benefit to including and testing that code before you actually plan to expose and support it.

This poses a slight complication in that our CI and testing infrastructure will, at the beginning, require a zcashd participant.

I'm imagining that in future instead of relying on a zcashd node for block production, the test harness could instantiate one or more "block miner" entities, each obtaining candidate transactions from specific Zebra nodes in the test setup. This could also make it easier to test network behaviour under (distributions of) various transaction selection policies.

[hdevalence]

Okay, what do we think of the following plan:

For Zebra 2, remove mining code and the ability to generate new blocks.

For Zebra 3, add the following functionality:

• block templating functionality that can prepare block templates for external block producers;
• the ability to accept externally-produced blocks and broadcast them to the network;
• a testing-oriented implementation of an external block producer for use in testnets.

[gtank]

This sounds good to me. We'll want those features in place to be able to fully participate in NU3, but we don't need to support them until then.

[hdevalence]

Okay, if there are no objections (@dconnolly ?) I'll close this issue as having been resolved in favor of the plan above.