Add Web3 1.0 "subscription" support with a polling subprovider

Question

Add Web3 1.0 "subscription" support with a polling subprovider

danfinlay opened this issue 7 years ago · 36 comments

The MVP version of #3639, a refined version of the older/messy #2350.

To allow dapp developers to use web3 1.0 in the least possible time, we could add support for the "subscription" API (provided by websockets normally), even over our current polling-based HTTP connection.

This will involve adding a subscription subprovider (very similar to our current filter subprovider to MetaMask's current provider, which is composed from json-rpc-engine as viewable here. A subscription subprovider already exists in a similar but incompatible form in provider-engine here, and that one can be used as a model.

Major differences between the provider-engine and json-rpc-engine versions:

provider-engine subproviders are objects with a handleRequest method. json-rpc-engine subproviders are simple functions.
provider-engine handlers take three params: (payload, next, end). json-rpc-engine breaks params into two params: (req, res, next, end).
provider-engine included a reference to a block-tracker, json-rpc-engine is unaware of the ethereum protocol, and so subproviders need these data sources passed in as function parameters.

Additionally, these subscriptions should not persist with each individual page, and should clean up themselves. One approach to this we've explored in the past is to initialize this "subscription" not in the main network provider, but in the inpage provider which pipes each page's API requests into the core MetaMask process. This allows the memory management of the "fake" (client-side) subscriptions to benefit from the browser's own content-script memory management.

When this works, users should be able to import web3 1.0 and use the subscription API with no issue, like this:

// Importing the latest version from NPM
const Web3 = require('web3')

// initializing a local instance
const web3 = new Web3(window.web3.currentProvider)
web3.eth.subscribe('logs', {
  address: '0x86fa049857e0209aa7d9e616f7eb3b3b78ecfdb0',  // EOS token is popular atm.
  topics: [], // A token transfer log would be good for now.
}, function (err, result) {
  if (err) throw err
  console.log('Success!', result)
})

I will try to continue refining and updating this spec here on the parent comment to minimize required reading.

Some other optional approaches to solving this issue are reviewed in this comment.

A brief summary of the architecture involved in this feature is in this comment #3639.

Bounty information

Because we have moved our develop branch over to json-rpc-engine, which lacks subscription support, this issue is now a deploy-blocker. We never officially supported the subscription method before, but it was merged in, and since trying to publish our latest version, we've become aware that several major dapps and at least one framework (Drizzle) build on our subscription support. This dramatically increases the urgency of the completion of this feature.

To be paid out, the bounty hunter should:

provide a subscription middleware
include tests similar to the previous subprovider.
that at least matches the coverage of the previous implementation.
should perform at least as well as the previous implementation.

This is a contest-type bounty, and so multiple devs can take up the cause at once, but we will only pay out an acceptable implementation, so please do communicate as you progress, and ask about any major tradeoffs you're considering making.

We will also be pursuing this feature, but since we have been so slow to complete it, we invite you to race us, and get credit for this valuable feature.

matthewlilley commented 7 years ago

#4001

Answer 1 · 2018-03-22T19:13:57.000Z

To help clarify what would be involved in adding websocket support (optional to this feature, since this can also be achieved with the existing HTTP subprovider), I've enhanced the issue description here:

MetaMask/web3-provider-engine#189

Answer 2 · 2018-03-22T23:14:58.000Z

This issue now has a funding of 1.0 ETH (527.42 USD @ $527.42/ETH) attached to it.

If you would like to work on this issue you can claim it here.
If you've completed this issue and want to claim the bounty you can do so here
Questions? Get help on the Gitcoin Slack
$3549.77 more Funded OSS Work Available at: https://gitcoin.co/explorer

Answer 3 · 2018-03-25T07:26:44.000Z

@danfinlay I'd like to see if I can help on this but I may need some guidance...

I'm a bit confused by the inpage-provider using the json-rpc-engine and the suggestion to use provider-engine/subprovider/subscriptions.js. Those appear to be incompatible with each other? How are those supposed to work together ?

Answer 4 · 2018-03-26T06:14:31.000Z

I've had a chance to look at the code today and I think I am going to take a step back and let someone else give this a go. This seems to be a lot more complicated than I initially anticipated and i don't know the ethereum ecosystem well enough right now. I'll monitor this and help if I can but for now I don't think Ill be able to help.

Answer 5 · 2018-03-26T18:22:02.000Z

We need a second bounty for doing this for json-rpc-engine.

Answer 6 · 2018-03-26T21:26:21.000Z

@lazaridiscom I think you linked to the wrong issue.

Answer 7 · 2018-03-27T14:02:39.000Z

@danfinlay I've got subscriptions working out of the box with currentProvider and Web 1.0, based on previous work by benjamincburns. I'll try to get it wrapped up tomorrow if possible.

Answer 8 · 2018-04-01T19:00:49.000Z

@lazaridiscom Sorry, I haven't had a chance to pick this back up again, I'll stop work on this for now since it's Easter and kids are off School, so I'm not going to have much free time for the next week. Benjamincburns has done most of the work for this already. Once the subscription subprovider is added to ZeroProvider, you can subscribe over the RPC stream, and get back a subscription id from the RPC handle method. I can see the successful subscription through the generated background page, but the Web3 subscribe callback seems to get lost somewhere.

Answer 9 · 2018-04-07T22:21:00.000Z

Howdy @danfinlay: Want me to pub this one out to the community or is it closer to being completed in house? Curious on your feedback here 🤔

@lazaridiscom Good to see you 🙂

Answer 10 · 2018-04-09T09:13:54.000Z

@lazaridiscom @danfinlay I'm back Today so I'll finish this off. Last week I setup a minimal example using the same wiring as metamask extension. I've just tested that setup in metamask extension Today and it's working. I'll submit a PR asap.

Answer 11 · 2018-04-09T11:44:40.000Z

I have a bug with web3.clearSubscriptions, but it seems to be related to web3 itself since it wasn't working with the regular websocket provider either. Other than that, this looks good. I'll make the PRs to the effected repos Today.

Answer 12 · 2018-04-09T23:10:33.000Z

@lazaridiscom @danfinlay

Can you try this https://github.com/matthewlilley/metamask-extension

Answer 13 · 2018-04-10T17:31:49.000Z

@lazaridiscom Just build the extension, "npm run start", and add the dist/chrome folder as a Chrome extension.

Then you can use Web 1.0 subscription code like @danfinlay posted in the OP, with the Metamask inpage provider.

Answer 14 · 2018-04-24T16:15:22.000Z

@matthewlilley are you still working on this issue?

Answer 15 · 2018-04-27T16:15:32.000Z

@matthewlilley are you still working on this issue?

warning 1 (3 days)
warning 2 (6 days)
auto removal (10 days)

Answer 16 · 2018-04-28T13:30:03.000Z

@danfinlay @kumavis This one looks like it may be in your review? Apologies for Gitcoin Bot's commentary, @matthewlilley!

cc @owocki

Answer 17 · 2018-04-30T16:15:18.000Z

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

@matthewlilley has been removed from this issue due to inactivity (10 days) on the github thread. @matthewlilley if you believe this was done in error, please go to the bounty and click 'start work' again.

warning 1 (3 days)
warning 2 (6 days)
auto removal (10 days)

Answer 18 · 2018-05-03T07:26:52.000Z

@vs77bb Yes, just waiting for review on this one. #4047

Answer 19 · 2018-05-10T09:55:16.000Z

@danfinlay @vs77bb Any updates on this? I'm wondering if there's a reason why it cannot be merged, since this is only supposed to provide support until websocket subscriptions are supported by default.

Answer 20 · 2018-05-10T17:13:39.000Z

Hi @matthewlilley, my understanding is that @kumavis is currently working on a robust version of this feature, and prefers that solution to this one.

I'm very sorry you've been dragged along here. It was probably unfair to post a bounty on an issue that was so large and we were going to be so picky about the implementation of.

I would be willing to pay out the bounty, because it looks like you have a working version, but I think we're hoping to go full websockets soon, so we might not need to push this to production.

@vs77bb Go ahead and pay out, we shouldn't have made Matthew wait so long.

Answer 21 · 2018-05-10T17:30:40.000Z

@danfinlay Thanks for getting back to me. I'd assumed you were working on a more thorough solution.

It's okay, I understand. This was merely supposed to provide MVP support, since the transition to a Websocket provider was underway, and I wanted a way to use subscriptions with MetaMask out of the box when building dApps.

Appreciate it!

Answer 22 · 2018-05-14T13:58:24.000Z

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

@matthewlilley

has committed to working on this project to be completed 1 month from now.

Answer 23 · 2018-05-14T14:31:02.000Z

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work for 1.0 ETH (711.13 USD @ $711.13/ETH) has been submitted by:

@matthewlilley

If you are the bounty funder, please take a look at the submitted work:

(Link Not Provided) by @matthewlilley

Learn more on the Gitcoin Issue Details page
Questions? Checkout Gitcoin Help or the Gitcoin Slack
$6,918.86 more funded OSS Work available on the Gitcoin Issue Explorer

Answer 24 · 2018-05-15T23:05:08.000Z

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

The funding of 1.0 ETH (712.71 USD @ $712.71/ETH) attached to this issue has been approved & issued to @matthewlilley.

Learn more at on the Gitcoin Issue Details page
Questions? Checkout Gitcoin Help or the Gitcoin Slack
$6,566.87 more funded OSS Work available on the Gitcoin Issue Explorer

Answer 25 · 2018-05-23T11:26:10.000Z

I'm very interested in this feature, where can I see the status? Is there a PR?
Thanks 😄

Answer 26 · 2018-05-23T18:10:35.000Z

Current progress on this feature is resuming on this issue:
#4279

Answer 27 · 2018-07-22T21:30:29.000Z

I hope #4279 is merged soon but till that time if someone is desperate like me they can use upperwal/web3-event-compatibility

Someone suggested in the comments to use two versions of web3 so I tried that and it worked 💪. So thanks to that guy (sorry don't remember who that was)

If you want to see how it works check out upperwal/react-box. It is a truffle box for react which supports web3 v1.x.x

Refer this for how to use it (supports existing API but only callback) and this to initialise it.

Answer 28 · 2018-09-18T00:52:26.000Z

More work is still required after #4279. In 4.10 we still get the The current provider doesn't support subscriptions: MetamaskInpageProvider error message when using web3.eth.subscribe.

Answer 29 · 2018-10-05T17:44:12.000Z

Is there an ETA for subscription support in MetaMask? I don't intend to push, I just need to make decisions for a project I'm working on. With an ETA I know how to move forward. Thanks!

Answer 30 · 2018-10-05T20:04:21.000Z

We inadvertently introduced support for subscriptions in April, but have since introduced a refactor of our provider logic which no longer includes this feature. We are actively working on it now, and I am about to post a bounty to push its urgency up.

Answer 31 · 2018-10-05T20:11:39.000Z

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 5.0 ETH (1116.53 USD @ $223.31/ETH) attached to it.

If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
Want to chip in? Add your own contribution here.
Questions? Checkout Gitcoin Help or the Gitcoin Slack
$39,917.78 more funded OSS Work available on the Gitcoin Issue Explorer

Answer 32 · 2018-10-05T20:12:12.000Z

Bounty information (Added to parent post)

Because we have moved our develop branch over to json-rpc-engine, which lacks subscription support, this issue is now a deploy-blocker. We never officially supported the subscription method before, but it was merged in, and since trying to publish our latest version, we've become aware that several major dapps and at least one framework (Drizzle) build on our subscription support. This dramatically increases the urgency of the completion of this feature.

To be paid out, the bounty hunter should:

provide a subscription middleware
include tests similar to the previous subprovider.
that at least matches the coverage of the previous implementation.
should perform at least as well as the previous implementation.

This is a contest-type bounty, and so multiple devs can take up the cause at once, but we will only pay out an acceptable implementation, so please do communicate as you progress, and ask about any major tradeoffs you're considering making.

We will also be pursuing this feature, but since we have been so slow to complete it, we invite you to race us, and get credit for this valuable feature.

Answer 33 · 2018-10-07T11:14:43.000Z

@danfinlay I'm interested in this one, but I feel that I need some help understanding the repos.

Before start, I'me trying to understand the middlewares flow.

Let's take the wallet middleware as example. It defines a dictionary of Ethereum JSON RPC methods.

Can I assume that when client uses web3.eth.accounts eventually those methods are called?
I see that wallet middleware do just part of the job. Other part of the workflow is given in functions like lookupAccounts, lookupDefaultAccount etc. Can you show me where those methods defined?

Answer 34 · 2018-10-09T15:59:39.000Z

Sorry for any bounty hunters who had interest in this, we have a PR we expect to close this issue:
#5458

Answer 35 · 2018-10-23T09:24:35.000Z

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 4 months ago.
Please review their action plans below:

1) liamzebedee has been approved to start work.

I encountered this issue a couple weeks back at work. Currently am connecting to two providers, one for tx's and one for realtime events. Confident that I can approach this issue and fix it, and I'm sure it's a big trouble for many others too.

Learn more on the Gitcoin Issue Details page.