Fix local development with Pub/Sub Emulator

[gkinsman]

Hi there,

Currently the dotnet functions framework doesn't seem to work with the pubsub emulator, due to I believe the same issue as with other functions framework implementations (GoogleCloudPlatform/functions-framework-nodejs#272).

I've just hit this, as I'm receiving an error that isn't occurring when the function runs against real pub/sub.

[Google.Cloud.Functions.Framework.CloudEventAdapter] [fail] Event is malformed; does not contain a payload, or the event ID is missing.

[jskeet]

I'll look at this when I'm back at work in January.

[jskeet]

Okay, now I'm back at work... could you clarify what exactly you'd expect the flow to be for local development? If you're expecting the emulator to push events to the Functions Framework, but the emulator doesn't support creating CloudEvents, then that feels like it's more of either a missing feature of the emulator, or something missing in-between. I wouldn't want the Functions Framework to take a dependency on a different format that the emulator happens to create.

If you can explain carefully what you're expecting, I can try to work out some potential options, and will consult with @matthewrobertson about them too.

[gkinsman]

Thanks for the reply!

I'm trying to establish an end-to-end local environment for running tests and local development that doesn't depend on anything in GCP itself.

I have these components:

1. A container that publishes cloud events to PubSub via emulator or live, depending on config. This cloud event gets wrapped up in PubSub's own MessagePublishedData, which the function handles.
2. A GCP function that consumes MessagePublishedData from the topic that is published to by (1). Locally this is run on a dev port, i.e. http://localhost:8000. This cloud event is unwrapped in my code to extract the real cloud event, but it never reaches that as the payload that the emulator sends to the function appears to be in the wrong format.

How I use the emulator
Because function topic subscriptions are handled by function deployment, I manually subscribe my local running function (on localhost:8000) to the topic in the emulator using a PushConfig.PushEndpoint set to the URL of the locally running function. It's possible this is the wrong wireup, but I couldn't find any docs on how else this could be setup.

The issue is that that function is receiving a message with this format:

{
  "subscription": "topicname",
  "message":  "base64messagedata", // This is my own cloud event, _not_ MessagePublishedData
  "messageId": 1,
  "attributes": { 
    "Content-Type": "application/cloudevents+json; charset=utf-8"
  }
}

Which is different to what GCP sends it. This causes the function framework to fail when using a typed CloudEvent with the following error:
"[Google.Cloud.Functions.Framework.CloudEventAdapter] [fail] Event is malformed; does not contain a payload, or the event ID is missing."

It appears that there is some internal marshalling logic that happens in the GCP function framework to change the format to something the function understands. Other functions frameworks seem to have solved this by adding the marshalling logic to the function runtime.

I hope I'm on the right track here!

[jskeet]

Right - the PushConfig.PushEndpoint really isn't designed to send a CloudEvent or the legacy PubSub event format that a function would normally see.

I'm assuming that in production, you'll end up subscribing to PubSub events either using Eventarc or using the gcloud functions deploy --trigger-topic option.

Rather than add this very-special-casing support to the Functions Framework, I would personally suggest adding a sort of proxy, probably via an IHttpFunction which accepts the format from PushConfig.PushEndpoint and then sends a request to your ICloudEventFunction<MessagePublishedData> creating an actual CloudEvent.

We could potentially have the code for that within this repo - I'd just prefer not to have it within the Functions Framework itself. Basically I want to move away from legacy event formats rather than accepting more of them :) I'll discuss it with @matthewrobertson though. (It's definitely feasible to change the adapter to accept the format from PubSub; it's just not something I'd like to do if we can avoid it.)

[gkinsman]

Ah that makes sense - I had a hunch the push endpoint behaved a little differently.

Your assumption is correct - the function is deployed using --trigger-topic.

I think your suggestion would work - the unknown for me is in accurately re-creating that MessagePublishedData format from what is supplied by PushEndpoint, but that might be straightforward. I'll give it a go.

Totally understand not wanting to have this in the functions framework itself - it doesn't feel like it belongs there as its emulatorey test stuff. I guess my issue has been in getting this happy path running locally, so if there was a template or code sample that setup that proxy to forward to a function (could some kind of host extension provide this?) then that would make the happy path pretty great!

[jskeet]

I can try to create some sample code this afternoon, to at least unblock you while we work out a longer-term approach. Glad we're all on the same page now - sometimes that can be half the difficulty!

[gkinsman]

Thanks so much, and thanks for the quick responses!

[jskeet]

Right, I've created a brief demo here: #235

It consists of three projects:

• The "proxy" function
• A demo CloudEvent PubSub function which just logs the message ID and text
• A console app which creates a topic and subscription, and publishes messages to the topic

Hopefully it's reasonably self-explanatory. Running it consists of having four tabs open:

• Start the emulator
• Start the proxy function on a non-default port - I've used dotnet run -- --port 8081
• Start the "real" function
• Run the console app, e.g. dotnet run -- projects/demo/topics/demo-topic http://127.0.0.1:8081 10

I'm not sure whether we'll want to include this in the repo anywhere, but hopefully it'll be useful to you as a starting point.

[gkinsman]

Ah thanks very much, that's pretty much what I expected. Is connecting a function to the emulator not a more common scenario? I'm a little surprised this hasn't come up before (at least not in my googling) as I'd have thought it would be preferable to testing with a real GCP project.

Thanks very much for your help 🙂

[jskeet]

All I can say is that I don't remember see this before. But I only see feature requests etc for .NET.

[jskeet]

@gkinsman: I've now added support for this in the Functions Framework, but we're not going to release until #239 has been resolved.
(Your input into that discussion would be welcome, by the way.)