/GoogleCloud_Pub_Sub_with_dotNET8_WebAPI_consumer

Primary LanguageC#

GoogleCloud Pub/Sub with dotNET8 WebAPI messages consumer

Source code for this repo is available in the github:

https://github.com/luiscoco/GoogleCloud_Pub_Sub_with_dotNET8_WebAPI_consumer

1. Create Google Cloud Pub/Sub service

Google Cloud Project: You'll need a Google Cloud Project, if you don't have one, create one at https://console.cloud.google.com

Pub/Sub API Enabled: Make sure the Pub/Sub API is enabled in your project. You can check this or enable it in the "APIs & Services" section of the Google Cloud Console

We first have to log in to Google Cloud Service

image

We have to search for Pub/Sub service

image

1.1. Create a Topic

We press CREATE TOPIC button

image

We input the topic name an press the CREATE button

image

We verify the new topic and the corresponding subscription were created

image

1.2. Create a Service Account

We first navigate to the IAM and Admin service

image

We select the Service Account option in the left hand side menu, then we press the CREATE SERVICE ACCOUNT button

image

We input the Service Account details and we press the CREATE AND CONTINUE button

image

We grant the permission to the service account

image

Once we granted the permissions we press the Continue button

image

We finally press the Done button

image

We verify our new service accoung. We press on the email link

image

We also create a new Key with a JSON format. For this porpuse we click on the Keys tab

image

Then we press Create new key option and we select the JSON option in the subsequent dialog box

image

image

We can see in the download folder the JSON credentials file downloaded from Google Cloud Service Account

image

1.3. Set Environment Variable

We run the application to edit the environmental variables

image

We have to create a new environmental variable called GOOGLE_APPLICATION_CREDENTIALS

This variable have to point to the path where we have placed the JSON key file for our Google Cloud Service Account

image

image

2. Create a .NET8 WebAPI with VSCode

Creating a .NET 8 Web API using Visual Studio Code (VSCode) and the .NET CLI is a straightforward process

This guide assumes you have .NET 8 SDK, VSCode, and the C# extension for VSCode installed. If not, you'll need to install these first

Step 1: Install .NET 8 SDK

Ensure you have the .NET 8 SDK installed on your machine: https://dotnet.microsoft.com/es-es/download/dotnet/8.0

You can check your installed .NET versions by opening a terminal and running:

dotnet --list-sdks

If you don't have .NET 8 SDK installed, download and install it from the official .NET download page

Step 2: Create a New Web API Project

Open a terminal or command prompt

Navigate to the directory where you want to create your new project

Run the following command to create a new Web API project:

dotnet new webapi -n GooglePubSubReceiverApi

This command creates a new directory with the project name, sets up a basic Web API project structure, and restores any necessary packages

Step 3: Open the Project in VSCode

Once the project is created, you can open it in VSCode by navigating into the project directory and running:

code .

This command opens VSCode in the current directory, where . represents the current directory

3. Load project dependencies

We run this command to load the project library

dotnet add package Google.Cloud.PubSub.V1

image

4. Create the project structure

image

5. Create the Controller

PubSubReceiverApiController.cs

using Google.Cloud.PubSub.V1;
using Google.Protobuf;
using Microsoft.AspNetCore.Mvc;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

namespace PubSubReceiverApi.Controllers
{
    public class MessageDto
    {
        public string? Body { get; set; }
        public string? Priority { get; set; }
    }

    [ApiController]
    [Route("api/[controller]")]
    public class PubSubController : ControllerBase
    {
        private static string projectId = "endless-set-412215";
        private static string subscriptionId = "mytopic-sub";
        private static SubscriberClient? subscriber;
        private static ConcurrentQueue<MessageDto> receivedMessages = new ConcurrentQueue<MessageDto>();

        [HttpGet("receive")]
        public ActionResult<IEnumerable<MessageDto>> ReceiveMessages(string? priority = null)
        {
            if (string.IsNullOrEmpty(priority))
            {
                return receivedMessages.ToList();
            }
            else
            {
                return receivedMessages.Where(m => m.Priority == priority).ToList();
            }
        }

        // Ensure this method is defined and matches what is called in Program.cs
        public static async Task StartMessageProcessing(CancellationToken stoppingToken)
        {
            SubscriptionName subscriptionName = SubscriptionName.FromProjectSubscription(projectId, subscriptionId);
            subscriber = await SubscriberClient.CreateAsync(subscriptionName);
            await subscriber.StartAsync((message, cancellationToken) =>
            {
                // Message handling logic
                string body = message.Data.ToStringUtf8();
                message.Attributes.TryGetValue("priority", out string priority);
                Console.WriteLine($"Received message: {body}, Priority: {priority}");

                receivedMessages.Enqueue(new MessageDto { Body = body, Priority = priority ?? "normal" });

                return Task.FromResult(SubscriberClient.Reply.Ack);
            });
        }
    }
}

6. Modify the application middleware(program.cs)

Program.cs

using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.OpenApi.Models;
using PubSubReceiverApi.Controllers;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHostedService<PubSubBackgroundService>();

// Add services to the container.
builder.Services.AddControllers();

// Add Swagger
builder.Services.AddSwaggerGen(c =>
{
    c.SwaggerDoc("v1", new OpenApiInfo { Title = "PubSubReceiverApi", Version = "v1" });
});

var app = builder.Build();

// Configure the HTTP request pipeline.
app.UseRouting();

// Enable middleware to serve generated Swagger as a JSON endpoint.
app.UseSwagger();

// Enable middleware to serve swagger-ui (HTML, JS, CSS, etc.), specifying the Swagger JSON endpoint.
app.UseSwaggerUI(c =>
{
    c.SwaggerEndpoint("/swagger/v1/swagger.json", "PubSubReceiverApi v1");
});

app.UseAuthorization();

app.MapControllers();

app.Run();

7. Create a BackgroundService

PubSubBackgroundService

using Microsoft.Extensions.Hosting;
using System.Threading;
using System.Threading.Tasks;
using PubSubReceiverApi.Controllers;
public class PubSubBackgroundService : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        // Assuming PubSubController has been adjusted to support this pattern
        await PubSubController.StartMessageProcessing(stoppingToken);
    }
}

8. Run and Test the application

We execute the following command to run the application

dotnet run

We first run the producer application and we send a message to the Google Cloud Pub/Sub Topic

https://github.com/luiscoco/GoogleCloud_Pub_Sub_with_dotNET8_WebAPI_producer

image

image

image

Then we run the receiver application and we receive the message from the Google Cloud Pub/Sub Subscription

image

image

image

We also verify in the Google Cloud the message received. First we have to stop the Receiver application, because the background service is continously pulling the messages

image

image