Proposal for Keycloak integration
Opened this issue · 26 comments
I propose adding a Keycloak component to our system for managing authentication and login processes. This integration would be incredibly beneficial, particularly for those using Keycloak, as it would:
- Ensure consistency in different environments.
- Aid in replicating production issues for better troubleshooting.
- Enhance testing capabilities for authentication-related functionalities.
Below is a basic example of how this could be implemented in the program.cs file:
// Example implementation in program.cs
// Adding Keycloak container
var authServer = builder.AddKeyCloakContainer();
// Setting up projects with KeyCloak authentication
var publicSite = builder.AddProject<Projects.PublicSite>().References(authServer);
var postloginSite = builder.AddProject<Projects.PostLoginSite>().References(authServer);
// Additional configurations...@josephaw1022 the APIs your example shows are for hosting, not components. Are you suggesting we add both Aspire.Hosting extensions for Keycloak and a Keycloak component?
@DamianEdwards yes! I am currently working on the prometheus and grafana one right now, but I will have a pr for this eventually soon.
It's worth noting that Keycloak:
- May expose the admin UI separately from the IP
- Open Telemetry protocol can be turned on (but it's off by default)
Do you have an example or how to configure Keycloak to use OpenTelemetry, and specifically export via OTLP? When I looked into this the examples I found seemed to rely on an external collector and/or were setup for Prometheus and Jaeger.
I'll look into the option to expose the admin and service endpoints separately too, that's a good thing to ensure we consider.
Do you have an example or how to configure Keycloak to use OpenTelemetry, and specifically export via OTLP? When I looked into this the examples I found seemed to rely on an external collector and/or were setup for Prometheus and Jaeger.
Not detailed, but I looked into it because I need to go there for some of my customers.
The latest doc tells that OTEL is supported but must be turned on:
https://www.keycloak.org/keycloak-benchmark/kubernetes-guide/latest/util/otel
The KC_OTEL option is mentioned here: https://www.keycloak.org/keycloak-benchmark/kubernetes-guide/latest/customizing-deployment#KC_OTEL
In my own docker file (which does not include OTEL yet) I customize the Keycloak image using other boolean switches:
https://github.com/raffaeler/authentication/blob/main/docs/install.md#preparing-a-custom-image
In theory, you should just turn on OTEL and instructing Prometheus to get the telemetry from KC.
@raffaeler those are the docs I saw too, but they don't appear to actually support setting an OTLP gRPC endpoint to egress to. The doc explicitly says it's not using an OpenTelemetry collector:
@DamianEdwards I read that too, but I assumed the data was accessible using Prometheus, which is one of the options in Aspire.
Can't you just start with the Prometheus support? This could probably be the most popular option.
@raffaeler Prometheus is not enabled by default in Aspire and is a bit heavy for local development. The optimal experience the defaults are assuming is to have resources send OpenTelemetry to the OTLP endpoint hosted by the Aspire dashboard. It seems it would be possible to do this with Keycloak but requires manually installing & configuring the OpenTelemetry Java agent to egress to the OTLP endpoint.
@DamianEdwards I always configure KC with a custom image as the default one misses at lot of important features. It is also needed because you have to add the PKI certificates in an enterprise (or dev) environment.
Adding java stuff to the image should not be a big deal.
If you already found a java configuration that fits the Aspire scenario, it should not be hard to put it all together.
What's the link to the java agent you mentioned? I saw something on GitHub, but I am not sure the they could fit the requirements.
I was thinking of just using the simple jboss/keycloak image and following the pattern of "opinionated but configurable". So you could change the variant and use some custom command argument if you want. I have a really simple example of keycloak running in an aspire project communicating with a postgres db in a repo here
https://github.com/josephaw1022/ScaleStore/blob/master/KEDAScalingUI.AppHost/Program.cs
I have an example running (can share at a later point) that uses the standard Keycloak image customized to import a realm via a binding mount and run using start-dev locally, and start in the publish manifest. We don't currently have support in Aspire for running a container via a dockerfile, so any custom image would have to manually built outside of Aspire before running the app. The standard Keycloak image looks pretty extensible via environment variables and bind mounts/volumes though so I'm inclined to stick to it for anything we add in-box in Aspire.
@raffaeler the only mention of Keycloak with OpenTelemetry I can find is in the Keycloak Benchmark documentation, which is about a helm-chart based configuration. OTel doesn't appear to be supported out of the box by the standard Keycloak container image. To make it work, we'd have to do some digging into how to customize the container to acquire the Java OTel agent and configure it to export via OTLP to the dashboard. If nothing else this should be doable by changing the container entry point to use a mounted shell script that does the needful.
My suggestion is to model our approach after the rabbitmq component. We should clearly state in the documentation which components are and aren't supported and just go from there.
The plan would be to start with the DeFacto existing container image and explicitly mention in the documentation that metrics and telemetry data are not yet available. Whether that image is the quay or jboss one, doesn't matter too much I would think. Just as long as we are getting the newest DeFacto version.
And when a custom image with the appropriately configured Java OTel agent is ready and published to a registry which it sounds like Damian is already looking at, we can replace the initial image in Aspire. Following this, we can update the documentation to include open telemetry information and remove the disclaimer about the absence of components missing once the newly published image is in a good spot.
Would that work? @DamianEdwards @raffaeler
I'm uneasy about using unofficial container images for known services in the official Aspire.Hosting package. To be clear, integration of any new service in Aspire has two separate but related areas:
Aspire.Hostingintegration, i.e. ability to spin up a Keycloak container when launching the AppHost project and have Keycloak details emitted in the publish manifest- An Aspire Component, i.e. a NuGet package that integrates a client library with the Aspire defaults
Regarding Keycloak, I'm mostly interested in the first (hosting integration), as using Keycloak as an IDP from ASP.NET Core apps is already covered via the standard Open ID Connect and JWT Bearer Token support ASP.NET Core comes with.
All that said, we could indeed add support for Keycloak to Aspire.Hosting just using the official container from quay.io to start with, and folks can customize it using the various extensibility points it has via environment variables and bind mounts/volumes. At any point later we could update that to support having Keycloak emit OTLP to the dashboard too. I don't think is a high priority for v1 of Aspire though, as such integration can be done by anyone with very little code and shipped in a standalone package. I'm not against it though.
The example of this I've been experimenting with is here.
Thank you both.
I am ok with the approach proposed by @DamianEdwards, it makes sense. I will try to set up the Java OTLP client to see how far I can go. Said that I've never been able to use the default KC image for anything. For example in development you have to add the certificates for both oAuth crypto stuff and Java certificate storage. I've documented this in my repo and I have a more verbose dockerfile for more complex scenarios.
Also, even if KC is interfaced using OIDC to ASP.NET, there are plenty of settings (beyond URLs) that are very different depending on the scenario (development, test and production).
I am still not sure how to model these differences in Aspire so that I can avoid custom variables to switch from a configuration/behavior to another.
What's the best way to do this in absence of the Aspire Component package?
For example in development you have to add the certificates for both oAuth crypto stuff and Java certificate storage.
This could be achieved with a binding mount and custom entry point shell script without requiring a Dockerfile though, right? Similar to what I do in this example here. That said, having support in Aspire.Hosting to add a resource via a Dockerfile to simplify using custom containers seems like a reasonable suggestion. Feel free to log that issue 😄
RE changing configuration between environments, that's just "standard" ASP.NET Core environment aware configuration and doesn't require an Aspire Component package. As to modeling that in the AppHost project, we're working on that support right now for preview.4 so that e.g. you'll be able to easily run against a Keycloak container in development but point to an existing instance in the publish manifest.
This could be achieved with a binding mount and custom entry point shell script without requiring a Dockerfile though, right?
It can be done for the crypto/oAuth certs, but unfortunately not for the Java keystore. AFAIK it cannot be moved out to a mount or at least I tried and it didn't work.
That said, having support in Aspire.Hosting to add a resource via a Dockerfile to simplify using custom containers seems like a reasonable suggestion. Feel free to log that issue
Done #1852 :-)
RE changing configuration between environments, that's just "standard" ASP.NET Core environment aware configuration and doesn't require an Aspire Component package. As to modeling that in the AppHost project, we're working on that support right now for preview.4 so that e.g. you'll be able to easily run against a Keycloak container in development but point to an existing instance in the publish manifest.
Sure, I know the config environments. I was referring to model that in AppHost. Great to know it's coming in a future preview.
It can be done for the crypto/oAuth certs, but unfortunately not for the Java keystore. AFAIK it cannot be moved out to a mount or at least I tried and it didn't work.
Can always copy the keystore file from the bind location to the required place in the image on startup before starting Keycloak though right? Or am I misunderstanding the scenario? Looking at your example what about that couldn't be done in a custom container entry point script (for dev time) assuming the certificate files were in a bind mount? I'm thinking we could actually leverage the ASP.NET Core dev certificate to make this fairly seamless for local dev, as it gets installed and trusted as part of setting up the .NET SDK/VS.
not to get carried away with what ifs, but I think it would be cool if you could pass in a db resource as a parameter argument and have it set up the db environment variables for you. so
var pg = builder.AddPostgres("pgserver");
var sqlserver = builder.AddSqlServer("sql-server");
var authServer = builder.AddKeyCloakContainer("keycloak", pg);
// or var authServer = builder.AddKeyCloakContainer("keycloak", sqlserver);
I mean we can just stick to using environment variables for full control, but this could be some shorthand syntax to speed up the process of configuring keycloak utilizing a db.
I believe we could do mysql, postgres, sql-server, and maybe Oracle? But I know the first 3 for sure. And I am sure there are some other ones as well that I am not thinking of
Can always copy the keystore file from the bind location to the required place in the image on startup before starting Keycloak though right? Or am I misunderstanding the scenario? Looking at your example what about that couldn't be done in a custom container entry point script (for dev time) assuming the certificate files were in a bind mount? I'm thinking we could actually leverage the ASP.NET Core dev certificate to make this fairly seamless for local dev, as it gets installed and trusted as part of setting up the .NET SDK/VS.
When I first tried to move the Java keystore out of the default location, it didn't work. But I may have missed something.
After all, for me it was not a big deal creating a custom image which you have to do to import / export data when migrating from one version to another.
I mean we can just stick to using environment variables for full control, but this could be some shorthand syntax to speed up the process of configuring keycloak utilizing a db.
It is hardly applicable, there are so many options... I would stick to the most versatile solution.
I am afraid this could result in a deno-only solution.
Here is an example of how to run Keycloak and Aspire:
Docs: https://nikiforovall.github.io/keycloak-authorization-services-dotnet/devex/aspire.html#aspire-support
Blog: https://nikiforovall.github.io/dotnet/keycloak/2024/06/02/aspire-support-for-keycloak.html
Note: It is not production-ready and favors local development.
This is awesome!!! Really really good stuff.
I think the latest PR out for this (#4289) is enough for an initial version of this resource (assuming the WithDataVolume method works well).
Some ideas for further investigation in the future:
- I'm keen to explore @josephaw1022's comment about adding support for configuring Keycloak database storage, but via
WithReferencerather than an argument to theAddKeycloakcall (to keep it aligned with the established patterns). Packaging will be interesting for this as these methods would require a dependency on the various database resource hosting packages, e.g.WithReference(this IResourceBuilder<KeycloakResource> keycloak, IResourceBuilder<PostgresDatabaseResource> postgres) - Configuring an ASP.NET Core application to use Keycloak for authentication via OIDC is still kinda involved though so it would be worthwhile investigating what a Keycloak OIDC component might look like
- Ability to add realms, clients, users, etc. to the Keycloak instance as part of the resource configuration. This could be implemented via calling out to the admin CLI (perhaps we include it in the package?) or by directly making calls to the Admin REST API. Both approaches would require coordination with the lifetime of the container of course. We could also consider adding methods to easily import realms from an exported realms JSON file. Note these options really only make sense in dev mode.
- Configuring the Keycloak instance to use HTTPS during local development via the ASP.NET Core developer HTTPS certificate
- I'm keen to explore @josephaw1022's comment about adding support for configuring Keycloak database storage, but via
WithReferencerather than an argument to theAddKeycloakcall (to keep it aligned with the established patterns). Packaging will be interesting for this as these methods would require a dependency on the various database resource hosting packages, e.g.WithReference(this IResourceBuilder<KeycloakResource> keycloak, IResourceBuilder<PostgresDatabaseResource> postgres)
I experimented with this idea, and was able to implement a simple extension as you suggested:
public static IResourceBuilder<KeycloakResource> WithReference(this IResourceBuilder<KeycloakResource> builder, IResourceBuilder<PostgresServerResource> source)
{
var resource = source.Resource;
return builder.WithEnvironment(context =>
{
var primaryEndpoint = resource.PrimaryEndpoint;
context.EnvironmentVariables["KC_DB"] = "postgres";
context.EnvironmentVariables["KC_DB_URL_HOST"] = primaryEndpoint.ContainerHost;
context.EnvironmentVariables["KC_DB_URL_PORT"] = ReferenceExpression.Create($"{primaryEndpoint.Property(EndpointProperty.Port)}");
context.EnvironmentVariables["KC_DB_URL_DATABASE"] = "postgres"; // Keycloak does not create the database by itself. Using the default database for the time being.
context.EnvironmentVariables["KC_DB_SCHEMA"] = "public"; // Keycloak does not create the schema by itself. Using default schema for the time being.
context.EnvironmentVariables["KC_DB_USERNAME"] = (object?)resource.UserNameParameter ?? "postgres";
context.EnvironmentVariables["KC_DB_PASSWORD"] = resource.PasswordParameter;
});
}This extension, used together with the linked PR, allows the Keycloak container to startup using the PostgreSQL container.
Configuring the Keycloak instance to use HTTPS during local development via the ASP.NET Core developer HTTPS certificate
There's an implementation of this in the sample I'm working on for Keycloak over at https://github.com/dotnet/aspire-samples/blob/b741f5e78a86539bc9ab12cd7f4a5afea7aa54c4/samples/Keycloak/Keycloak.AppHost/KeycloakExtensions.cs#L16
We have a keycloak integration package now https://learn.microsoft.com/en-us/dotnet/aspire/authentication/keycloak-integration?tabs=dotnet-cli
It's very lightweight at the moment.