Updating HTTP server configuration on the fly (ie. TLS settings)

[dimitribouniol]

I would like to dynamically update some of the HTTP server's configuration on the fly, specifically TLS settings like with an updated certificate, but without bringing down the server and dropping all clients. It seems like this would be straightforward to implement since the HTTPServerConnection configures child handlers as they come in, so this configuration could be dynamic, but I had some questions and wanted to make sure the idea jived with the project before pushing a PR 😅

Details

Unfortunately, Application.http.server.configuration becomes read-only once a server starts, despite some configuration options still being configurable for new connections or other non-server-start tasks:

• responseCompression
• requestDecompression
• supportPipelining
• supportVersions
• tlsConfiguration
• serverName
• reportMetrics
• logger
• shutdownTimeout

Additionally, this configuration is cached by HTTPServerConnection's various closures, which would need to be updated as well.

I propose adding the following public API to HTTPServer to allow updating safe configuration options dynamically:

public func updateConfiguration(
    responseCompression: CompressionConfiguration? = nil,
    requestDecompression: DecompressionConfiguration? = nil,
    supportPipelining: Bool? = nil,
    supportVersions: Set<HTTPVersionMajor>?? = nil,
    tlsConfiguration: TLSConfiguration?? = nil,
    serverName: String?? = nil,
    reportMetrics: Bool? = nil,
    logger: Logger?? = nil,
    shutdownTimeout: TimeAmount?
) {
    var updatedConfiguration = _configuration.withLockedValue { $0 }
    
    if let responseCompression { updatedConfiguration.responseCompression = responseCompression }
    if let requestDecompression { updatedConfiguration.requestDecompression = requestDecompression }
    if let supportPipelining { updatedConfiguration.supportPipelining = supportPipelining }
    if let supportVersions { updatedConfiguration.supportVersions = supportVersions }
    if let tlsConfiguration { updatedConfiguration.tlsConfiguration = tlsConfiguration }
    if let serverName { updatedConfiguration.serverName = serverName }
    if let reportMetrics { updatedConfiguration.reportMetrics = reportMetrics }
    if let logger { updatedConfiguration.logger = logger }
    if let shutdownTimeout { updatedConfiguration.shutdownTimeout = shutdownTimeout }

    self.application.storage[Application.HTTP.Server.ConfigurationKey.self] = newValue
    _configuration.withLockedValue { $0 = newValue }
}

This would allow any of the safe options to be re-specified individually.

Then, HTTPServerConnection can fetch the configuration on-demand in its .childChannelInitializer {} closure:

    static func start(
        application: Application,
        server: HTTPServer,
        responder: Responder,
        configuration: HTTPServer.Configuration,
        on eventLoopGroup: EventLoopGroup
    ) -> EventLoopFuture<HTTPServerConnection> {
        let quiesce = ServerQuiescingHelper(group: eventLoopGroup)
        let bootstrap = ServerBootstrap(group: eventLoopGroup)
            // Specify backlog and enable SO_REUSEADDR for the server itself
            .serverChannelOption(ChannelOptions.backlog, value: Int32(configuration.backlog))
            .serverChannelOption(ChannelOptions.socket(SocketOptionLevel(SOL_SOCKET), SO_REUSEADDR), value: configuration.reuseAddress ? SocketOptionValue(1) : SocketOptionValue(0))
            
            // Set handlers that are applied to the Server's channel
            .serverChannelInitializer { channel in
                channel.pipeline.addHandler(quiesce.makeServerChannelHandler(channel: channel))
            }
            
            // Set the handlers that are applied to the accepted Channels
            .childChannelInitializer { [unowned application, unowned server] channel in
                let configuration = server.configuration
                // add TLS handlers if configured
                if var tlsConfiguration = configuration.tlsConfiguration {

From that point onwards, the in-scope configuration will be used for that child connection until it is closed, but new connections would grab a fresh configuration struct as needed.

Unfortunately, this introduces a few double optionals, which isn't great. An alternative would be to offer a dynamicConfiguration property on HTTPServer which would be a safe view into the underlying configuration, but that would instead require another type to be maintained along-side the main configuration type.

A final alternative could be to only warn in the configuration setter if an unsafe property were changed:

    public var configuration: Configuration {
        get { _configuration.withLockedValue { $0 } }
        set {
            let oldValue = _configuration.withLockedValue { $0 }
            var canBeUpdatedDynamically = true
            
            if oldValue.address != newValue.address
                || oldValue.backlog != newValue.backlog
                || oldValue.reuseAddress != newValue.reuseAddress
                || oldValue.tcpNoDelay != newValue.tcpNoDelay {
                canBeUpdatedDynamically = false
            }
            
            guard canBeUpdatedDynamically || !didStart.withLockedValue({ $0 }) else {
                _configuration.withLockedValue({ $0 }).logger.warning("Cannot modify server configuration after server has been started.")
                return
            }
            self.application.storage[Application.HTTP.Server.ConfigurationKey.self] = newValue
            _configuration.withLockedValue { $0 = newValue }
        }
    }

… but this version would be less permissive to forgetting to add a new configuration to the unsafe list, though the only downside would be not seeing the warning and not getting the desired effect, so maybe that's not too bad?

Anywho, curious to hear which we would prefer. In the meantime I'll fork and work on a POC.

[0xTim]

Just as a heads up the start function is next inline to undergo some major changes as we migrate to an async start and NIOAsyncChannel

[dimitribouniol]

Would there be any issues getting this in before that rewrite (I'll probably have a PR in a day or two)? Seems like NIOAsyncChannel still would take a childChannelInitializer, so the assumption that we could refresh the configuration at that point in time still stands:

    @available(macOS 10.15, iOS 13, tvOS 13, watchOS 6, *)
    public func bind<Output: Sendable>(
        _ socket: NIOBSDSocket.Handle,
        cleanupExistingSocketFile: Bool = false,
        serverBackPressureStrategy: NIOAsyncSequenceProducerBackPressureStrategies.HighLowWatermark? = nil,
        childChannelInitializer: @escaping @Sendable (Channel) -> EventLoopFuture<Output>
    ) async throws -> NIOAsyncChannel<Output, Never> 

[dimitribouniol]

I ended up going for the 3rd approach I outlined, and I think it works the best in practice. Implemented and tested in #3132