Implement Consumers Pause

[Jarema]

Proposed change

Implement the new API for pausing consumers.

The Server PR with related change has been merged here: nats-io/nats-server#5066
Schemas PR, which is probably the best source of information how to implement the feature:
https://github.com/nats-io/jsm.go/pull/522/files
ADR issue: nats-io/nats-architecture-and-design#266

Details

Request and response schemas:

// io.nats.jetstream.api.v1.consumer_pause_request
type JSApiConsumerPauseRequest struct {
	PauseUntil time.Time `json:"pause_until,omitempty"`
}

// io.nats.jetstream.api.v1.consumer_pause_response
type JSApiConsumerPauseResponse struct {
	JSApiResponse
	Paused         bool          `json:"paused"`
	PauseUntil     time.Time     `json:"pause_until"`
	PauseRemaining time.Duration `json:"pause_remaining,omitempty"`
}

Consumer info includes 2 new fields:

Paused bool json:"paused,omitempty"
PauseRemaining time.Duration json:"pause_remaining,omitempty"

The paused state and time time would need to be persisted to the raft layer such that server restarts would not unpause paused consumers. This is done using the consumer configuration that has a new value:

PauseUntil time.Time json:"pause_until,omitempty"

If in doubt, refer to the Server/Schema PR

Feature be available in the server version 2.11
It's implementation should be hidden behind feature server_2_11.
This feature should be disabled by default, and switch to enabled by default when 2.11 is released.

Hints

Probably the easiest way to implement it, is to look how calls to methods for creating / deleting streams and consumers are done.

Use case

It is difficult to schedule maintenance on central resources on a large distributed system where 100s or 1000s of clients are accessing data in a stream.

This change allows to pause a Consumer such that it appears healthy but just doesnt deliver any messages.

During the pause maintenance can happen and resources accessed by clients will not be under constant pressure, later the stream can be unpaused and work will continue.

This happen without impacting running clients - other than they would see pending messages in stream info but not get any deliveries.

Applies to push and pull consumers.

Contribution

No response