Unable to get a value from a mirror

Question

Unable to get a value from a mirror

Kazmirchuk opened this issue a year ago · 13 comments

Observed behavior

I have a KV bucket MYKV and its mirror MIRROR_KV that was created as follows:

nats kv add  --history=20 --storage=memory --mirror=MYKV MIRROR_KV

Both buckets exist in the same server and account.

When I put a value into the mirror, I can see in the server log (-DV) that NATS CLI rewrites the subject to the origin bucket as expected:

nats kv put MIRROR_KV key3 val3

[PUB $KV.MYKV.key3 _INBOX.itGcBlv5V82bWxBooq3K4Z.iJvIDoAr 4]
cid:44 - <<- MSG_PAYLOAD: ["val3"]

However, when I try to get a value from the mirror, I get an error "key not found":

nats --trace kv get MIRROR_KV key3

19:25:10 >>> $JS.API.DIRECT.GET.KV_MIRROR_KV.$KV.MIRROR_KV.key3
19:25:10 <<< $JS.API.DIRECT.GET.KV_MIRROR_KV.$KV.MIRROR_KV.key3:
nats: error: nats: key not found

Also, I've checked that MIRROR_KV is in sync with the origin bucket:

nats stream view KV_MIRROR_KV
...
[3] Subject: $KV.MYKV.key3 Received: 2023-11-28T17:14:18+01:00

val3

Expected behavior

I expect to get a key/value from the mirror. NATS CLI should rewrite the subject using the origin bucket name before querying DIRECT.GET

Server and client version

nats --version
0.1.1
nats-server --version
nats-server: v2.10.3

Host environment

No response

Steps to reproduce

No response

Answer 1 · 2023-11-28T18:47:59.000Z

Looking at func mapStreamToKVS in nats.go/jetstream/kv.go, I suspect that nats.go might have the same problem, because kv.pre is not overwritten when info.Config.Mirror exists but m.External == nil. But I don't know Go and don't have Go dev.env. I know Python and tried nats.py, and couldn't do either "put" or "get" via a mirror.

Thanks in advance

Answer 2 · 2023-11-28T19:11:47.000Z

hmm I see ADR-8 recently has received major updates regarding mirrors... should I read that first?

Answer 3 · 2023-11-28T20:40:11.000Z

For mirrors you dont read from them, they are automatically used when reading - the nearest one is used and you cannot use them by name.

If you made a replica by source you could connect to it by name but we have no helpers for that. The only other option is there if you use leafnodes.

Answer 4 · 2023-11-28T20:43:00.000Z

Sorry, correction, nats kv does have --source for source based replicas. Mirror as is should only be used with leafnodes and different domains.

Answer 5 · 2023-12-01T09:58:03.000Z

thank you! so I should bind to a mirror only when its origin is in another domain.

I'm struggling to understand the test func TestKeyValueMirrorDirectGet in nats.go (and tests in other clients are based on it too of course). Both PUTs and GETs are done on the TEST bucket, so how does it prove that the mirror actually responds to DIRECT.GET? I.e. if I remove js.CreateStream from this test, it will nevertheless pass AFAICS.

sorry for off-topic, and probably questions like these belong to Slack. OTOH people can find answers on Github from googling.

Answer 6 · 2023-12-01T10:15:44.000Z

You never can bind to a mirror - its subjects are just wrong. They are there for speeding up reads.

If you make a source based replicas as described in the recent edits to the ADR or using --source in the latest nats release it will be something you can bind to as the messages inside will have been rewritten to have the right subjects.

I suspect the test is just not very good :)

Answer 7 · 2023-12-01T10:21:53.000Z

You never can bind to a mirror

but this is exactly what func TestKeyValueMirrorCrossDomains does! 😮

// Bind locally from leafnode and make sure both get and put work.
mkv, err := ljs.KeyValue(ctx, "MIRROR")

(proceeds with testing Put, Get and WatchAll)

Answer 8 · 2023-12-01T10:23:43.000Z

Cross domains there's some special handling indeed, personally I'd avoid that. I dont want to support that method for a long time.

The source based method is right for all cases now.

Answer 9 · 2023-12-01T10:25:15.000Z

ok, thank you so much for the clarification! I'll send everyone to read the updated ADR-8 :)

Answer 10 · 2023-12-08T17:23:11.000Z

sorry for bothering you again, but do you know if any NATS client has (at least draft) implementation of ADR-8 v1.1, i.e. with the latest additions of KVAggregateConfig + KVAggregateOrigin ? I'd like to have some reference while implementing it. I thought that I might find it in some branch in nats.go, but nope =\

Answer 11 · 2023-12-08T17:31:12.000Z

No, nothing is at 1.1 yet - feature list not complete either.

Try https://github.com/nats-io/jsm.go/compare/main...ripienaar:jsm.go:kv_topologies?expand=1

Answer 12 · 2023-12-11T12:32:01.000Z

Thanks! I've implemented it - quite straightforward and indeed much cleaner than the previous approach with read and write subject prefixes that the client had to figure out based on a mirror configuration.

Do you mind if I provide some feedback about type KVAggregateOrigin? I found it a bit confusing that Stream is mandatory while Bucket is optional, and so users need to remember to prefix the stream name with KV_. What about designing it the other way round, i.e. Bucket is mandatory and Stream is optional (only for the cases when we source from a mirror)? Then users don't need to remember about the "KV_" prefix.

You can see how it looks like here lines 159 (only Bucket) and 198 (both Stream and Bucket). API docs here.

Answer 13 · 2023-12-11T12:34:34.000Z

Thanks for the feedback, will check it out. The code I linked was pretty much just me making sure the spec can be implemented :) but will update there thank you!