[BUG] thin-client fails silently and blocks requests if exception is encountered during initialization

[ZacAttack]

Willingness to contribute

No. I cannot contribute a bug fix at this time.

Venice version

0.4.139

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 20.0): Mariner 5.15.111.1-1.cm2
• JDK version: 17

Describe the problem

If thin client runs into issue during lazy initialization of keySerializer in AbstractAvroStoreClient.getKeySerializerForRequest, it will block all subsequent requests without emitting any metrics. The failure should be as loud as we can get it, maybe even auto remediate if it's possible since the initialization today is lazy.

Tracking information

No response

Code to reproduce bug

No response

What component(s) does this bug affect?

• Controller: This is the control-plane for Venice. Used to create/update/query stores and their metadata.
• Router: This is the stateless query-routing layer for serving read requests.
• Server: This is the component that persists all the store data.
• VenicePushJob: This is the component that pushes derived data from Hadoop to Venice backend.
• VenicePulsarSink: This is a Sink connector for Apache Pulsar that pushes data from Pulsar into Venice.
• Thin Client: This is a stateless client users use to query Venice Router for reading store data.
• Fast Client: This is a stateful client users use to query Venice Server for reading store data.
• Da Vinci Client: This is an embedded, stateful client that materializes store data locally.
• Alpini: This is the framework that fast-client and routers use to route requests to the storage nodes that have the data.
• Samza: This is the library users use to make nearline updates to store data.
• Admin Tool: This is the stand-alone client used for ad-hoc operations on Venice.
• Scripts: These are the various ops scripts in the repo.

[nisargthakkar]

This should be fixed using #582. Closing it