[BUG] Event Consumer throws an error when deployed in azure cluster

Question

[BUG] Event Consumer throws an error when deployed in azure cluster

PunithkumarJagadees opened this issue 2 years ago · 2 comments

PunithkumarJagadees commented 2 years ago

Describe the bug
https://github.com/Azure-Samples/azure-spring-boot-samples/tree/spring-cloud-azure_v4.2.0/eventhubs/spring-cloud-azure-stream-binder-eventhubs/eventhubs-binder
The code sample present with above link works totally fine in my local using service principal. But it fails to consume while running in azure pipeline.

Exception or Stack Trace

"class":"c.a.s.i.e.i.EventHubsInboundChannelAdapter","message":"Error occurred on partition: NONE. Error: {} com.azure.storage.blob.models.BlobStorageException: If you are using a StorageSharedKeyCredential, and the  
server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair          
'Azure-Storage-Log-String-To-Sign': true to the appropriate method call.nIf you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign 
  with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call.nPlease remember to disable    
  'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII.nStatus code 403, \"<?xml version=\"1.0\" encoding=\"utf-8\"?  
  ><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.nRequestId:ef6b4624-001e-0001-7838-7fce5e000000nTime:2022-06-13T15:14:18.3797615Z</Message></Error>\"n\tat 
  java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:729)n\tat com.azure.core.http.rest.ResponseExceptionConstructorCache.invoke(ResponseExceptionConstructorCache.java:56)n\tat com.azure.core.
  http.rest.RestProxy.instantiateUnexpectedException(RestProxy.java:439)n\tat com.azure.core.http.rest.RestProxy.lambda$ensureExpectedStatus$2(RestProxy.java:393)n\tat reactor.core.publisher.
  FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)n\tat reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2397)n\tat reactor.core.publisher.
  FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:169)n\tat reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2193)n\tat reactor.core.publisher.
  Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2067)n\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)n\tat reactor.core.publisher.MonoJust.
  subscribe(MonoJust.java:54)n\tat reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)n\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)n\tat reactor.core.
  publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)n\tat reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)n\tat reactor.core.publisher.Operators$MonoSubscriber.complete
  (Operators.java:1815)n\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)n\tat reactor.core.publisher.FluxDelaySubscription$DelaySubscriptionMainSubscriber.onNext(FluxDelaySubscription.
  java:188)n\tat reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)n\tat reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)n\tat reactor.core.publisher.
  FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:179)n\tat reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1815)n\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.
  java:151)n\tat reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:284)n\tat reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:187)n\tat reactor.core.
  publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)n\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)n\tat reactor.core.publisher.
  Operators$MonoSubscriber.complete(Operators.java:1815)n\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)n\tat reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.
  java:99)n\tat reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:173)n\tat reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)n\tat 
  reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2663)n\tat reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180)n\tat reactor.core.publisher.
  MonoFlatMapMany$FlatMapManyInner.onComplete(MonoFlatMapMany.java:260)n\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:150)n\tat reactor.core.publisher.
  FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:145)n\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:150)n\tat reactor.core.publisher.
  Operators$MonoSubscriber.complete(Operators.java:1816)n\tat reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159)n\tat reactor.core.publisher.FluxHandle$HandleSubscriber.onComplete
  (FluxHandle.java:212)n\tat reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:269)n\tat reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:401)n\tat reactor.netty.
  channel.ChannelOperations.onInboundComplete(ChannelOperations.java:416)n\tat reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:470)n\tat reactor.netty.http.client.HttpClientOperations.
  onInboundNext(HttpClientOperations.java:685)n\tat reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94)n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead
  (AbstractChannelHandlerContext.java:379)n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)n\tat io.netty.channel.AbstractChannelHandlerContext.
  fireChannelRead(AbstractChannelHandlerContext.java:357)n\tat io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)n\tat datadog.trace.
  instrumentation.netty41.client.HttpClientResponseTracingHandler.channelRead(HttpClientResponseTracingHandler.java:45)n\tat io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.
  java:251)n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead
  (AbstractChannelHandlerContext.java:365)n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)n\tat io.netty.channel.
  CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)n\tat 
  io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)n\tat io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)n\tat io.netty.channel.
  AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)n\tat io.netty.
  channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1368)n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible
  (SslHandler.java:1234)n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1280)n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)n\tat io.netty.
  handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)n\tat io.netty.channel.
  AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)n\tat io.netty.

To Reproduce
Deploy from local to azure pipeline

Code Snippet
https://github.com/Azure-Samples/azure-spring-boot-samples/tree/spring-cloud-azure_v4.2.0/eventhubs/spring-cloud-azure-stream-binder-eventhubs/eventhubs-binder

Expected behavior
Events should be picked by consumer

Screenshots

Setup (please complete the following information):

OS:
IDE:
Sample Path:
Library/Libraries:
Java version:
App Server/Environment:
Frameworks:

Additional context

Information Checklist

Bug Description Added
Repro Steps Added
Setup information Added

Answer 1 · 2022-06-17T02:18:07.000Z

Hi @PunithkumarJagadees , this error happens when your credentials for Storage Blobs is not valid. And as you mentioned it works in your local environment with service principle, could you check the related credentials in your Azure Pipeline is also as expected? Also you could share a sample of your configuration yaml in the Azure pipeline environment.

Answer 2 · 2022-06-17T12:41:30.000Z

@yiliuTo this started working once we gave network access for traffic using azure portal. It was never an issue with code.
Thank you for your support