Azure Stream analytics on Edge has issues when I send data to the Azure SQL server PaaS database.

Question

Azure Stream analytics on Edge has issues when I send data to the Azure SQL server PaaS database.

abbaasalif opened this issue 3 years ago · 1 comments

My solution involves using the Azure Stream analytics on the Edge to send data to the Azure SQL database but I am seeing that my node is getting killed and throwing an exception which is causing some issues.

I have the same setup on the ASA cloud job which works perfectly fine.

I have also configured a BrokeredEndpoint required to send a telemetry data from my downstream device to the IOT edge hub.

Please advise and I am willing to provide further information.

I am pasting my ASA edge logs here:

2021-07-06 07:50:43.969 +00:00 [INF] - ASA module registered message endpoint : [temperature]
2021-07-06 07:50:45.679 +00:00 [INF] - Module client initializing ... 
2021-07-06 07:50:46.611 +00:00 [INF] - Opened module client connection.
2021-07-06 07:50:46.613 +00:00 [INF] - Setting Product Info Id as Microsoft.stream-analytics-on-iot-edge.
2021-07-06 07:50:46.863 +00:00 [INF] - ASA getting path: https://csg10032000aa0677c7.blob.core.windows.net/asa-edge-ba36438e-7072-410b-9718-41f1233030c1/ASAEdgeJobs/ba36438e-7072-410b-9718-41f1233030c1/06a1164c-11c6-4e2d-96a7-8d1ac305e67c/ASAEdgeJobDefinition.zip?sv=2018-03-28&sr=b&sig=eUTkbf%2Fa6q9diMOtUm7%2F1cWlNY6nq410dV9mOsVVuVA%3D&st=2021-07-06T07%3A29%3A07Z&se=2024-07-06T07%3A39%3A07Z&sp=r
2021-07-06 07:50:46.911 +00:00 [INF] - Download ASA Job Package ...
2021-07-06 07:50:47.510 +00:00 [INF] - Unzip /tmp/e87b217f-b142-40cf-8e49-15f1f62410a0/ASAEdgeJobDefinition.zip ...
2021-07-06 07:50:47.774 +00:00 [INF] - Read job definition from /tmp/6a7d1bfe-28cc-492f-9422-fd7183f24484/EdgeJobDefinition.txt.
2021-07-06 07:50:47.777 +00:00 [INF] - Read job configuration from /tmp/6a7d1bfe-28cc-492f-9422-fd7183f24484/EdgeJobConfiguration.txt.
2021-07-06 07:50:47.781 +00:00 [INF] - Initalizing ASA Engine ...
2021-07-06 07:50:47.781 +00:00 [INF] - The directory for the executing assembly is /app.
2021-07-06 07:50:47.781 +00:00 [INF] - The directory for the generated query is /tmp/6a7d1bfe-28cc-492f-9422-fd7183f24484/QueryBinaries.
2021-07-06 07:50:47.781 +00:00 [INF] - Load codegen dll :6a7d1bfe-28cc-492f-9422-fd7183f24484__GeneratedQueryCode__.
2021-07-06 07:50:47.782 +00:00 [INF] - Setting assembly resolver for query runtime binaries. Folder location is :/tmp/6a7d1bfe-28cc-492f-9422-fd7183f24484/QueryBinaries.
2021-07-06 07:50:48.322 +00:00 [INF] - 
=============================Metrics report=============================
 LateInputEvents                              :                   0
 InputEventsSources                           :                   0
 OutputEvents                                 :                   0
 DeserializationError                         :                   0
 InputEvents                                  :                   0
 ConversionErrors                             :                   0
 DroppedOrAdjustedEvents                      :                   0
 OutputWatermarkDelaySeconds                  :                 N/A
 Errors                                       :                   0
 InputEventBytes                              :                 0 B
 EarlyInputEvents                             :                   0
========================================================================

2021-07-06 07:50:49.044 +00:00 [INF] - ASA module registered message endpoint : [temperature]
2021-07-06 07:50:53.254 +00:00 [WRN] - [ASA Warning-"FxWarn"] ["streamingNode0.sql-sql#0$0", "M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\BasePartitionController.ProcessBatch", "0115", "Output adapter has encountered an error while initializing.", "OutputAdapterError\UserActionableException", "Exception: Microsoft.Streaming.Diagnostics.Exceptions.UserActionableException
Message: Cannot initialize SQL Database output with database info: SQL Server Azure instance 'iotdbtemp.database.windows.net', table [iot].[[dbo].[Temperature]].
StackTrace:   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.ExecutionPrep() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 289
   at Microsoft.Streaming.Processors.OutputAdapters.OutputAdapterBase.PrepareExecution() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\OutputAdapterBase.cs:line 207
InnerException
Exception: System.InvalidOperationException
Message: Internal connection fatal error.
StackTrace:   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.DrainData(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.ThrowUnsupportedCollationEncountered(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.GetCodePage(SqlCollation collation, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.TryCommonProcessMetaData(TdsParserStateObject stateObj, _SqlMetaData col)
   at System.Data.SqlClient.TdsParser.TryProcessMetaData(Int32 cColumns, TdsParserStateObject stateObj, _SqlMetaDataSet& metaData)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.BuildBulkCopyOptions(SqlConnection connection, String tableName) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 169
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.CreateSqlBulkCopyAsync(Boolean recreateConnection) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 971
", "1.2.919.1"]
2021-07-06 07:51:03.301 +00:00 [INF] - [ASA Info-"AzureUserEventSource"] ["", 2, "", "06a1164c-11c6-4e2d-96a7-8d1ac305e67c", "", "8500fb21-2f8d-42f4-9d9d-15dad7c8ba22", "8500fb21-2f8d-42f4-9d9d-15dad7c8ba22", "streamingNode0", "Microsoft.StreamAnalytics", "2021-07-06T07:51:03.2912033Z", "", 2, "bd2f3a2e-872d-413e-bb49-36c079c9e583", "Initialize Adapter:  Failed to initialize output adapter SQL", "{\"Message Time\":\"2021-07-06 07:51:03Z\",\"ErrorCategory\":\"ExternalError\",\"ErrorCode\":\"SQLDatabaseOutputInitializationError\",\"Message\":\"Cannot initialize SQL Database output with database info: SQL Server Azure instance 'iotdbtemp.database.windows.net', table [iot].[[dbo].[Temperature]]. [OutputSourceAlias:SQL;AdapterType:SQL;ShardId:0;]\",\"Type\":\"SqlOutputAdapterInitializationFailure\",\"Correlation ID\":\"06a1164c-11c6-4e2d-96a7-8d1ac305e67c\",\"Error\":\"- Internal connection fatal error.\r\n\"}", "IoT", "Microsoft.StreamAnalytics", "/subscriptions/1d94a983-1ce4-47c9-9592-7e0aaee29d0a/resourceGroups/IoT/providers/Microsoft.StreamAnalytics/streamingjobs/local_stream", "Failed", "1d94a983-1ce4-47c9-9592-7e0aaee29d0a", ""]
2021-07-06 07:51:03.302 +00:00 [INF] - [ASA Info-"AzureUserEventSource"] ["streamingNode0", 07/06/2021 07:51:03, "/SUBSCRIPTIONS/1D94A983-1CE4-47C9-9592-7E0AAEE29D0A/RESOURCEGROUPS/IOT/PROVIDERS/MICROSOFT.STREAMANALYTICS/STREAMINGJOBS/LOCAL_STREAM", "Initialize Adapter:  Failed to initialize output adapter SQL", "Execution", "Failed", "Error", "{\"ErrorCategory\":\"ExternalError\",\"ErrorCode\":\"SQLDatabaseOutputInitializationError\",\"Message\":\"Cannot initialize SQL Database output with database info: SQL Server Azure instance 'iotdbtemp.database.windows.net', table [iot].[[dbo].[Temperature]]. [OutputSourceAlias:SQL;AdapterType:SQL;ShardId:0;]\",\"Type\":\"SqlOutputAdapterInitializationFailure\",\"Correlation ID\":\"06a1164c-11c6-4e2d-96a7-8d1ac305e67c\",\"Error\":\"- Internal connection fatal error.\r\n\"}"]
2021-07-06 07:51:03.305 +00:00 [ERR] - [ASA Error-"FxErr"] ["streamingNode0", "", "", "Cannot initialize SQL Database output with database info: SQL Server Azure instance 'iotdbtemp.database.windows.net', table [iot].[[dbo].[Temperature]]., - Internal connection fatal error.
, SQL Server Azure instance 'iotdbtemp.database.windows.net', table [iot].[[dbo].[Temperature]] - Failed to initialize. , message: System.InvalidOperationException: Internal connection fatal error.
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.DrainData(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.ThrowUnsupportedCollationEncountered(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.GetCodePage(SqlCollation collation, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.TryCommonProcessMetaData(TdsParserStateObject stateObj, _SqlMetaData col)
   at System.Data.SqlClient.TdsParser.TryProcessMetaData(Int32 cColumns, TdsParserStateObject stateObj, _SqlMetaDataSet& metaData)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.BuildBulkCopyOptions(SqlConnection connection, String tableName) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 169
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.CreateSqlBulkCopyAsync(Boolean recreateConnection) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 971, : OutputSourceAlias:SQL;AdapterType:SQL;ShardId:0;;PhysicalTargetListAsString:;PhysicalName:sql-sql#0$0;", "SqlOutputAdapterInitializationFailure\UserActionableException", "Exception: Microsoft.Streaming.Diagnostics.Exceptions.UserActionableException
Message: Cannot initialize SQL Database output with database info: SQL Server Azure instance 'iotdbtemp.database.windows.net', table [iot].[[dbo].[Temperature]].
StackTrace:   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.ExecutionPrep() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 289
   at Microsoft.Streaming.Processors.OutputAdapters.OutputAdapterBase.PrepareExecution() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\OutputAdapterBase.cs:line 207
   at Microsoft.Streaming.Processors.OutputProcessor.BasePartitionController.CreateAdapter(OutputAdapterConfiguration adapterConfiguration) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\BasePartitionController.cs:line 401
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.Streaming.Processors.OutputProcessor.SingleAdapterPartitionController.ProcessOutputBatches(List`1 outputBatches) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\SingleAdapterPartitionController.cs:line 103
   at Microsoft.Streaming.Processors.OutputProcessor.BasePartitionController.ProcessBatch(StreamingBatch streamEvents) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\BasePartitionController.cs:line 140
   at Microsoft.Streaming.Processors.OutputProcessor.ProcessBatchExecutor.AccumulateBatch(StreamingBatch batch, StreamingBatch& accumulatedBatch) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\ProcessBatchExecutor.cs:line 266
   at Microsoft.Streaming.Processors.OutputProcessor.OutputProcessor.ProcessBatchAsync(StreamingBatch batch, String streamId, Boolean dropOutput, Boolean completeUpTo) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\OutputProcessor.cs:line 498
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.CreateProcessingTask(StreamingBatch batch, String streamId, IAnchor anchor, Boolean completeUpTo) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 807
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.<>c__DisplayClass96_0.<Process>b__0() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 790
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.Process(StreamingBatch batch, String streamId, RequestContext requestContext, TimeSpan getBatchTime, IAnchor anchor, Boolean completeUpTo) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 800
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.ProcessBatch(Boolean async) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 378
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.Process(Boolean isInitialization) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 504
   at Microsoft.Streaming.Processors.OutputProcessor.OutputProcessor.Process(Boolean isInitialization) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\OutputProcessor.cs:line 156
   at Microsoft.Streaming.StreamingNode.ProcessorScheduler.Execute() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorScheduler.cs:line 557
   at Microsoft.Streaming.StreamingNode.ProcessorScheduler.ThreadProc() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorScheduler.cs:line 661
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Streaming.StreamingNode.StreamingNode.WaitForExit(CancellationToken cancellationToken) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\StreamingNode.cs:line 676
   at Microsoft.Streaming.StreamingNode.StreamingNode.StartNode(CancellationToken initializationCancellationToken) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\StreamingNode.cs:line 292
InnerException
Exception: System.InvalidOperationException
Message: Internal connection fatal error.
StackTrace:   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.DrainData(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.ThrowUnsupportedCollationEncountered(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.GetCodePage(SqlCollation collation, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.TryCommonProcessMetaData(TdsParserStateObject stateObj, _SqlMetaData col)
   at System.Data.SqlClient.TdsParser.TryProcessMetaData(Int32 cColumns, TdsParserStateObject stateObj, _SqlMetaDataSet& metaData)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.BuildBulkCopyOptions(SqlConnection connection, String tableName) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 169
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.CreateSqlBulkCopyAsync(Boolean recreateConnection) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 971
", "1.2.919.1"]
2021-07-06 07:51:03.307 +00:00 [ERR] - Exception: IsFatal: False; Microsoft.Streaming.Diagnostics.Exceptions.UserActionableException: Cannot initialize SQL Database output with database info: SQL Server Azure instance 'iotdbtemp.database.windows.net', table [iot].[[dbo].[Temperature]]. ---> System.InvalidOperationException: Internal connection fatal error.
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.DrainData(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.ThrowUnsupportedCollationEncountered(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.GetCodePage(SqlCollation collation, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.TryCommonProcessMetaData(TdsParserStateObject stateObj, _SqlMetaData col)
   at System.Data.SqlClient.TdsParser.TryProcessMetaData(Int32 cColumns, TdsParserStateObject stateObj, _SqlMetaDataSet& metaData)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.BuildBulkCopyOptions(SqlConnection connection, String tableName) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 169
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.CreateSqlBulkCopyAsync(Boolean recreateConnection) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 971
   --- End of inner exception stack trace ---
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.ExecutionPrep() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 289
   at Microsoft.Streaming.Processors.OutputAdapters.OutputAdapterBase.PrepareExecution() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\OutputAdapterBase.cs:line 207
   at Microsoft.Streaming.Processors.OutputProcessor.BasePartitionController.CreateAdapter(OutputAdapterConfiguration adapterConfiguration) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\BasePartitionController.cs:line 401
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.Streaming.Processors.OutputProcessor.SingleAdapterPartitionController.ProcessOutputBatches(List`1 outputBatches) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\SingleAdapterPartitionController.cs:line 103
   at Microsoft.Streaming.Processors.OutputProcessor.BasePartitionController.ProcessBatch(StreamingBatch streamEvents) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\BasePartitionController.cs:line 140
   at Microsoft.Streaming.Processors.OutputProcessor.ProcessBatchExecutor.AccumulateBatch(StreamingBatch batch, StreamingBatch& accumulatedBatch) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\ProcessBatchExecutor.cs:line 266
   at Microsoft.Streaming.Processors.OutputProcessor.OutputProcessor.ProcessBatchAsync(StreamingBatch batch, String streamId, Boolean dropOutput, Boolean completeUpTo) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\OutputProcessor.cs:line 498
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.CreateProcessingTask(StreamingBatch batch, String streamId, IAnchor anchor, Boolean completeUpTo) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 807
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.<>c__DisplayClass96_0.<Process>b__0() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 790
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.Process(StreamingBatch batch, String streamId, RequestContext requestContext, TimeSpan getBatchTime, IAnchor anchor, Boolean completeUpTo) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 800
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.ProcessBatch(Boolean async) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 378
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.Process(Boolean isInitialization) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 504
   at Microsoft.Streaming.Processors.OutputProcessor.OutputProcessor.Process(Boolean isInitialization) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\OutputProcessor.cs:line 156
   at Microsoft.Streaming.StreamingNode.ProcessorScheduler.Execute() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorScheduler.cs:line 557
   at Microsoft.Streaming.StreamingNode.ProcessorScheduler.ThreadProc() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorScheduler.cs:line 661
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Streaming.StreamingNode.StreamingNode.WaitForExit(CancellationToken cancellationToken) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\StreamingNode.cs:line 676
   at Microsoft.Streaming.StreamingNode.StreamingNode.StartNode(CancellationToken initializationCancellationToken) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\StreamingNode.cs:line 292
   at Microsoft.Streaming.Runtime.Container.Edge.Container.RunNodeAndHandleExceptions(String jobId, CancellationToken token) in M:\ASAFXVS19-004\_work\5\s\Product\Source\DotNetCore\Container.Edge\Container.cs:line 500; InnerException: System.InvalidOperationException: Internal connection fatal error.
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.DrainData(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.ThrowUnsupportedCollationEncountered(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.GetCodePage(SqlCollation collation, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.TryCommonProcessMetaData(TdsParserStateObject stateObj, _SqlMetaData col)
   at System.Data.SqlClient.TdsParser.TryProcessMetaData(Int32 cColumns, TdsParserStateObject stateObj, _SqlMetaDataSet& metaData)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.BuildBulkCopyOptions(SqlConnection connection, String tableName) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 169
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.CreateSqlBulkCopyAsync(Boolean recreateConnection) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 971
Microsoft.Streaming.Diagnostics.Exceptions.UserActionableException: Cannot initialize SQL Database output with database info: SQL Server Azure instance 'iotdbtemp.database.windows.net', table [iot].[[dbo].[Temperature]]. ---> System.InvalidOperationException: Internal connection fatal error.
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.DrainData(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.ThrowUnsupportedCollationEncountered(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.GetCodePage(SqlCollation collation, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.TryCommonProcessMetaData(TdsParserStateObject stateObj, _SqlMetaData col)
   at System.Data.SqlClient.TdsParser.TryProcessMetaData(Int32 cColumns, TdsParserStateObject stateObj, _SqlMetaDataSet& metaData)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.BuildBulkCopyOptions(SqlConnection connection, String tableName) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 169
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.CreateSqlBulkCopyAsync(Boolean recreateConnection) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 971
   --- End of inner exception stack trace ---
   at Microsoft.Streaming.Processors.OutputAdapters.SqlOutputAdapter.ExecutionPrep() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\SQL\SqlOutputAdapter.cs:line 289
   at Microsoft.Streaming.Processors.OutputAdapters.OutputAdapterBase.PrepareExecution() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputAdapters\Adapters\OutputAdapterBase.cs:line 207
   at Microsoft.Streaming.Processors.OutputProcessor.BasePartitionController.CreateAdapter(OutputAdapterConfiguration adapterConfiguration) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\BasePartitionController.cs:line 401
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.Streaming.Processors.OutputProcessor.SingleAdapterPartitionController.ProcessOutputBatches(List`1 outputBatches) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\SingleAdapterPartitionController.cs:line 103
   at Microsoft.Streaming.Processors.OutputProcessor.BasePartitionController.ProcessBatch(StreamingBatch streamEvents) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\BasePartitionController.cs:line 140
   at Microsoft.Streaming.Processors.OutputProcessor.ProcessBatchExecutor.AccumulateBatch(StreamingBatch batch, StreamingBatch& accumulatedBatch) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\ProcessBatchExecutor.cs:line 266
   at Microsoft.Streaming.Processors.OutputProcessor.OutputProcessor.ProcessBatchAsync(StreamingBatch batch, String streamId, Boolean dropOutput, Boolean completeUpTo) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\OutputProcessor.cs:line 498
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.CreateProcessingTask(StreamingBatch batch, String streamId, IAnchor anchor, Boolean completeUpTo) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 807
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.<>c__DisplayClass96_0.<Process>b__0() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 790
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.Process(StreamingBatch batch, String streamId, RequestContext requestContext, TimeSpan getBatchTime, IAnchor anchor, Boolean completeUpTo) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 800
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.ProcessBatch(Boolean async) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 378
   at Microsoft.Streaming.StreamingNode.ProcessorsCommon.BaseStreamingProcessor.Process(Boolean isInitialization) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorsCommon\BaseStreamingProcessor.cs:line 504
   at Microsoft.Streaming.Processors.OutputProcessor.OutputProcessor.Process(Boolean isInitialization) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\OutputProcessor\OutputProcessor.cs:line 156
   at Microsoft.Streaming.StreamingNode.ProcessorScheduler.Execute() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorScheduler.cs:line 557
   at Microsoft.Streaming.StreamingNode.ProcessorScheduler.ThreadProc() in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\ProcessorScheduler.cs:line 661
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Streaming.StreamingNode.StreamingNode.WaitForExit(CancellationToken cancellationToken) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\StreamingNode.cs:line 676
   at Microsoft.Streaming.StreamingNode.StreamingNode.StartNode(CancellationToken initializationCancellationToken) in M:\ASAFXVS19-004\_work\5\s\Product\Source\Framework\StreamingNode\StreamingNode.cs:line 292
   at Microsoft.Streaming.Runtime.Container.Edge.Container.RunNodeAndHandleExceptions(String jobId, CancellationToken token) in M:\ASAFXVS19-004\_work\5\s\Product\Source\DotNetCore\Container.Edge\Container.cs:line 500
2021-07-06 07:51:03.340 +00:00 [INF] - Metrics is Cancelled.
2021-07-06 07:51:03.340 +00:00 [INF] - Node Exited.
2021-07-06 07:51:03.340 +00:00 [INF] - Job fatal exception caught. Failing job.
2021-07-06 07:51:03.340 +00:00 [INF] - Will restart engine in one minute.

Answer 1 · 2021-07-07T07:01:36.000Z

Closing this here as it is irrelevant. I have raised the same in stream analytics github repo