Deadlock in in OracleConnectionDispenser.ProcessInstanceUp()

Question

Deadlock in in OracleConnectionDispenser.ProcessInstanceUp()

Closed this issue a month ago · 4 comments

Hi,

we're at oracle.manageddataaccess 21.14.0 and just experienced a deadlock during parallel OracleConnection.Open(). The stacks of three threads:

win32u.dll!NtUserMsgWaitForMultipleObjectsEx() + 20 bytes	Unknown	No symbols loaded.
user32.dll!MsgWaitForMultipleObjectsEx() + 158 bytes	Unknown	No symbols loaded.
combase.dll!CoGetMarshalSizeMax() + 8409 bytes	Unknown	No symbols loaded.
combase.dll!WindowsCompareStringOrdinal() + 1765 bytes	Unknown	No symbols loaded.
combase.dll!CoWaitForMultipleHandles() + 118 bytes	Unknown	No symbols loaded.
[Managed to Native Transition]		Annotated Frame
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OracleConnectionDispenser<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.ProcessInstanceUp(OracleInternal.ConnectionPool.OraclePoolManager currentPM, Oracle.ManagedDataAccess.Client.OracleHAEventArgs haEvent, bool bUnplannedOutage, OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1631	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.TreatConnectionCreationAsUpEvent(OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1318	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.CreateServiceCtx(OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1263	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.PutNewPR(OracleInternal.ServiceObjects.OracleConnectionImpl pr, bool bForPoolPopulation) Line 1092	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.CreateNewPR(int reqCount, bool bForPoolPopulation, OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, bool bIncrSemObtained, bool bMaxSemObtained, bool bIgnoreIdleSem, out bool bIdleSemObtainedWhileCreatingPR, string instanceName, System.Collections.Generic.List<string> switchFailedInstNames) Line 1799	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OraclePoolManager.CreateNewPR(int reqCount, bool bForPoolPopulation, OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, bool bIncrSemObtained, bool bMaxSemObtained, bool bIgnoreIdleSem, out bool bIdleSemObtainedWhileCreatingPR, string instanceName, System.Collections.Generic.List<string> switchFailedInstNames) Line 407	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.Get(OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, bool bGetForApp, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, string affinityInstanceName, bool bForceMatch) Line 3329	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OraclePoolManager.Get(OracleInternal.ConnectionPool.ConnectionString csWithNewPassword, bool bGetForApp, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, string affinityInstanceName, bool bForceMatch) Line 467	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OracleConnectionDispenser<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.Get(OracleInternal.ConnectionPool.ConnectionString cs, OracleInternal.ConnectionPool.OraclePoolManager conPM, OracleInternal.ConnectionPool.ConnectionString pmCS, System.Security.SecureString securedPassword, System.Security.SecureString securedProxyPassword, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria) Line 580	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!Oracle.ManagedDataAccess.Client.OracleConnection.Open() Line 2484	C#	Symbols loaded.
TechTalk.Genome.dll!TechTalk.Genome.DatabaseConnection.OpenConnection()	Unknown	No symbols loaded.
[...]


ntdll.dll!NtWaitForMultipleObjects() + 20 bytes	Unknown	No symbols loaded.
KERNELBASE.dll!WaitForMultipleObjectsEx() + 240 bytes	Unknown	No symbols loaded.
[Managed to Native Transition]		Annotated Frame
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OracleConnectionDispenser<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.ProcessInstanceUp(OracleInternal.ConnectionPool.OraclePoolManager currentPM, Oracle.ManagedDataAccess.Client.OracleHAEventArgs haEvent, bool bUnplannedOutage, OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1631	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.TreatConnectionCreationAsUpEvent(OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1318	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.CreateServiceCtx(OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1263	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.PutNewPR(OracleInternal.ServiceObjects.OracleConnectionImpl pr, bool bForPoolPopulation) Line 1092	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.CreateNewPR(int reqCount, bool bForPoolPopulation, OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, bool bIncrSemObtained, bool bMaxSemObtained, bool bIgnoreIdleSem, out bool bIdleSemObtainedWhileCreatingPR, string instanceName, System.Collections.Generic.List<string> switchFailedInstNames) Line 1799	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OraclePoolManager.CreateNewPR(int reqCount, bool bForPoolPopulation, OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, bool bIncrSemObtained, bool bMaxSemObtained, bool bIgnoreIdleSem, out bool bIdleSemObtainedWhileCreatingPR, string instanceName, System.Collections.Generic.List<string> switchFailedInstNames) Line 407	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.Get(OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, bool bGetForApp, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, string affinityInstanceName, bool bForceMatch) Line 3329	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OraclePoolManager.Get(OracleInternal.ConnectionPool.ConnectionString csWithNewPassword, bool bGetForApp, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, string affinityInstanceName, bool bForceMatch) Line 467	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OracleConnectionDispenser<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.Get(OracleInternal.ConnectionPool.ConnectionString cs, OracleInternal.ConnectionPool.OraclePoolManager conPM, OracleInternal.ConnectionPool.ConnectionString pmCS, System.Security.SecureString securedPassword, System.Security.SecureString securedProxyPassword, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria) Line 580	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!Oracle.ManagedDataAccess.Client.OracleConnection.Open() Line 2484	C#	Symbols loaded.
TechTalk.Genome.dll!TechTalk.Genome.DatabaseConnection.OpenConnection()	Unknown	No symbols loaded.
[...]


ntdll.dll!NtWaitForMultipleObjects() + 20 bytes	Unknown	No symbols loaded.
KERNELBASE.dll!WaitForMultipleObjectsEx() + 240 bytes	Unknown	No symbols loaded.
[Managed to Native Transition]		Annotated Frame
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OracleConnectionDispenser<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.ProcessInstanceUp(OracleInternal.ConnectionPool.OraclePoolManager currentPM, Oracle.ManagedDataAccess.Client.OracleHAEventArgs haEvent, bool bUnplannedOutage, OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1631	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.TreatConnectionCreationAsUpEvent(OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1318	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.CreateServiceCtx(OracleInternal.ServiceObjects.OracleConnectionImpl pr) Line 1263	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.PutNewPR(OracleInternal.ServiceObjects.OracleConnectionImpl pr, bool bForPoolPopulation) Line 1092	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.CreateNewPR(int reqCount, bool bForPoolPopulation, OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, bool bIncrSemObtained, bool bMaxSemObtained, bool bIgnoreIdleSem, out bool bIdleSemObtainedWhileCreatingPR, string instanceName, System.Collections.Generic.List<string> switchFailedInstNames) Line 1799	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OraclePoolManager.CreateNewPR(int reqCount, bool bForPoolPopulation, OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, bool bIncrSemObtained, bool bMaxSemObtained, bool bIgnoreIdleSem, out bool bIdleSemObtainedWhileCreatingPR, string instanceName, System.Collections.Generic.List<string> switchFailedInstNames) Line 407	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.PoolManager<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.Get(OracleInternal.ConnectionPool.ConnectionString csWithDiffOrNewPwd, bool bGetForApp, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, string affinityInstanceName, bool bForceMatch) Line 3329	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OraclePoolManager.Get(OracleInternal.ConnectionPool.ConnectionString csWithNewPassword, bool bGetForApp, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria, string affinityInstanceName, bool bForceMatch) Line 467	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!OracleInternal.ConnectionPool.OracleConnectionDispenser<OracleInternal.ConnectionPool.OraclePoolManager, OracleInternal.ConnectionPool.OraclePool, OracleInternal.ServiceObjects.OracleConnectionImpl>.Get(OracleInternal.ConnectionPool.ConnectionString cs, OracleInternal.ConnectionPool.OraclePoolManager conPM, OracleInternal.ConnectionPool.ConnectionString pmCS, System.Security.SecureString securedPassword, System.Security.SecureString securedProxyPassword, Oracle.ManagedDataAccess.Client.OracleConnection connRefForCriteria) Line 580	C#	Symbols loaded.
Oracle.ManagedDataAccess.dll!Oracle.ManagedDataAccess.Client.OracleConnection.Open() Line 2484	C#	Symbols loaded.
TechTalk.Genome.dll!TechTalk.Genome.DatabaseConnection.OpenConnection()	Unknown	No symbols loaded.
[...]

I also have a process dump (zipped around 300MB) at the deadlock if you want me to send it (on private channels).

Not reproducible.

Answer 1 · 2024-08-02T06:49:25.000Z

Was this behavior observed after an ODP.NET upgrade, meaning the same code ran without a deadlock with the earlier ODP.NET version? If so, what was the version? Is this ODP.NET Core or managed ODP.NET?

Since this problem isn't reproducible on another system, I recommend opening up a service request with Oracle Support. Typically, we'll need to collect further diagnostics to determine the root cause. And the only way that would be possible is to run those diagnostics in the environment with the problem.

Answer 2 · 2024-08-05T12:58:10.000Z

We've been on 21.14.0 since May 1st and had no issues since then.

This was only a one time thing. We have never seen it afterwards so I cannot offer more information than the memory dump I created when the deadlock was happening.

Answer 3 · 2024-08-10T08:23:09.000Z

I still recommend the Oracle Support SR route if you want to get the dump analyzed. The Oracle Support team has the most experience with working with memory dumps.

Answer 4 · 2024-08-14T02:25:36.000Z

Closing. The next step is to open an SR for Oracle Support to analyze the dump and collect more diagnostics.