OpenTelemetry.AutoInstrumentation 1.7.0 NuGet | Fatal Error System.AccessViolationException + Missing DLL's referenced in instrument.cmd and instrument.sh scripts
Closed this issue · 10 comments
Bug Report
Fatal error thrown when trying to launch application using instrument.cmd or instrument.sh
Symptom
Describe the bug
Following the NuGet installation steps and running
./instrument.cmd dotnet run --project ../../../webapplication1.csproj
results in the following error
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. Repeat 2 times: at OpenTelemetry.AutoInstrumentation.NativeMethods+Windows.AddInstrumentations(System.String, OpenTelemetry.AutoInstrumentation.NativeCallTargetDefinition[], Int32) at OpenTelemetry.AutoInstrumentation.NativeMethods.AddInstrumentations(System.String, OpenTelemetry.AutoInstrumentation.NativeCallTargetDefinition[]) at OpenTelemetry.AutoInstrumentation.Instrumentation.RegisterBytecodeInstrumentations(Payload) at OpenTelemetry.AutoInstrumentation.Instrumentation.Initialize() at System.RuntimeMethodHandle.InvokeMethod(System.Object, Void**, System.Signature, Boolean) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(System.Object, System.Reflection.BindingFlags) at System.Reflection.MethodBase.Invoke(System.Object, System.Object[]) at OpenTelemetry.AutoInstrumentation.Loader.Loader.TryLoadManagedAssembly() at OpenTelemetry.AutoInstrumentation.Loader.Loader..cctor() at OpenTelemetry.AutoInstrumentation.Loader.Loader..ctor() at System.RuntimeType.CreateInstanceDefaultCtor(Boolean, Boolean) at System.RuntimeType.CreateInstanceImpl(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) at System.Reflection.Assembly.CreateInstance(System.String) at StartupHook.Initialize() at System.RuntimeMethodHandle.InvokeMethod(System.Object, Void**, System.Signature, Boolean) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(System.Object, System.Reflection.BindingFlags) at System.StartupHookProvider.CallStartupHook(StartupHookNameOrPath) at System.StartupHookProvider.ProcessStartupHooks(System.String)
Expected behavior
The application should launch with automatic instrumentation working.
Screenshots
If applicable, add screenshots to help explain your problem.
Runtime environment (please complete the following information):
- OpenTelemetry Automatic Instrumentation version: 1.7.0
- OS: Microsoft Windows 10 Pro
- .NET version: .NET 8
Additional context
I also noticed that the install scripts reference DLL's that don't appear to be installed as transitive dependencies
I can see that OpenTelemtry.AutoInstrumentation.Runtime.Native is installed as a transitive dependency, but the DLL never gets compiled, and even if it did, the path names would be incorrect.
I can also see the DLL is logged as missing in the autoinstrumentation logs
Reproduce
Steps to reproduce the behaviour:
- First step; Create a new ASP.NET core web application bootstrapped with .NET 8 and OpenAPI
- Second step; Install the OpenTelemetry.AutoInstrumentation NuGet package
- Third step: Run the command
./instrument.cmd dotnet run --project ../../../webapplication1.csprojand 💥
The same error also occurs when specifying a RID i.e
.\instrument.cmd dotnet run --project ../../../webapplication1.csproj -r win-x64
I think that you are messing 2 things. Building process with instrumentation: .\instrument.cmd dotnet run --project ../../../webapplication1.csproj -r win-x64
The expected solution is
- Execute build:
dotnet build --project ../../../webapplication1.csproj -r win-x64 - Execute your instrumented application
.\instrument.cmd your-application
Please let me know if it helps.
@3mv3, please provide Minimal, Reproducible Example. Without this, it can be hard to advice. It can be separate GH repository where we can find the code and all steps to follow/execute.
@Kielek @lachmatt I was getting throw off by this line in the docs
Use the script in the output folder of the build to launch the application with automatic instrumentation activated.
As I understand it, the default 'output' folder of a build is ./bin/<configuration>/<framework>/, and given that I could see the instrument scripts there, I assumed they were the correct ones:
I now understand you mean the output folder for the specific RID used to initiate the build, the path to which would be ./bin/<configuration>/<framework>/<rid>/
Another thing I noticed is, even without specifying a RID during a build, the native DLL's are compiled regardless and placed in a runtimes folder in the output directory:
That allowed me to build 'as normal' in Visual Studio, and use launchSettings to configure the environment variables, negating the need to launch each time with the instrument scripts:
I now understand you mean the output folder for the specific RID used to initiate the build, the path to which would be
./bin/<configuration>/<framework>/<rid>/
That was what launch scripts expected. They were recently updated to verify that assumption, print a warning, and fallback to the native profiler for detected architecture from runtimes directory you mentioned (see #3594).
Another thing I noticed is, even without specifying a RID during a build, the native DLL's are compiled regardless and placed in a runtimes folder in the output directory
That's correct, but it results in native profilers for all the supported architectures being copied to the build/publish output. New build target was added to NuGet package to warn if app is being published with a package reference to autoinstrumentation, without runtime identifier specified (see #3601 )