Running dotnet-dump on a service running under a non-root user results in error.

Question

Running dotnet-dump on a service running under a non-root user results in error.

jwilliamsonveeam opened this issue 4 months ago · 5 comments

Description

Running dotnet-dump on a service running under a non-root user results in error.
Expected: Dump is created without error
Actual:

If I change the user to root in the systemd service file and restart the service I can create the dump. I have tried setting AmbientCapabilities and/or setcap to give my process CAP_SYS_PTRACE and CAP_SYS_ADMIN and even though getcap shows it has these capabilties there is an error reading the trace.

The error is more clear when I run as root:

Writing full to /tmp/1188543.dump
[createdump] The process or container does not have permissions or access: open(/proc/283757/mem) FAILED Permission denied (13)
[createdump] Failure took 0ms

Full Stack trace as logged in user:

Unhandled exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.Net.Sockets.SocketException (13): Permission denied
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP)
   at System.Net.Sockets.Socket.BeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state)
   at Microsoft.Diagnostics.NETCore.Client.IpcSocket.Connect(EndPoint remoteEP, TimeSpan timeout) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcSocket.cs:line 52
   at Microsoft.Diagnostics.NETCore.Client.IpcUnixDomainSocket.Connect(EndPoint localEP, TimeSpan timeout) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcUnixDomainSocket.cs:line 30
   at Microsoft.Diagnostics.NETCore.Client.IpcEndpointHelper.Connect(IpcEndpointConfig config, TimeSpan timeout) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcTransport.cs:line 68
   at Microsoft.Diagnostics.NETCore.Client.PidIpcEndpoint.Connect(TimeSpan timeout) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcTransport.cs:line 243
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuation(IpcEndpoint endpoint, IpcMessage message) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 40
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessage(IpcEndpoint endpoint, IpcMessage message) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 25
   at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.WriteDump(DumpType dumpType, String dumpPath, WriteDumpFlags flags) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs:line 171
   at Microsoft.Diagnostics.Tools.Dump.Dumper.Collect(IConsole console, Int32 processId, String output, Boolean diag, Boolean crashreport, DumpTypeOption type, String name) in /_/src/Tools/dotnet-dump/Dumper.cs:line 136
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<BuildInvocationChain>b__0(InvocationContext invocationContext, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_2.<BuildInvocationChain>b__3(InvocationContext c)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseErrorReporting>b__21_0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<UseParseErrorReporting>b__21_0(InvocationContext context, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_2.<BuildInvocationChain>b__3(InvocationContext c)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseHelp>b__0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<UseHelp>b__0(InvocationContext context, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_2.<BuildInvocationChain>b__3(InvocationContext c)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass25_0.<<UseVersionOption>b__0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass25_0.<UseVersionOption>b__0(InvocationContext context, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_2.<BuildInvocationChain>b__3(InvocationContext c)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass23_0.<<UseTypoCorrections>b__0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass23_0.<UseTypoCorrections>b__0(InvocationContext context, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_2.<BuildInvocationChain>b__3(InvocationContext c)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__22_0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<UseSuggestDirective>b__22_0(InvocationContext context, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_2.<BuildInvocationChain>b__3(InvocationContext c)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseDirective>b__20_0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<UseParseDirective>b__20_0(InvocationContext context, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_2.<BuildInvocationChain>b__3(InvocationContext c)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseDebugDirective>b__11_0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<UseDebugDirective>b__11_0(InvocationContext context, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_2.<BuildInvocationChain>b__3(InvocationContext c)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<UseEnvironmentVariableDirective>b__12_0(InvocationContext context, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_2.<BuildInvocationChain>b__3(InvocationContext c)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<<UseExceptionHandler>b__0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<UseExceptionHandler>b__0(InvocationContext context, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_2.<BuildInvocationChain>b__3(InvocationContext c)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<CancelOnProcessTermination>b__5_0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<CancelOnProcessTermination>b__5_0(InvocationContext context, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_1.<BuildInvocationChain>b__2(InvocationContext ctx, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_1.<BuildInvocationChain>b__2(InvocationContext ctx, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_1.<BuildInvocationChain>b__2(InvocationContext ctx, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_1.<BuildInvocationChain>b__2(InvocationContext ctx, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_1.<BuildInvocationChain>b__2(InvocationContext ctx, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_1.<BuildInvocationChain>b__2(InvocationContext ctx, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_1.<BuildInvocationChain>b__2(InvocationContext ctx, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_1.<BuildInvocationChain>b__2(InvocationContext ctx, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_1.<BuildInvocationChain>b__2(InvocationContext ctx, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_1.<BuildInvocationChain>b__2(InvocationContext ctx, Func`2 next)
   at System.CommandLine.Invocation.InvocationPipeline.InvokeAsync(IConsole console)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Invocation.InvocationPipeline.InvokeAsync(IConsole console)
   at System.CommandLine.Parsing.ParseResultExtensions.InvokeAsync(ParseResult parseResult, IConsole console)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Parsing.ParseResultExtensions.InvokeAsync(ParseResult parseResult, IConsole console)
   at System.CommandLine.Parsing.ParserExtensions.InvokeAsync(Parser parser, String[] args, IConsole console)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.CommandLine.Parsing.ParserExtensions.InvokeAsync(Parser parser, String[] args, IConsole console)
   at Microsoft.Diagnostics.Tools.Dump.Program.Main(String[] args) in /_/src/Tools/dotnet-dump/Program.cs:line 27
   at Microsoft.Diagnostics.Tools.Dump.Program.<Main>(String[] args)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskToAsyncResult.End(IAsyncResult asyncResult)
   at Microsoft.Diagnostics.NETCore.Client.IpcSocket.Connect(EndPoint remoteEP, TimeSpan timeout) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcSocket.cs:line 56
   at Microsoft.Diagnostics.NETCore.Client.IpcUnixDomainSocket.Connect(EndPoint localEP, TimeSpan timeout) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcUnixDomainSocket.cs:line 30
   at Microsoft.Diagnostics.NETCore.Client.IpcEndpointHelper.Connect(IpcEndpointConfig config, TimeSpan timeout) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcTransport.cs:line 68
   at Microsoft.Diagnostics.NETCore.Client.PidIpcEndpoint.Connect(TimeSpan timeout) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcTransport.cs:line 243
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuation(IpcEndpoint endpoint, IpcMessage message) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 40
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessage(IpcEndpoint endpoint, IpcMessage message) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 25
   at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.WriteDump(DumpType dumpType, String dumpPath, WriteDumpFlags flags) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/DiagnosticsClient.cs:line 171
   at Microsoft.Diagnostics.Tools.Dump.Dumper.Collect(IConsole console, Int32 processId, String output, Boolean diag, Boolean crashreport, DumpTypeOption type, String name) in /_/src/Tools/dotnet-dump/Dumper.cs:line 136
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseErrorReporting>b__21_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass25_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass23_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__22_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseDirective>b__20_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseDebugDirective>b__11_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<<UseExceptionHandler>b__0>d.MoveNext()

### Configuration
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
App Armor purged

.NET SDK:
 Version:           8.0.108
 Commit:            665a05cea7
 Workload version:  8.0.100-manifests.109ff937

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  22.04
 OS Platform: Linux
 RID:         ubuntu.22.04-x64
 Base Path:   /usr/lib/dotnet/sdk/8.0.108/

.NET workloads installed:
 Workload version: 8.0.100-manifests.109ff937
There are no installed workloads to display.

Host:
  Version:      8.0.8
  Architecture: x64
  Commit:       08338fcaa5

.NET SDKs installed:
  8.0.108 [/usr/lib/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.8 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.8 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]

Regression?

Other information

Seems related to some other issues.
#4375
dotnet/sdk#15187
dotnet/runtime#62781
#2246
https://learn.microsoft.com/en-us/dotnet/core/diagnostics/faq-dumps

Answer 1 · 2024-08-30T22:41:57.000Z

SysInternals procdump also requires me to change the service to run as root.

Answer 2 · 2024-09-03T17:43:07.000Z

Is the service running as non-root using the same UID as the one you're using to collect the dump @jwilliamsonveeam ? If not, this is by design. Only the UID of the process is allowed to communicate with the target process.

Answer 3 · 2024-09-04T15:12:25.000Z

Is the service running as non-root using the same UID as the one you're using to collect the dump @jwilliamsonveeam ? If not, this is by design. Only the UID of the process is allowed to communicate with the target process.

Hi @hoyosjs ! I switched to the user that the process runs under and get the same error when I run dotnet-dump:
[createdump] The process or container does not have permissions or access: open(/proc/283757/mem) FAILED Permission denied (13)
[createdump] Failure took 0ms

Procdump gives the following:
[15:20:38 - ERROR]: Failed to open /proc/283757/fdinfo [Permission denied]
[15:20:38 - ERROR]: Failed to get number of file descriptors
[15:20:38 - ERROR]: Procdump is not running with elevated credentials or the effective uid does not match the effective uid of the target process (pid 283757).
[15:20:38 - ERROR]: MonitorProcesses: Failed to start the monitor.

Answer 4 · 2024-09-04T20:43:14.000Z

Is the service running in any container technology? Or systemd with Private Temp?

Another question - does the entrypoint of the service use capability restrictions (like setcap or using similar cap settings in k8s et al)? See under PR_SET_DUMPABLE in prctl(2) - Linux manual page (man7.org). This immediately means procfs gets owned by root. That means communications and dumping become infeasible for most tools.

Answer 5 · 2024-09-05T15:28:29.000Z

@hoyosjs Ah. We have setcap "CAP_WAKE_ALARM+ep" because we use the system timers and yes, the proc folders for our pid are owned by root. I will close this. Thanks for the help!