SIGSEGV interrupt after TLS warning without having TLS enabled
bartvp1 opened this issue · 10 comments
Describe the bug
Even though I dont use TLS in my OvenMediaEngine configuration, it still occasionally warns about not being able to do something related to TLS, which often coincides with a crash of the Origin server.
I can't confirm whether this is related to the TLS warning, but since most crashes end with these same warnings I have my suspicions.
To Reproduce
No steps to reproduce, this seems to happen out of the blue
Expected behavior
I would expect there not to be any logs or crashes related to TLS if I'm not using any TLS for this service, because the load balancer terminates it.
Logs
Origin:
[2024-10-15 08:19:33.916] W [SPRTMP-t1935:11] OpenSSL | tls.cpp:328 | Tls::Read() returns 5 (errno: 11): [OpenSSL] error:00000000:lib(0)::reason(0) (0)
[2024-10-15 08:19:33.916] W [SPAPISvr-t8080:20] OpenSSL | tls.cpp:328 | Tls::Read() returns 5 (errno: 11): [OpenSSL] error:00000000:lib(0)::reason(0) (0)
[2024-10-15 08:19:33.916] C [SPRTMP-t1935:11] OvenMediaEngine | signals.cpp:124 | OME v0.16.8 ((From archive)) received signal 11 (SIGSEGV), interrupt.
[2024-10-15 08:19:33.916] C [SPAPISvr-t8080:20] OvenMediaEngine | signals.cpp:124 | OME v0.16.8 ((From archive)) received signal 11 (SIGSEGV), interrupt.
Server
- OS: Linux, using K8s containers
- OvenMediaEngine Version: v0.16.8
- Branch: release
Since there is no need to use TLS in the SPRTMP thread, the related logs should not be output. It seems unusual that they are being printed.
It is likely that an issue such as a stack overflow may have caused incorrect execution. For a more detailed analysis, please share the full logs, Server.xml, and any dump files generated in the /usr/share/ovenmediaengine/dumps directory inside the Docker container.
Here's the server configuration for the origin:
<?xml version="1.0" encoding="UTF-8"?>
<Server version="8">
<Name>OvenMediaEngine</Name>
<Type>origin</Type>
<IP>*</IP>
<StunServer>stun.l.google.com:19302</StunServer>
<Managers>
<Host>
<Names>
<Name>*</Name>
</Names>
</Host>
<API>
<AccessToken>${env:OME_API_ACCESS_TOKEN}</AccessToken>
</API>
</Managers>
<Bind>
<Managers>
<API>
<Port>8080</Port>
</API>
</Managers>
<Providers>
<RTMP>
<Port>1935</Port>
</RTMP>
</Providers>
<Publishers>
<OVT>
<Port>9000</Port>
</OVT>
</Publishers>
</Bind>
<VirtualHosts>
<VirtualHost include="VHost*.xml" />
<VirtualHost>
<Name>default</Name>
<Host>
<Names>
<Name>*</Name>
</Names>
</Host>
<AdmissionWebhooks>
<ControlServerUrl>${env:ADMISSION_ADDRESS}</ControlServerUrl>
<SecretKey>${env:ADMISSION_SECRET_KEY}</SecretKey>
<Timeout>3000</Timeout>
<Enables>
<Providers>rtmp</Providers>
</Enables>
</AdmissionWebhooks>
<OriginMapStore>
<RedisServer>
<Host>${env:REDIS_HOST}:${env:REDIS_PORT}</Host>
<Auth>${env:REDIS_PASSWORD}</Auth>
</RedisServer>
<OriginHostName>${env:POD_IP}</OriginHostName>
</OriginMapStore>
<Applications>
<Application>
<Name>app</Name>
<Type>live</Type>
<OutputProfiles>
<OutputProfile>
<Name>ABR</Name>
<OutputStreamName>${OriginStreamName}</OutputStreamName>
<Playlist>
<Name>abr</Name>
<FileName>abr</FileName>
<Options>
<WebRtcAutoAbr>true</WebRtcAutoAbr>
</Options>
<Rendition>
<Name>Full HD</Name>
<Video>video_1080</Video>
<Audio>cond_audio_opus</Audio>
</Rendition>
<Rendition>
<Name>HD</Name>
<Video>video_720</Video>
<Audio>cond_audio_opus</Audio>
</Rendition>
<Rendition>
<Name>SD</Name>
<Video>video_360</Video>
<Audio>cond_audio_opus</Audio>
</Rendition>
</Playlist>
<Encodes>
<Video>
<Name>video_1080</Name>
<Codec>h264</Codec>
<Height>1080</Height>
<Bitrate>2048000</Bitrate>
<Framerate>24</Framerate>
<ThreadCount>4</ThreadCount>
<Preset>slower</Preset>
</Video>
<Video>
<Name>video_720</Name>
<Codec>h264</Codec>
<Height>720</Height>
<Bitrate>1024000</Bitrate>
<Framerate>24</Framerate>
<ThreadCount>4</ThreadCount>
<Preset>slower</Preset>
</Video>
<Video>
<Name>video_360</Name>
<Codec>h264</Codec>
<Height>360</Height>
<Bitrate>700000</Bitrate>
<Framerate>24</Framerate>
<ThreadCount>4</ThreadCount>
<Preset>slower</Preset>
</Video>
<Audio>
<Name>bypass_audio</Name>
<Bypass>true</Bypass>
</Audio>
<Audio>
<Name>cond_audio_opus</Name>
<Codec>opus</Codec>
<Bitrate>128000</Bitrate>
<Samplerate>48000</Samplerate>
<Channel>2</Channel>
</Audio>
</Encodes>
</OutputProfile>
</OutputProfiles>
<Providers>
<RTMP>
<BlockDuplicateStreamName>true</BlockDuplicateStreamName>
</RTMP>
</Providers>
<Publishers>
<OVT />
<FILE>
<RootPath>/opt/ovenmediaengine/recordings</RootPath>
<FilePath>/${VirtualHost}/${Application}/${Stream}/manual-recording-${StartTime:YYYYMMDDhhmmss}.ts</FilePath>
<InfoPath>/${VirtualHost}/${Application}/${Stream}/manual-recording.xml</InfoPath>
<StreamMap>
<Enable>${env:ENABLE_AUTOMATED_RECORDING:false}</Enable>
<Path>./record.xml</Path>
</StreamMap>
</FILE>
</Publishers>
</Application>
</Applications>
</VirtualHost>
</VirtualHosts>
</Server>
I wasn't able to get any dumps from as the K8s container was deleted
These are the startup logs, I wasn't able to retrieve any other logs before the crash since they havent been persisted
I [OvenMediaEngine:1] Config | config_manager.cpp:239 | Trying to set logfile in directory... (/var/log/ovenmediaengine)
W [OvenMediaEngine:1] Socket | socket.cpp:1092 | [#5] [0x7f79ada341d0] Could not send data: -1 ([errno] Network is unreachable (101))
W [OvenMediaEngine:1] WebRTC Publisher | webrtc_publisher.cpp:105 | WebRTC Publisher is disabled by configuration
W [OvenMediaEngine:1] LLHLS Publisher | llhls_publisher.cpp:78 | LLHLS Publisher is disabled by configuration
W [OvenMediaEngine:1] Thumbnail | thumbnail_publisher.cpp:79 | ThumbnailPublisher is disabled by configuration
W [OvenMediaEngine:1] HLS Publisher | hls_publisher.cpp:78 | HLS Publisher is disabled by configuration
W [OvenMediaEngine:1] Transcoder | transcoder_gpu.cpp:54 | No supported NVIDIA Accelerator
W [OvenMediaEngine:1] Transcoder | transcoder_gpu.cpp:64 | No supported Xilinx Media Accelerator
W [OvenMediaEngine:1] Transcoder | transcoder_gpu.cpp:74 | No supported Intel QuickSync Accelerator
W [OvenMediaEngine:1] Transcoder | transcoder_gpu.cpp:84 | No supported Netint VPU Accelerator
W [OvenMediaEngine:1] WebRTC Provider | webrtc_provider.cpp:139 | WebRTCProvider is disabled by configuration
W [OvenMediaEngine:1] MpegtsProvider | mpegts_provider.cpp:147 | MPEGTSProvider is disabled by configuration
W [OvenMediaEngine:1] SrtProvider | srt_provider.cpp:50 | SrtProvider is disabled by configuration
W [OvenMediaEngine:1] Publisher | publisher.cpp:68 | Push publisher is disabled in #default#app application, so it was not created
W [OvenMediaEngine:1] Provider | provider.cpp:91 | OVT provider is disabled in #default#app application, so it was not created
W [OvenMediaEngine:1] Provider | provider.cpp:91 | RTSP Pull provider is disabled in #default#app application, so it was not created
W [OvenMediaEngine:1] Provider | provider.cpp:91 | Scheduled provider is disabled in #default#app application, so it was not created
W [OvenMediaEngine:1] Provider | provider.cpp:91 | Multiplex provider is disabled in #default#app application, so it was not created
@bartvp1
Thank you for your response.
Since the code did not run as expected, we may need to pinpoint the error location through the crash dump or review the logs to identify the situation before the issue occurred.
If you could mount the logs and dump path to persistent storage and share the relevant details when the issue is reproduced, it would be greatly appreciated.
(Even if the logs seem unnecessary, such as the OME version and host information, please do not remove them and upload the full logs.)
Thank you, I'll post the logs here when this problem appears again!