hrydgard/ppsspp

The 1.10 Android mystery crash thread!

hrydgard opened this issue · 19 comments

Here we go again, analyzing new-looking crashes from Google Play to see if it's worth doing a 1.10.1 release.

First up is this ~Download crash:

42 minutes ago on app version 110000000
Huawei HUAWEI Y5 lite (HWDRA-MG), 1024MB RAM, Android 8.1
Report 1 of 3

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 000000000001a3ce  /system/lib/libc.so (abort+63)
  #01  pc 000000000001a5f1  /system/lib/libc.so (__assert2+20)
  #02  pc 0000000000a5ba67  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so
  #03  pc 0000000000a5baef  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so
  #04  pc 0000000000a5a129  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so
  #05  pc 0000000000a5a0c1  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so (std::terminate()+28)
  #06  pc 0000000000a58865  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so (std::__ndk1::thread::~thread()+12)
  #07  pc 000000000053335f  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so (http::Download::~Download()+152)
  #08  pc 000000000053489d  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so (std::__ndk1::__shared_ptr_pointer<http::Download*, std::__ndk1::default_delete<http::Download>, std::__ndk1::allocator<http::Download>>::__on_zero_shared()+12)
  #09  pc 00000000005347bf  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, std::__ndk1::__bind<void (http::Download::*)(std::__ndk1::shared_ptr<http::Download>), http::Download*, std::__ndk1::shared_ptr<http::Download>&> > >(void*)+166)
  #10  pc 000000000004752f  /system/lib/libc.so (__pthread_start(void*)+22)
  #11  pc 000000000001af9d  /system/lib/libc.so (__start_thread+32)

This one feels like we still have some Vulkan mistake on shutdown, so not so critical (or it's just a driver bug, god knows there are enough of those):

27 minutes ago on app version 110000000
Huawei Y9 Prime 2019 (HWSTK-HF), 3840MB RAM, Android 10
Report 1 of 1
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 000000000006f06c  /apex/com.android.runtime/lib64/bionic/libc.so (abort+160)
  #01  pc 00000000000cf73c  /apex/com.android.runtime/lib64/bionic/libc.so (__fortify_fatal(char const*, ...)+116)
  #02  pc 00000000000cee70  /apex/com.android.runtime/lib64/bionic/libc.so (HandleUsingDestroyedMutex(pthread_mutex_t*, char const*)+52)
  #03  pc 00000000000cecd4  /apex/com.android.runtime/lib64/bionic/libc.so (pthread_mutex_lock+148)
  #04  pc 000000000245145c  /vendor/lib64/egl/libGLES_mali.so
  #05  pc 00000000024513c0  /vendor/lib64/egl/libGLES_mali.so
  #06  pc 000000000228a36c  /vendor/lib64/egl/libGLES_mali.so
  #07  pc 000000000228a124  /vendor/lib64/egl/libGLES_mali.so (vkQueueSubmit+80)
  #08  pc 000000000070e5e8  /data/app/org.ppsspp.ppsspp-4uJCUxd3fSg8xSVKpyIaoQ==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (VulkanRenderManager::Submit(int, bool)+816)
  #09  pc 000000000070e758  /data/app/org.ppsspp.ppsspp-4uJCUxd3fSg8xSVKpyIaoQ==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (VulkanRenderManager::EndSubmitFrame(int)+64)
  #10  pc 000000000070b2b4  /data/app/org.ppsspp.ppsspp-4uJCUxd3fSg8xSVKpyIaoQ==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (VulkanRenderManager::ThreadFunc()+368)
  #11  pc 000000000070ec68  /data/app/org.ppsspp.ppsspp-4uJCUxd3fSg8xSVKpyIaoQ==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (VulkanRenderManager::*)(), VulkanRenderManager*> >(void*)+64)
  #12  pc 00000000000ce1b0  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+36)
  #13  pc 0000000000070ba8  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)

To be continued!

For Download, we just do this:

	if (thread_.joinable())
		thread_.join();

I figured this was equivalent to converting it to a pointer and doing a nullcheck, but maybe there's some difference? As long as STL is behaving as advertised, this makes me worry about a double free or memory corruption?

-[Unknown]

I don't know if we maybe have to do if (thread && thread.joinable()) maybe? but ideally we shouldn't be in a position where thread can be "null" though anyway....

Yeah, we start both downloads immediately, so there should always be a thread. And nothing else joins it.

I think we still have exceptions off - I wonder what happens if the thread fails to start... but I mean, according to the trace thread was destructing after Download destructed, and STL should:

If *this has an associated thread (joinable() == true), std::terminate() is called.

And a postcondition of join(), which we call, is that joinable() becomes false.

-[Unknown]

Indeed, weird. Don't really have a lot of data yet (rollout at 1.5%), got 4 reports so far of it, 4 that got coalesced and 2 separate ones (only those two got auto-marked with NEW).

I'll take a look at the code myself tomorrow but I don't expect to figure out much more than you did...

Not seeing anything else dramatic so far. I'm gonna ramp up the rollout to 5%.

The Download thing continues to be the most common new crash.

This one in PPGe is the runner-up so far:

Acer Liquid S1 (a10), 1024MB RAM, Android 10
Report 1

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 00000000003cb582  /data/app/org.ppsspp.ppsspp-V6uDCI3e1gytN9BnXhsn2A==/split_config.armeabi_v7a.apk!libppsspp_jni.so (offset 0x1000)
  #01  pc 00000000003cb209  /data/app/org.ppsspp.ppsspp-V6uDCI3e1gytN9BnXhsn2A==/split_config.armeabi_v7a.apk!libppsspp_jni.so (offset 0x1000) (PPGeDrawText(char const*, float, float, PPGeAlign, float, unsigned int)+156)
  #02  pc 00000000002d2667  /data/app/org.ppsspp.ppsspp-V6uDCI3e1gytN9BnXhsn2A==/split_config.armeabi_v7a.apk!libppsspp_jni.so (offset 0x1000) (PSPSaveDialog::DisplaySaveDataInfo1()+1290)
  #03  pc 00000000002d3609  /data/app/org.ppsspp.ppsspp-V6uDCI3e1gytN9BnXhsn2A==/split_config.armeabi_v7a.apk!libppsspp_jni.so (offset 0x1000) (PSPSaveDialog::Update(int)+1956)
  #04  pc 0000000000387efb  /data/app/org.ppsspp.ppsspp-V6uDCI3e1gytN9BnXhsn2A==/split_config.armeabi_v7a.apk!libppsspp_jni.so (offset 0x1000)
  #05  pc 00000000002f7fef  /data/app/org.ppsspp.ppsspp-V6uDCI3e1gytN9BnXhsn2A==/split_config.armeabi_v7a.apk!libppsspp_jni.so (offset 0x1000) (CallSyscallWithoutFlags(HLEFunction const*)+14)
  #06  pc 0000000000484402  <anonymous>

I have a theory about the Download thing. Apparently we send in a shared_ptr to the thread, to keep "self" alive. But if that is the only owner of self.. the destruction seems to contain contradictions. It seems in that case the destructor would run on the thread itself when the thread function exits, and a thread joining itself doesn't sound healthy...

Oh, that's a good theory. Actually sounds likely it would become the last owner...

PPGe - the DisplaySaveDataInfo1 crash is new? I guess it must be related to the text drawer, maybe it's some save list with 100 saves and we ran out of space? Hm.

-[Unknown]

Yeah, that one's new. Far less common than the Download one.

Here's one that's not new, but it's pretty high in the chart:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 0000000000499108  /data/app/org.ppsspp.ppsspp-6uhzoIIKLtxYP2LX_cVf7A==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (AfterMatchingMipsCall::run(MipsCall&)+80)
  #01  pc 00000000003fd494  /data/app/org.ppsspp.ppsspp-6uhzoIIKLtxYP2LX_cVf7A==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (HLEReturnFromMipsCall()+540)
  #02  pc 00000000003fd978  /data/app/org.ppsspp.ppsspp-6uhzoIIKLtxYP2LX_cVf7A==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (CallSyscallWithoutFlags(HLEFunction const*)+32)
  #03  pc 0000000000070740  <anonymous>

That's in the Adhoc matchmaking code it seems.

Here's another interesting one, pretty rare though:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 00000000004c9662  /data/app/org.ppsspp.ppsspp-dHsZvlMFDlK8Cv_0xwbyJQ==/lib/arm/libppsspp_jni.so (GameScreen::render()+1529)
  #01  pc 00000000005555c9  /data/app/org.ppsspp.ppsspp-dHsZvlMFDlK8Cv_0xwbyJQ==/lib/arm/libppsspp_jni.so (ScreenManager::render()+84)
  #02  pc 00000000004a158f  /data/app/org.ppsspp.ppsspp-dHsZvlMFDlK8Cv_0xwbyJQ==/lib/arm/libppsspp_jni.so (NativeRender(GraphicsContext*)+502)
  #03  pc 000000000049b86f  /data/app/org.ppsspp.ppsspp-dHsZvlMFDlK8Cv_0xwbyJQ==/lib/arm/libppsspp_jni.so (UpdateRunLoopAndroid(_JNIEnv*)+22)
  #04  pc 000000000049d0d9  /data/app/org.ppsspp.ppsspp-dHsZvlMFDlK8Cv_0xwbyJQ==/lib/arm/libppsspp_jni.so
  #05  pc 00000000002ac665  /data/app/org.ppsspp.ppsspp-dHsZvlMFDlK8Cv_0xwbyJQ==/lib/arm/libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+24)
  #06  pc 0000000000047d3f  /system/lib/libc.so (__pthread_start(void*)+22)
  #07  pc 000000000001b029  /system/lib/libc.so (__start_thread+32)

And an old one that seems suspicious, though can also be a driver bug. Don't completely trust our descriptor set handling code....

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 00000000000cd6c0  /vendor/lib64/hw/vulkan.sdm660.so (A5xDescriptorSet::HwUpdateWriteDescriptors(QglDescriptorWriteData const*)+576)
  #01  pc 0000000000054d1c  /vendor/lib64/hw/vulkan.sdm660.so (qglinternal::vkUpdateDescriptorSets(VkDevice_T*, unsigned int, VkWriteDescriptorSet const*, unsigned int, VkCopyDescriptorSet const*)+532)
  #02  pc 000000000053e994  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (DrawEngineVulkan::GetOrCreateDescriptorSet(VkImageView_T*, VkSampler_T*, VkBuffer_T*, VkBuffer_T*, VkBuffer_T*, bool)+1308)
  #03  pc 000000000053f950  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (DrawEngineVulkan::DoFlush()+3184)
  #04  pc 00000000005b23e0  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (GPUCommon::FastRunLoop(DisplayList&)+192)
  #05  pc 00000000005b1f40  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (GPUCommon::InterpretList(DisplayList&)+904)
  #06  pc 00000000005b11a0  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (GPUCommon::ProcessDLQueue()+112)
  #07  pc 00000000005b1018  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (GPUCommon::EnqueueList(unsigned int, unsigned int, int, PSPPointer<PspGeListArgs>, bool)+1588)
  #08  pc 000000000041fb60  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (void WrapU_UUIU<&(sceGeListEnQueue(unsigned int, unsigned int, int, unsigned int))>()+60)
  #09  pc 00000000003fd978  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (CallSyscallWithoutFlags(HLEFunction const*)+32)
  #10  pc 0000000000022e28  <anonymous>

Anyway, should already be at an acceptable level, will probably release 1.10.1 tomorrow or the day after.

Did a quiet 10% rollout of 1.10.1 on Android. Here's one I hadn't spotted before, 19 times on a single device "A44 (itel-A44)" (so probably not much to worry about, but got to be annoying for that person):

ktrace:
  #00  pc 000000000004ad34  /system/lib/libc.so (tgkill+12)
  #01  pc 00000000000484d3  /system/lib/libc.so (pthread_kill+34)
  #02  pc 000000000001d485  /system/lib/libc.so (raise+10)
  #03  pc 0000000000018fc1  /system/lib/libc.so (__libc_android_abort+34)
  #04  pc 0000000000017024  /system/lib/libc.so (abort+4)
  #05  pc 0000000000319845  /system/lib/libart.so (art::Runtime::Abort()+252)
  #06  pc 00000000000b4cb9  /system/lib/libart.so (art::LogMessage::~LogMessage()+864)
  #07  pc 0000000000332dd3  /system/lib/libart.so (art::Thread::AssertNoPendingException() const+602)
  #08  pc 00000000000e494d  /system/lib/libart.so (art::ClassLinker::FindClass(art::Thread*, char const*, art::Handle<art::mirror::ClassLoader>)+28)
  #09  pc 0000000000260ccf  /system/lib/libart.so (art::JNI::FindClass(_JNIEnv*, char const*)+1878)
  #10  pc 000000000049c603  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (__cameraGetDeviceList()+218)
  #11  pc 0000000000386329  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (Camera::getDeviceList()+4)
  #12  pc 00000000004ce08f  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (GameSettingsScreen::CreateViews()+14990)
  #13  pc 00000000005570e1  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UIScreen::DoRecreateViews()+140)
  #14  pc 000000000055731d  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UIScreen::render()+24)
  #15  pc 0000000000555539  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (ScreenManager::render()+84)
  #16  pc 00000000004a163f  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (NativeRender(GraphicsContext*)+502)
  #17  pc 000000000049b91f  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UpdateRunLoopAndroid(_JNIEnv*)+22)
  #18  pc 000000000049d189  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so
  #19  pc 00000000002ac6a5  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+24)
  #20  pc 0000000000047fa3  /system/lib/libc.so (__pthread_start(void*)+22)
  #21  pc 0000000000019a0d  /system/lib/libc.so (__start_thread+6)

Download crashes are gone.

This oldie but goodie is rising through the ranks:

  #00  pc 000000000001e698  /system/lib64/libc.so (abort+120)
  #01  pc 00000000000083e4  /system/lib64/liblog.so (__android_log_assert+296)
  #02  pc 0000000000618fac  /data/app/org.ppsspp.ppsspp-LoZRddpxiOkiOn5A8galeA==/lib/arm64/libppsspp_jni.so (AndroidAssertLog(char const*, char const*, int, char const*, char const*, ...)+168)
  #03  pc 0000000000627690  /data/app/org.ppsspp.ppsspp-LoZRddpxiOkiOn5A8galeA==/lib/arm64/libppsspp_jni.so (NativeInitGraphics(GraphicsContext*)+324)
  #04  pc 0000000000622a74  /data/app/org.ppsspp.ppsspp-LoZRddpxiOkiOn5A8galeA==/lib/arm64/libppsspp_jni.so
  #05  pc 0000000000395d50  /data/app/org.ppsspp.ppsspp-LoZRddpxiOkiOn5A8galeA==/lib/arm64/libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+44)
  #06  pc 0000000000077e08  /system/lib64/libc.so (__pthread_start(void*)+36)
  #07  pc 000000000001fa74  /system/lib64/libc.so (__start_thread+68)

This guy is still around:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 000000000049a260  /data/app/org.ppsspp.ppsspp-TJeQtrHHAei0silzKB4lvg==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (AfterMatchingMipsCall::run(MipsCall&)+80)
  #01  pc 00000000003fd5d0  /data/app/org.ppsspp.ppsspp-TJeQtrHHAei0silzKB4lvg==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (HLEReturnFromMipsCall()+540)
  #02  pc 00000000003fdab4  /data/app/org.ppsspp.ppsspp-TJeQtrHHAei0silzKB4lvg==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (CallSyscallWithoutFlags(HLEFunction const*)+32)
  #03  pc 00000000000a8ce4  <anonymous>

Another new one I spotted while going through stuff that was buried before:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 00000000000220ac  /system/lib64/libc.so (abort+116)
  #01  pc 0000000000c1de50  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000)
  #02  pc 0000000000c1ded0  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000)
  #03  pc 0000000000c1ae98  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000)
  #04  pc 0000000000c1ae40  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000) (std::terminate()+52)
  #05  pc 000000000048f950  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000) (__NetAdhocInit()+452)
  #06  pc 00000000004399f0  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000) (__KernelInit()+188)
  #07  pc 0000000000458b30  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000) (__KernelLoadExec(char const*, unsigned int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>>*)+392)
  #08  pc 00000000004fa574  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000)
  #09  pc 0000000000091fac  /system/lib64/libc.so (__pthread_start(void*)+36)
  #10  pc 0000000000023968  /system/lib64/libc.so (__start_thread+68)

And a real oddball that's also not new:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 0000000000558a1a  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::PopupMultiChoice::ChoiceCallback(int)+37)
  #01  pc 0000000000558561  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (ListPopupScreen::OnListChoice(UI::EventParams&)+48)
  #02  pc 000000000055e303  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::Event::Dispatch(UI::EventParams&)+30)
  #03  pc 0000000000552e65  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::DispatchEvents()+340)
  #04  pc 0000000000553ff1  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::UpdateViewHierarchy(UI::ViewGroup*)+1072)
  #05  pc 00000000005578e1  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (PopupScreen::update()+24)
  #06  pc 000000000055501d  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (ScreenManager::update()+44)
  #07  pc 00000000004a2067  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (NativeUpdate()+190)
  #08  pc 000000000049b913  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UpdateRunLoopAndroid(_JNIEnv*)+10)
  #09  pc 000000000049d189  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so
  #10  pc 00000000002ac6a5  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+24)
  #11  pc 0000000000041adb  /system/lib/libc.so (__pthread_start(void*)+30)
  #12  pc 0000000000019355  /system/lib/libc.so (__start_thread+6)

Here's an ANR that might be slightly serious, seems to be from using joypad on android while downloads are cancelling.

one thread:

  #00  pc 000000000001ed2c  /system/lib64/libc.so (syscall+28)
  #01  pc 000000000002202c  /system/lib64/libc.so (__futex_wait_ex(void volatile*, bool, int, bool, timespec const*)+140)
  #02  pc 00000000000840d0  /system/lib64/libc.so (NonPI::MutexLockWithTimeout(pthread_mutex_internal_t*, bool, timespec const*)+632)
  #03  pc 0000000000c10e88  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (std::__ndk1::recursive_mutex::lock()+8)
  #04  pc 000000000071be8c  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (ScreenManager::axis(AxisInput const&)+36)
  #05  pc 000000000062996c  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (NativeAxis(AxisInput const&)+584)
  #06  pc 0000000000620d30  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (Java_org_ppsspp_ppsspp_NativeApp_accelerometer+84)
  at org.ppsspp.ppsspp.NativeApp.accelerometer (Native method)
  at org.ppsspp.ppsspp.NativeGLView.onSensorChanged (NativeGLView.java:114)
  at android.hardware.SystemSensorManager$SensorEventQueue.dispatchSensorEvent (SystemSensorManager.java:833)
  at android.os.MessageQueue.nativePollOnce (Native method)
  at android.os.MessageQueue.next (MessageQueue.java:327)
  at android.os.Looper.loop (Looper.java:196)
  at android.app.ActivityThread.main (ActivityThread.java:6760)
  at java.lang.reflect.Method.invoke (Native method)
  at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:493)
  at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:911)

Another thread:

  #00  pc 000000000001ed2c  /system/lib64/libc.so (syscall+28)
  #01  pc 0000000000083720  /system/lib64/libc.so (pthread_join+208)
  #02  pc 0000000000c189fc  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (std::__ndk1::thread::join()+28)
  #03  pc 00000000006ef228  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (http::Downloader::CancelAll()+136)
  #04  pc 00000000006b325c  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (StoreScreen::~StoreScreen()+44)
  #05  pc 00000000006b33ec  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (StoreScreen::~StoreScreen()+16)
  #06  pc 000000000071c2e0  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (ScreenManager::processFinishDialog()+368)
  #07  pc 00000000006287a8  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (NativeRender(GraphicsContext*)+416)
  #08  pc 00000000006207dc  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (UpdateRunLoopAndroid(_JNIEnv*)+48)
  #09  pc 0000000000622ab4  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (???)
  #10  pc 0000000000395d50  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+44)
  #11  pc 0000000000083114  /system/lib64/libc.so (__pthread_start(void*)+36)
  #12  pc 00000000000233bc  /system/lib64/libc.so (__start_thread+68)

Third thread:

  #00  pc 000000000001ed2c  /system/lib64/libc.so (syscall+28)
  #01  pc 000000000002202c  /system/lib64/libc.so (__futex_wait_ex(void volatile*, bool, int, bool, timespec const*)+140)
  #02  pc 000000000008261c  /system/lib64/libc.so (pthread_cond_wait+60)
  #03  pc 0000000000bd7910  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (std::__ndk1::condition_variable::wait(std::__ndk1::unique_lock<std::__ndk1::mutex>&)+20)
  #04  pc 00000000006fdd50  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (GLRenderManager::ThreadFrame()+196)
  at org.ppsspp.ppsspp.NativeRenderer.displayRender (Native method)
  at org.ppsspp.ppsspp.NativeRenderer.onDrawFrame (NativeRenderer.java:27)
  at android.opengl.GLSurfaceView$GLThread.guardedRun (GLSurfaceView.java:1578)
  at android.opengl.GLSurfaceView$GLThread.run (GLSurfaceView.java:1274)

Downloader::Do:

"Downloader::Do" prio=5 (not attached)
  | sysTid=21245 nice=-10 cgrp=default
  | state=S schedstat=( 4981001 4416999 9 ) utm=0 stm=0 core=4 HZ=100
  #00  pc 0000000000070164  /system/lib64/libc.so (recvfrom+4)
  #01  pc 00000000006c2ab4  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (Buffer::Read(int, unsigned long)+124)
  #02  pc 00000000006ec9e8  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (http::Client::ReadResponseHeaders(Buffer*, std::__ndk1::vector<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>>, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>>>>&, float*)+96)
  #03  pc 00000000006ec860  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (http::Client::GET(char const*, Buffer*, std::__ndk1::vector<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>>, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>>>>&, float*, bool*)+144)
  #04  pc 00000000006ee508  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (http::Download::PerformGET(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&)+968)
  #05  pc 00000000006edbd8  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (http::Download::Do()+284)
  #06  pc 00000000006ef598  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, std::__ndk1::__bind<void (http::Download::*)(), http::Download*> > >(void*)+64)
  #07  pc 0000000000083114  /system/lib64/libc.so (__pthread_start(void*)+36)
  #08  pc 00000000000233bc  /system/lib64/libc.so (__start_thread+68)

Not all of these are necessarily involved in the likely deadlock though.


  #00  pc 000000000001ed2c  /system/lib64/libc.so (syscall+28)
  #01  pc 000000000002202c  /system/lib64/libc.so (__futex_wait_ex(void volatile*, bool, int, bool, timespec const*)+140)
  #02  pc 000000000008261c  /system/lib64/libc.so (pthread_cond_wait+60)
  #03  pc 0000000000bd7910  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (std::__ndk1::condition_variable::wait(std::__ndk1::unique_lock<std::__ndk1::mutex>&)+20)
  #04  pc 0000000000717248  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (PrioritizedWorkQueue::Pop()+156)
  #05  pc 00000000007175ec  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (???)
  #06  pc 0000000000083114  /system/lib64/libc.so (__pthread_start(void*)+36)
  #07  pc 00000000000233bc  /system/lib64/libc.so (__start_thread+68)

Hm, I think it's possible that the above one is simply a Buffer::Read blocking for a long time, making cancellation kinda not happen.

Agreed, that's what it looks like. We'd need to convert to non-blocking IO or send a signal to cancel.

-[Unknown]

InstallZipScreen???

backtrace:
  #00  pc 00000000006bca30  /data/app/org.ppsspp.ppsspp-1/lib/arm64/libppsspp_jni.so (InstallZipScreen::update()+192)
  #01  pc 000000000071b92c  /data/app/org.ppsspp.ppsspp-1/lib/arm64/libppsspp_jni.so (ScreenManager::update()+72)
  #02  pc 0000000000629400  /data/app/org.ppsspp.ppsspp-1/lib/arm64/libppsspp_jni.so (NativeUpdate()+276)
  #03  pc 00000000006207cc  /data/app/org.ppsspp.ppsspp-1/lib/arm64/libppsspp_jni.so (UpdateRunLoopAndroid(_JNIEnv*)+32)
  #04  pc 0000000000622ab4  /data/app/org.ppsspp.ppsspp-1/lib/arm64/libppsspp_jni.so
  #05  pc 0000000000395d50  /data/app/org.ppsspp.ppsspp-1/lib/arm64/libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+44)
  #06  pc 0000000000067508  /system/lib64/libc.so (__pthread_start(void*)+52)
  #07  pc 000000000001ed84  /system/lib64/libc.so (__start_thread+16)

Found another oddity, I think things have gone pretty wrong already when we get something like this:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 00000000003afe76  /data/app/org.ppsspp.ppsspp-Msn-tDGoUw-6IPEZ50vqRw==/split_config.armeabi_v7a.apk (offset 0x1000) (Memory::Write_Opcode_JIT(unsigned int, Memory::Opcode const&)+14)
  #01  pc 000000000039b2a1  /data/app/org.ppsspp.ppsspp-Msn-tDGoUw-6IPEZ50vqRw==/split_config.armeabi_v7a.apk (offset 0x1000) (JitBlockCache::FinalizeBlock(int, bool)+76)
  #02  pc 000000000028ddd3  /data/app/org.ppsspp.ppsspp-Msn-tDGoUw-6IPEZ50vqRw==/split_config.armeabi_v7a.apk (offset 0x1000) (MIPSComp::ArmJit::Compile(unsigned int)+118)
  #03  pc 0000000000000106  <anonymous>

Here's an interesting one that I got a single one of:

cktrace:
  #00  pc 0000000000606e64  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (VulkanDeviceAllocator::Free(VkDeviceMemory_T*, unsigned long)+140)
  #01  pc 0000000000602360  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (VulkanTexture::Destroy()+160)
  #02  pc 000000000070c964  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (Draw::VKTexture::~VKTexture()+40)
  #03  pc 00000000006f9a28  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (Draw::RefCountedObject::Release()+56)
  #04  pc 00000000006b86d0  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (HttpImageFileView::~HttpImageFileView()+68)
  #05  pc 00000000006b8784  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (HttpImageFileView::~HttpImageFileView()+16)
  #06  pc 0000000000731900  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (UI::ViewGroup::Clear()+100)
  #07  pc 00000000006b363c  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (ProductView::CreateViews()+56)
  #08  pc 00000000006b50e4  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (ProductView::Update()+44)
  #09  pc 0000000000734794  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (UI::ScrollView::Update()+92)
  #10  pc 00000000007321a0  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (UI::ViewGroup::Update()+80)
  #11  pc 00000000007321a0  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (UI::ViewGroup::Update()+80)
  #12  pc 000000000071c848  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (UI::UpdateViewHierarchy(UI::ViewGroup*)+1456)
  #13  pc 00000000006b582c  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (StoreScreen::update()+40)
  #14  pc 000000000071de1c  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (ScreenManager::update()+72)
  #15  pc 00000000006294ac  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (NativeUpdate()+276)
  #16  pc 0000000000622360  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (Java_org_ppsspp_ppsspp_NativeActivity_runEGLRenderLoop+632)
  #17  pc 00000000000031b8  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/oat/arm64/base.odex (offset 0x3000) (org.ppsspp.ppsspp.NativeActivity.runEGLRenderLoop+152)
  #18  pc 000000000055e988  /system/lib64/libart.so (art_quick_invoke_stub+584)
  #19  pc 00000000000cf940  /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
  #20  pc 0000000000283870  /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344)
  #21  pc 000000000027d82c  /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+948)
  #22  pc 0000000000531058  /system/lib64/libart.so (MterpInvokeVirtualQuick+584)
  #23  pc 0000000000554c94  /system/lib64/libart.so (ExecuteMterpImpl+29972)
  #24  pc 0000000000008efc  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/oat/arm64/base.vdex (org.ppsspp.ppsspp.NativeActivity$1.run+72)
  #25  pc 0000000000257328  /system/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool) (.llvm.1635852419)+496)
  #26  pc 000000000051e748  /system/lib64/libart.so (artQuickToInterpreterBridge+1032)
  #27  pc 0000000000567afc  /system/lib64/libart.so (art_quick_to_interpreter_bridge+92)
  #28  pc 000000000025d2e8  /system/framework/arm64/boot.oat (offset 0x114000) (java.lang.Thread.run+72)
  #29  pc 000000000055e988  /system/lib64/libart.so (art_quick_invoke_stub+584)
  #30  pc 00000000000cf940  /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
  #31  pc 0000000000464968  /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
  #32  pc 0000000000465a30  /system/lib64/libart.so (art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue*)+424)
  #33  pc 0000000000490da8  /system/lib64/libart.so (art::Thread::CreateCallback(void*)+1120)
  #34  pc 0000000000083814  /system/lib64/libc.so (__pthread_start(void*)+36)
  #35  pc 000000000002340c  /system/lib64/libc.so (__start_thread+68)

Closing this for #14082.

-[Unknown]