Memory leak in EVP_PKEY_decrypt()

[andrewhenle]

Using EVP_PKEY_decrypt() with an RSA private key leaks approximately 1KB of enclave heap for each call.

A zip file with a simple enclave application that demonstrates the leak is attached. Note line 304 of rsaencryption.c:

// set this to 1, and the leak disappears
#if 0
    int result = 1;
#else
    int result =  EVP_PKEY_decrypt( pkeyCtx, plaintext, &bytesDecrypted, ciphertext, ciphertextLen );
#endif

RSALeak.zip

[andrewhenle]

System is RHEL 8:

uname -a
Linux sgx 4.18.0-372.26.1.el8_6.x86_64 #1 SMP Sat Aug 27 02:44:20 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.7 (Ootpa)

Intel SGX SSL is latest version, based on OpenSSL 1.1.1t

SGX version is 2.19