public key encrypt differing buffer sizes to C_Encrypt calls

[williamcroberts]

I reciveed this bug report over at the tpm2-pkcs11 project.

I have been able to narrow down a bug, where depending on python versions, it calls C_Encrypt with differing buffer sizes. On the working setup, it uses a really big buffer size of 139846147882768, in the broken setup, it uses an output buffer of size 16, which is the same size as the input plaintext. The script is below, and it fails on the public.encrypt line.

I verified that it left the Python code in the self._encrypt call with the same parameters in a debugger/added print statements.

        if isinstance(data, bytes):
            print("self._encrypt: len(data){} kwargs: {}".format(len(data), kwargs))
            return self._encrypt(data, **kwargs)

Then I verified that it comes into the actual pkcs11 module with differing arguments on buffer size. This makes me believe that their is something wonky going on in the bindings. Have you seen, or are you aware of any issues in that?

I also noticed that if I change the mechanism from RSA_PKCS to RSA_X_509 and modified the plaintext to be keysize (256 bytes in this case) that it all worked as advertised.

import pkcs11

lib = pkcs11.lib('libtpm2_pkcs11.so')

token = lib.get_token(token_label='tpmhsm')

with token.open(rw=False, user_pin='123456') as session:
    print(session)

    public = session.get_key(label='rsakey',object_class=pkcs11.constants.ObjectClass.PUBLIC_KEY)
    print(public)

    private = session.get_key(label='rsakey',object_class=pkcs11.constants.ObjectClass.PRIVATE_KEY)
    print(private)

    plaintext = '1234567890123456'

    ciphertext = public.encrypt(plaintext, mechanism=pkcs11.mechanisms.Mechanism.RSA_PKCS)
    print(ciphertext)

    plaintext = 'notdecrypted'
    plaintext = private.decrypt(ciphertext, mechanism=pkcs11.mechanisms.Mechanism.RSA_PKCS)
    print(plaintext)

Common Setup

• tpm2-tss: 3.0.1
• tpm2-tools: 4.1.3
• tpm2-pkcs11: master
• python-pkcs11: 0.7 installed via pip.

Working Machine:

• ubuntu 18.04
• Python 3.8.1
• ibmtpm1628
• tpm2-arbmd: 2.3.3-rc0

Broken Machine:

• Ubuntu 20.04
• Python 3.8.5
• one of the two setups:
  • qemu emulated, using StefanBerger SWTPM interface and /dev/tpmrm0
  • tpm2-abrmd and ibm1628 simulator

[williamcroberts]

The actual python stack trace is:

Traceback (most recent call last):
  File "./test.py", line 23, in <module>
    ciphertext = public.encrypt(plaintext, mechanism=pkcs11.mechanisms.Mechanism.RSA_PKCS)
  File "/home/test/.local/lib/python3.8/site-packages/pkcs11/types.py", line 871, in encrypt
    return self._encrypt(data, **kwargs)
  File "pkcs11/_pkcs11.pyx", line 890, in pkcs11._pkcs11.EncryptMixin._encrypt
  File "pkcs11/_pkcs11.pyx", line 896, in pkcs11._pkcs11.EncryptMixin._encrypt
  File "pkcs11/_errors.pyx", line 88, in pkcs11._pkcs11.assertRV
MemoryError: Buffer was too small. Should never see this.

[williamcroberts]

The _pkcs11.pyx file seems to want to call C_Encrypt twice, once with a NULL buffer to get size, and then again with the size allocated.....

err duh, I broke the 5.2 return convention from pkcs11 when changing some internals... let

Not your problem! But I learned a ton about the python bindings, thats some cool stuff.