SHA256 is hard-coded during RSA signature preimage calculation

SHA256 is hard-coded here. If I understand well, RS512 should use SHA-512 instead.

Line 70 in 6d8c0bb

sha256_0=SHA256.new(jks0_input.encode('ascii'))

The provided examples use 2048-bit RSA keys and switch them to SHA-256, demonstrating that hash lengths and bit sizes of asymmetric keys are unrelated (the vulnerable libs work on serialized keys anyway). HMAC can consume arbitrary sized keys, the output length will be the same.

If you still see this as a problem, feel free to reopen with additional explanation.

The code did not work well with RSA512. N would always be 1. After the pull request #2 , we could recover the public key.

In the JWT token The header+payload is hashed with the wrong hash function so m0 and m1 is miscalculated. When using RS512, one would use SHA512 with the padding

You were talking about this SHA256 in HMAC: https://github.com/silentsignal/rsa_sign2n/pull/2/files#diff-1c91abcb7d462a6f1e27dc67c304a3fffb6c437ccc0e5bdc045eabb4f44ea90bR45

I was talking about the SHA used in RSA.

I think I can see the problem now, thanks!

In summary, RSA signatures are calculated on the SHA hash of the canonized message, but the original JWT's can use different hash algorithms here. We have to recalculate the hash to be signed with the appropriate SHA version in order to the GCD calculation (based on the hashed messages) work correctly.

I will reopen this and add some comments to the PR!