Why Dart sha512 and js sha512 has different result?

Question

Why Dart sha512 and js sha512 has different result?

mayqiyue opened this issue 6 years ago · 20 comments

mayqiyue commented 6 years ago

Answer 1 · 2019-08-17T18:35:25.000Z

Do you have any information to go on?

Answer 2 · 2019-09-02T17:22:02.000Z

I think I've got the same issue with Dart and C#...

Answer 3 · 2019-09-03T09:27:43.000Z

I've just tried the same code with the plugin pointycastle instead, and it work fine.
I hope you'll find a fix :)

Answer 4 · 2019-09-03T22:43:43.000Z

Can you please give us some data to go on? What did you try to hash, what was the expected outcome, what did you get instead? What version of the DartVM were you using?

Answer 5 · 2019-09-04T08:53:59.000Z

Can you please give us some data to go on? What did you try to hash, what was the expected outcome, what did you get instead? What version of the DartVM were you using?

Unfortunately as it is used for the auth security protocol of my client I cannot share the full code.
But here you can find a part of it

/// dart-lang/crypto method
var digest = List<int>();
for (int i = 0; i < repeatCount; i++) {
  digest = sha512.convert([...digest, ...saltedPassword]).bytes;
}

/// pointycastle method.
var sha512Digest = new SHA512Digest();

var digest = Uint8List(0);
for (int i = 0; i < repeatCount; i++) {
  digest = sha512Digest.process(Uint8List.fromList([...digest, ...saltedPassword]));
}

with repeatCount few hundreds.

I'm using last stable Flutter with Dart v2.4.0

I hope it will help you fix it, thanks for your time !

Answer 6 · 2019-09-06T02:57:47.000Z

I just imported both crypto and pointycastle into a program just like your examples:

main() {
  final int rounds = 2000;
  final saltedPassword = "now this is a strong password!".codeUnits;

  var digest = <int>[];
  print("crypto");
  for (int i = 0; i < rounds; i++) {
    digest = sha512.convert([...digest, ...saltedPassword]).bytes;
  }
  print(toHexPad(digest));

  final sha512p = SHA512Digest();
  var pointyDigest = Uint8List(0);
  print("pointy");
  for (int i = 0; i < rounds; i++) {
    pointyDigest = sha512p
        .process(Uint8List.fromList([...pointyDigest, ...saltedPassword]));
  }
  print(toHexPad(pointyDigest));
}

And as you can see the output is identical after 2000 rounds, and we're 57ms faster (64bit intel).

pub run main.dart
crypto
5f781fba196e4bacdd09ba4aa7f1b34d127bed3bfd952734085d1f16a23d5e95cc948ad7f978af6bc1cb45db127fad22f58ce049a319bf8da86d23a0aec8ef22
pointy
5f781fba196e4bacdd09ba4aa7f1b34d127bed3bfd952734085d1f16a23d5e95cc948ad7f978af6bc1cb45db127fad22f58ce049a319bf8da86d23a0aec8ef22

Answer 7 · 2019-09-06T03:21:23.000Z

I also just built a quick and dirty Flutter program... and the results are exactly the same with dart:crypto beating pointy castle 93ms to 260ms.

Answer 8 · 2019-09-06T03:24:54.000Z

Please note, I added the nist.gov's monte carlo tests for all sha-2 digests, and we match the expected outputs.

Answer 9 · 2019-09-06T06:35:07.000Z

Thank a lot for your time and test. I will take the time to investigate this more precisely too soon, I'll be back

Answer 10 · 2019-09-07T04:45:20.000Z

Hey, @Nico04 - Is C# Using SHA-3 512 or SHA-2 512?

Answer 11 · 2019-09-07T07:01:17.000Z

Not sure, but here the officiel doc :
https://docs.microsoft.com/en-us/dotnet/api/system.security.cryptography.sha512managed?view=netframework-4.8

And a post about that :
https://stackoverflow.com/questions/24098253/difference-between-hashlib-and-system-security-cryptography-hashalgorithm

Hope it'll help

Answer 12 · 2019-09-08T01:56:12.000Z

Thanks for the docs - are you using the Hashlib version or the system.security one? From the second link:

SHA-512 is SHA-2 512 bit - not the same as SHA-3 512. That said, to use it, you simply import System.Security.Cryptography - using in this case imports the namespace - making the classes inside the namespace available to your code.

We only have the SHA-2 512 version.

Answer 13 · 2019-09-08T09:27:28.000Z

I'm using the first one - System.Security.
So if I understand I'm using Sha-2 512 which is the same as this plugin, so it should output the same result right ?

Answer 14 · 2019-09-08T15:20:13.000Z

Yes, but I'd need more information since I believe my test code above matches your pseudocode above.

Answer 15 · 2019-09-08T16:35:08.000Z

I understrand, please let me few days to build some sample code.

Answer 16 · 2019-10-01T14:54:08.000Z

It took me few days, but here is the small sample that shows input that works (both methods return the same) and input that doens't (cryto output doesn't work, whereas PointyCastle does).

var saltedPassword = "now this is a strong password!".codeUnits;    //This works OK
saltedPassword = utf8.encode("now this is a strong password!"); //This works OK
saltedPassword = utf8.encode("AAAA{SKJSQDGQSGDKJGSDKJGSDKJQSGDKGDKJQGSDKJQGSDKJQGSKDJGQSD"); //This works OK
saltedPassword = utf8.encode("AAAA{3FXhiiyc5gGWlRrVQ2RlJ.6xj.DKvf6l0bJxqh0BzA}"); //This DOESN'T work
saltedPassword = utf8.encode("AAAA{3FXhiiyc5gGWlRrVQ2RlJDKvf6l0bJxqh0BzA}"); //This works OK
saltedPassword = utf8.encode("AAAA{3FXhiiyc5gGWlRrVQ2RlJ.6xjDKvf6l0bJxqh0BzA}"); //This works OK
saltedPassword = utf8.encode("AAAA{3FXhiiyc5gGWlRrVQ2RlJ6xj.DKvf6l0bJxqh0BzA}"); //This works OK
saltedPassword = utf8.encode("AAAA{3FXhiiyc5gGWlRrVQ.2RlJ6xj.DKvf6l0bJxqh0BzA}"); //This DOESN'T work
saltedPassword = utf8.encode("AAAA{rVQ.2RlJ6xj.DKvf6l0}"); //This works OK
saltedPassword = utf8.encode("AAAA{rFXhiiyc5gGWlVQ.2RlJ6xj.DKvf6lFXhiiyc5gGWl0}"); //This DOESN'T work
saltedPassword = utf8.encode("AAAA{rFXhiiyc5gGWlVQ.2RlJ6xj.DKvf6lFXhiiyc5gGWl0}ds54qQSD"); //This works OK

//----- Pointy Castle -----
var sha512Digest = new SHA512Digest();

var digest1 = Uint8List(0);
for (int i = 0; i < 2000; i++) {
  digest1 = sha512Digest.process(Uint8List.fromList([...digest1, ...saltedPassword]));
}

var resultPointyCastle = base64.encode(digest1);


//----- crypto.sha512 -----
var digest2 = List<int>();
for (int i = 0; i < 2000; i++) {
  digest2 = sha512.convert([...digest2, ...saltedPassword]).bytes;
}

var resultCrypto = base64.encode(digest2);


var isEqual = resultCrypto == resultPointyCastle;
print(isEqual);
print(resultPointyCastle);
print(resultCrypto);

I hope it is enought for you to find out ?
I tried to understand the patern, but I didn't found out...
Thanks

Answer 17 · 2019-10-15T04:23:39.000Z

Sorry it's taken me a while to follow up; I converted your test to the following code. This shows no errors on one loop (yay), but reports errors on round 2. Interestingly, it's only 512 that fails... that gives me a place to dig, but it is also not a good sign as 384 and 512 share the same base algorithm.

import 'dart:convert';
import 'dart:typed_data';

import 'package:ansicolor/ansicolor.dart';
import 'package:crypto/crypto.dart';
import 'package:pointycastle/digests/sha224.dart' as p;
import 'package:pointycastle/digests/sha256.dart' as p;
import 'package:pointycastle/digests/sha384.dart' as p;
import 'package:pointycastle/digests/sha512.dart' as p;
import 'package:pointycastle/api.dart' as p;

main() {
  final red = AnsiPen()..red();
  final green = AnsiPen()..green();
  final pens = {
    false: red('fail'),
    true: green('pass'),
  };

  var salts = [
    "now this is a strong password!".codeUnits, //This works OK
    utf8.encode("now this is a strong password!"), //This works OK
    utf8.encode(
        "AAAA{SKJSQDGQSGDKJGSDKJGSDKJQSGDKGDKJQGSDKJQGSDKJQGSKDJGQSD"), //This works OK
    utf8.encode(
        "AAAA{3FXhiiyc5gGWlRrVQ2RlJ.6xj.DKvf6l0bJxqh0BzA}"), //This DOESN'T work
    utf8.encode("AAAA{3FXhiiyc5gGWlRrVQ2RlJDKvf6l0bJxqh0BzA}"), //This works OK
    utf8.encode(
        "AAAA{3FXhiiyc5gGWlRrVQ2RlJ.6xjDKvf6l0bJxqh0BzA}"), //This works OK
    utf8.encode(
        "AAAA{3FXhiiyc5gGWlRrVQ2RlJ6xj.DKvf6l0bJxqh0BzA}"), //This works OK
    utf8.encode(
        "AAAA{3FXhiiyc5gGWlRrVQ.2RlJ6xj.DKvf6l0bJxqh0BzA}"), //This DOESN'T work
    utf8.encode("AAAA{rVQ.2RlJ6xj.DKvf6l0}"), //This works OK
    utf8.encode(
        "AAAA{rFXhiiyc5gGWlVQ.2RlJ6xj.DKvf6lFXhiiyc5gGWl0}"), //This DOESN'T work
    utf8.encode(
        "AAAA{rFXhiiyc5gGWlVQ.2RlJ6xj.DKvf6lFXhiiyc5gGWl0}ds54qQSD"), //This works OK
  ];
  int maxCount = 2;

  final digestors = [
    DigestTest('224', p.SHA224Digest(), sha224),
    DigestTest('256', p.SHA256Digest(), sha256),
    DigestTest('384', p.SHA384Digest(), sha384),
    DigestTest('512', p.SHA512Digest(), sha512),
  ];

  for (final digestor in digestors) {
    print("\n\nDIGEST: ${digestor.name}");
    for (final salt in salts) {
  //----- Pointy Castle -----
      var digest1 = Uint8List(0);
      for (int i = 0; i < maxCount; i++) {
        digest1 = digestor.pointy.process(Uint8List.fromList([...digest1, ...salt]));
      }

      var resultPointyCastle = base64.encode(digest1);

  //----- crypto.sha512 -----
      var digest2 = List<int>();
      for (int i = 0; i < maxCount; i++) {
        digest2 = digestor.crypto.convert([...digest2, ...salt]).bytes;
      }

      var resultCrypto = base64.encode(digest2);

      var isEqual = resultCrypto == resultPointyCastle;
      print(
          '${pens[isEqual]}: salt(${utf8.decode(salt)}):\n pointy v. crypto:'
          '\n  $resultPointyCastle\n  $resultCrypto');
    }
  }
}

class DigestTest {
  final String name;
  final p.Digest pointy;
  final Hash crypto;
  DigestTest(this.name, this.pointy, this.crypto);
}

Answer 18 · 2019-10-15T05:39:42.000Z

First hunch: Looks like it's the padding generated higher up. They account for only 224/256 needing 64bit, where 384/512 requires 128bit of space at the end.

Answer 19 · 2019-10-15T06:02:09.000Z

Yep; its ugly, but I was able to pass after hacking around only in the _finalizeData. I'll work on a patch for later this week.

Answer 20 · 2020-02-14T10:40:14.000Z

I've tested the patch and confirm it works (Sorry for the delay).
Moreover, it's way faster than the pointycastle one 👍
Thanks :)