`structuredClone` polyfill is broken

[romainmenke]

see : https://github.com/Financial-Times/polyfill-library/actions/runs/3981951365

pr : #1242

@CheloXL @JakeChampion It seems that there are some differences between browser versions in serialize/deserialize.

In Chrome 48 for example one part is a string and the other is true null :

Does anyone have time to resolve these issues?
If not, we should revert these changes until someone does have time.

[romainmenke]

config.toml is also not fully correct.
Fixed stats :

[browsers]
android = "*"
bb = "*"
chrome = "<98"
edge = "*"
edge_mob = "*"
firefox = "<94"
firefox_mob = "<94"
ie = "*"
ie_mob = "*"
opera = "<84"
op_mob = "<68"
op_mini = "*"
safari = "<15.4"
ios_saf = "<15.4"
samsung_mob = "<18.0"

[CheloXL]

@romainmenke I don't have any problems trying to fix that, but I don't know how to debug on chrome 48... How could I install that specific version on Windows?

[romainmenke]

Thank you for the speedy reply @CheloXL :)
I don't know how to install Chrome 48 or other old browser versions on Windows.

We use https://www.browserstack.com to test everything.
They have a desktop app to support local testing : https://www.browserstack.com/docs/local-testing.

I am not sure if they have a free plan or if there is a comparable service that does have a free plan :/

[CheloXL]

I will see if I can use the free tier, but from what I saw, I only have 1 minute to try to debug... not very useful. I already fixed the config.toml in my branch. Will let you know if I can fix the error.

[CheloXL]

@romainmenke Ok, I think I fixed the issue. Pull request

[romainmenke]

@CheloXL Thank you for that!

Unfortunately it still fails at the same point when I run the tests.
I also haven't checked any of the other failures yet, I only checked Chrome 48

I tried to debug it but the original polyfill doesn't have any comments to explain the purpose of parts and is written in a style that isn't the most readable for me.

Reverting this would take the pressure of :)
Thoughts?

[CheloXL]

@romainmenke Please do the revert. I would like to fix any issues, but I'm not sure how to continue. I already have a VM with chrome 48 installed and was able to reproduce the original issue (RegEx has no .flags property so I had to check individual flags g, i, m). After that fix, everything was working fine.
I can't use Browserstack. The free version allows use for one minute, and that doesn't give me time to even try to load the file.

As for you trying to debug the polyfill: Did you try to debug the original one (the one from ungap) or the one I re-wrote? You can't use the original one as it uses modern js features (for..of, arrow functions, etc) so it would not work at all.

Can you point me where the test fails now? (line# of the failing test). Maybe now there is another problem.

[romainmenke]

(RegEx has no .flags property so I had to check individual flags g, i, m). After that fix, everything was working fine.

The tests in CI don't get that far, they already fail on the checks for null : https://github.com/Financial-Times/polyfill-library/actions/runs/4031468826/jobs/6931056567

I already have a VM with chrome 48 installed

It is unexpected that this Chrome 48 is able to get past the test for null when our CI already fails at this point.

As for you trying to debug the polyfill: Did you try to debug the original one (the one from ungap) or the one I re-wrote? You can't use the original one as it uses modern js features (for..of, arrow functions, etc) so it would not work at all.

Trying the one you re-wrote, but I also briefly looked at the original.
I think it wasn't written for old browsers and also never tested in any browsers.
If you look at their CI it shows they only test node. (maybe tested outside of CI?)

Can you point me where the test fails now?

fails here : polyfills/structuredClone/tests.js:69

proclaim.equal(deserialized['null'], null);

---

Also want to thank you again for submitting the polyfill in the first place and for working so hard on a fix :)

[CheloXL]

Ok... this is strange. I was able to test in browserstack (online). Tested both in Chrome/Windows and Firefox/Windows. Chrome version 48/47/46. Firefox 34/33/32. Only found an issue in Firefox (that version doesn't has string.includes, so I replaced it with indexOf). But after that, everything passes.
Please note that for this I had to create a single html file with proclaim, the polyfill and the tests included. I'm attaching the file just in case it's useful.
I'm really blind now. I don't know why null != null (I would understand null != undefined). Not sure if it's a problem (feature?) of the platform you are using to run the tests, or what else...

sc1.zip

[romainmenke]

null != null is actually null != "null".
The log output is not the most helpful in this case.

But I found that by adding a few log statements with typeof at that point in the tests.
I have no idea why null would become a string or which part of the polyfill might be doing this.

[CheloXL]

mmm. The test for that is proclaim.equal(deserialized['null'], null), and if what you said is in the correct order, that means that deserialized['null'] is actually null (so the serialization/deserialization works), but the null part is coerced to string?

Or: The polyfill uses both Map/Set. Maybe the tests are including them even if Chrome 48 doesn't need them, and that's the problem?

Can you run the above file and see if you see the same error? The only thing that I can think that could be a problem is the name of the field in the object (that it is "null"). Maybe you can try to change that in the test ("null": null, to "isNull": null,) and later on proclaim.equal(deserialized['null'], null); by proclaim.equal(deserialized.isNull, null);.

Sorry to bother you with this, but I don't have a way to reproduce this. Again, running the above file through BrowserStack produces the right output.

[romainmenke]

I don't know why yet, but when I remove this bit it doesn't error :

	var bi = null;
	if ("BigInt" in window && typeof BigInt == "function"){
		try {
			bi = BigInt(1);
		} catch (e) {
			//no BigInt support.
		}
	}

If I remove that and all references to bi then there is only one null value.
I don't fully understand the polyfill, so I can't explain this.

---

running the above file through

Have you been able to get the local dev server running with npm run watch?
This ensures you are testing the polyfill with all dependencies.

[romainmenke]

/* eslint-env mocha, browser */
/* globals proclaim, structuredClone */

describe('structuredClone', function () {
	it('is a function', function () {
		proclaim.isFunction(structuredClone);
	});

	it('has correct arity', function () {
		proclaim.arity(structuredClone, 1);
	});

	var bi = null;

	var obj = {
		bigint: bi,
		"null": null
	};

	var deserialized = structuredClone(obj);
	console.log(deserialized);

	it('serializes values', function () {
		/* eslint-disable-next-line dot-notation */
		proclaim.equal(deserialized['null'], null);
	});
});

logs :

[romainmenke]

Why is it even using Map?
🤔 Is this some attempt to make it fast?

---

At least not to make it fast, has a stack overflow without the Map usage.

[CheloXL]

The map is to preserve references when cloning. And yes, without the map, you are cloning the same object to the infinity and beyond... :)

[CheloXL]

Ok. Found the problem: I'm not sure why, but on Chrome 48, the tests are loading AND executing the Map/Set polyfills. As I guessed, the problem is in there, and specifically with the keys null and 'null'.

I don't know why the Map/Set polyfills are beign loaded and are replacing their native counterparts. If I run the test manually (as shown on the file attached above, where no Map/Set PF are included), it works as expected. Also running the tests from the local url server on my latest Chrome version works fine (Map/Set PF are included, but they do not replace native Map/Set).

So, after changing the field name from 'null' to nullValue, that test passed. Then I found another problem (related to the non-native map/set implementation) and fixed that too.

Now it should be working (at least all tests passes).

[romainmenke]

I don't know why the Map/Set polyfills are beign loaded and are replacing their native counterparts.

That is how the polyfill library works :)
It covers more than just missing basic features.
Some features had bugs in the initial implementation, others had sub features added later.

So, after changing the field name from 'null' to nullValue, that test passed.

Is there a way to make this test pass with 'null' as the field name?

I also want to know and understand why this is happening.
Why is null becoming "null", which code is causing this?
Is this a bug in this polyfill or somewhere else?

Without fully understanding this I don't feel comfortable shipping this polyfill.

[CheloXL]

Is there a way to make this test pass with 'null' as the field name?

I don't think so, as the problem is in the Map polyfill.

I also want to know and understand why this is happening. Why is null becoming "null", which code is causing this? Is this a bug in this polyfill or somewhere else?

This is happening because the pf uses a Map to store references to values, so on deserialization it is able to preserve those references. When serializing, it first stores the key 'null' with an Id (the "position" of that value in the Map). Later on, it tries to store the value null. If you check the pf code, you will see that the first thing it does when serializing is trying to obtain a reference id from the map. If a reference is found, that ID is returned. Now, for the Map PF, both keys 'null' and null evaluates to the same index. On deserialization then, since it is preserving references, null becames 'null' because the second one was stored first (name of the field in the object).

It is not a problem on the polyfill, and I think it is a really contrived use case of Map (using null as a key). I see this more as a limitation of the Map polyfill.

So, basically the problem is the following:

var map = new Map();
map.set('null', 1);
map.set(null, 2);
console.log(map.get(null) === 2);// should be true, but with Map PF returns false.

[romainmenke]

Thank you for this!
Yes, I can now reproduce this purely in the tests for Map.

I will take a look at this and try to find a fix.

[romainmenke]

Pretty much all keywords have this issue.
true and false are equally affected.

[CheloXL]

Probably because of the underlining implementation of the Map. I didn't check, but I guess it is using an object to store the key/value pairs, and since an object only accepts strings as keys, that's where the problem is.

But again, I'm just guessing...

[romainmenke]

exactly that is happening yeah..

// If this is just a primitive, we can cast it to a string and return it
return ''+recordKey;

So there is no difference between true and 'true'

[romainmenke]

I think this will resolve the issues : #1275

Thank you for providing all this extra info 🙇

[romainmenke]

Issue is fixed, thank you for working with me on this one @CheloXL 🙇
I will prepare a release as soon as possible.

[CheloXL]

@romainmenke Thanks to you for your great support!!!