cloudflare/stpyv8

Upgrading notes for V8 v8.3.104

darwin opened this issue · 8 comments

darwin commented

I spent some time tweaking setup files and building debug version of STPyV8 and linking it with debug version of V8. For better stack traces, crash logs and debug checks.

Then I decided to upgrade V8 to latest. This turned out to be a good exercise and I discovered a few issues in the old code. See this commit range: darwin/naga@4dddc49...d09c0c4

The challenging part was to get debug V8 build play nice with debug build of STPyV8. It turns out that both builds must be based on the same macro defines and similar compiler options. If you get this wrong the library might still link (symbol names match) but the code will be crashing under rather puzzling circumstances (e.g. structure sizes, member alignments - do not match).

For now, for debug build I just copy&pasted settings from ninja-generated files during my debug V8 build:
darwin@a1e1710

But this is very fragile. Ideally STPyV8 should be built with the V8 toolchain using the same compiler+flags.

buffer commented

Thanks! Really appreciate!

FYI I have had not a great experience some days ago while attempting to port STPyV8 to V8 version 8. I will take a deep look at your patches later today. And again thanks for that. Meanwhile just wanted to share that V8 pointer compression [1] was causing me a lot of headaches. Disabling this feature when compiling V8 version 8 was enough to fix some (but not all) of the issues I have been observing.

[1] https://v8.dev/blog/pointer-compression

darwin commented

Today I'm going to look at V8's build system and take a stab at using it. I will use Electron project as a template.

I believe this should have several benefits:

  1. building the .so binary will be possible via a shell script without dependency on setup.py and working python infrastructure on your machine
  2. all macros and compiler options will properly match
  3. incremental builds will work
  4. it will be possible to tweak any V8 options, also allow to introduce ours on top
  5. windows build for free? and other cross-compilation targets should work same as with V8 (if python libs allow)
buffer commented

I was taking a look at your local fork. That's impressive. Great work! I was thinking about spending some time testing it. Should I use your repo as it is or you think it's better to create one or more branches for testing purposes?

darwin commented

Thanks. I have no preference, you can use as it is, I have added you as collaborator there. Or fork it and request some PRs.

Please note that there will be a lot of churn over couple few days. I will be refactoring the sources.

darwin commented

Also please see this proposal: https://github.com/darwin/stpyv8/blob/master/docs/dev.md

There are two remaining issues I currently see with the pybind port:

  1. __iter__ implementation, currently commented out, no tests test them directly, didn't look at iterators in pybind yet
  2. refcounts are different with pybind, not sure if it is a bug on my side or pybind is holding some references internally - needs investigation, I commented out the tests
darwin commented

It would be great if you did some testing and and tried the new gn build system. Also tested/added support for linux, if it is your os.

buffer commented

The pybind proposal is very interesting. Yes I was thinking about starting performing some tests of your repo in a Linux environment while you're going to refactor the sources. FYI I use both Linux and OSX here so I can perform the same tests on OSX as well. I was thinking about using Thug [1] for such purposes. Currently Thug unit test suite is quite large so it could be a first interesting testbed.

[1] https://github.com/buffer/thug

  • 👍1
darwin commented

We can move this conversation to email/chat. I guess it will be faster for you to hit me with questions if you encounter any problems. The build steps are described here: https://github.com/darwin/stpyv8/tree/master/gn

drop me an email to my git commit email and we can move from there