MMTk binding for Ruby

This repository hosts the binding code for MMTk Ruby.

In order for this binding to work, changes have been made to the Ruby core language to support generic third party heaps. Eventually, the aim is to upstream these changes. Until then, the modifications can be found under our fork here, on the branch third-party-heap.

This repository is based on previous work of Angus Atkinson, and the original repository can be found here, and the original Ruby fork can be found here.

Building/installation instructions

Checkout repositories

You need to clone both the Ruby fork and the MMTk Ruby binding. The location does not matter.

git clone https://github.com/mmtk/ruby.git
git clone https://github.com/mmtk/mmtk-ruby.git

Build the MMTk binding, first.

pushd mmtk-ruby/mmtk
cargo build --release
popd

This will give you a libmmtk_ruby.so in the target/release directory.

By default, mmtk-ruby uses the mmtk crate from the master branch of its official repository. If you want to hack the MMTk core itself, you can edit mmtk-ruby/mmtk/Cargo.toml to point to your local repository.

Then build our forked Ruby repository.

Configure.

cd ruby
./autogen.sh
./configure --with-mmtk-ruby=../mmtk-ruby --prefix=$PWD/build

Build a miniruby executable. We need to set some environment variables first.

export MMTK_PLAN=MarkSweep                  # Set the GC algorithm to MarkSweep
export THIRD_PARTY_HEAP_LIMIT=500000000     # 500M should be enough. If still out of memory, add more.

make miniruby -j

The miniruby executable should be able to execute simple Ruby programs. You can try the following command:

./miniruby -e 'puts "Hello world!"'

You can continue to build the full Ruby and install it with

make install -j

Debug build

You can build with debug options enabled for easy debugging.

Remove the --release option to build mmtk-ruby for debug. Note that the Cargo build system is smart enough to let the result of debug build and release build to coexist in the target/debug and target/release directories.

pushd mmtk-ruby/mmtk
cargo build
popd

By default, ./configure searches for libmmtk_ruby.so in mmtk-ruby/mmtk/target/debug, and the linker will subsequently link miniruby and ruby to that .so. Add --with-mmtk-ruby-debug so it will search for libmmtk_ruby.so in mmtk-ruby/mmtk/target/debug, instead.

Set the compiler option -DRUBY_DEBUG=1 to enable most assertions in Ruby.

Add the -O0 optimization flag so the debug can see the values of most local variables. But if it is too slow, try -O1 instead.

The --disable-install-doc will disable the generation of documentations. It can make the build process much faster.

The following is an example of configuring for debugging.

./configure --with-mmtk-ruby=../mmtk-ruby --with-mmtk-ruby-debug cppflags='-DRUBY_DEBUG=1' optflags='-O0' --prefix=$PWD/build --disable-install-doc

Also note that you can use the release build of mmtk-ruby with a debug build of ruby (with optflags='-O0' but without --with-mmtk-ruby-debug) and vice versa (with --with-mmtk-ruby-debug and optflags='-O1' or higher).

Test

To test Ruby, it is recommended that you add the ADDITIONAL_EXCLUDES option to exclude tests which make assumptions based on Ruby's current GC implementation, or are extremely memory intensive.

make test-all ADDITIONAL_EXCLUDES="--excludes-dir=./test/excludes/_third_party_heap"

Build with Vagrant

A Vagrant config for building on platforms other than Linux can be found here: https://github.com/chrisseaton/mmtk-ruby-macos

Current status

Known working:

./miniruby ./basictest/test.rb
make test
Regular Ruby programs (note: this hasn't been tested on any extensive real-world programs, only <100 line dummy programs)

Basic Rails 5 app utilising a sqlite3 database:

gem install rails -v 5.2.0
rails new hello
cd hello
rails generate scaffold User name:string email:string
rails db:migrate
rails server

# If you run into installation issues along the way, you may need to...
gem install sqlite3
gem install puma
# ...and try again

Known issues:

make test-all fails. Many tests are GC implementation-dependent, so exclusion files have been created to ignore most of these. There are still >50 errors, but not all have been triaged or fixed yet.
make test-rubyspec is currently failing; need to find a way to exclude GC-specific specifications.
GC implementation-specific modules (e.g. ObjectSpace, GC, WeakRef) and anything that relies on them (e.g. Coverage) are not supported. For now, there are no plans to implement these as many of the APIs are irrelevant (e.g. GC.stat); however some may be fixed in the future (e.g. ObjectSpace.each_object)
MJIT is not supported.

TODO

Add a runtime flag to enable MMTk/ruby, using environment variables. See #1
Rebase my changes onto the variable-sized objects heap (currently being developed by Shopify here)
Use separate mutators for every thread for cache locality benefits (and correctness in case the GVL is ever removed)
Implement allocation fast paths
Fix tests

Licensing

This work is dual-licensed under the MIT and Apache licenses, to be compatible with the MMTk-Core project. See the license notices in the root of this repository for further details.

wks/mmtk-ruby