v0.7.0, GitHub Organization, and Merging Atomic

[jdantonio]

Charles Nutter, aka @headius, has graciously offered to merge his (very successful) ruby-atomic and thread_safe gems into ours. I think we should accept his offer. He has done great work and has been an avid promoter of this gem. If we want to pursue this, we need a plan. Let's use this thread to discuss the roadmap. There are several open issues we need to discuss and make decisions about. Below are my initial thoughts, in no particular order. All input is welcome.

0.6.0 Release

As I discuss in this thread, I think we should release 0.6.0 ASAP. There is a ton of great stuff waiting to be released to the community.

0.7.0 Release Date, with Atomic

I think we should target early August for the release of 0.7.0 with ruby-atomic merged. @headius will be speaking at Steel City Ruby in Pittsburgh, PA on August 15. Pittsburgh is only 2 hours away, by car, from where I live in Akron, OH so I plan to attend (and I've submit a talk proposal so with a little luck I may be speaking, too). This would be a great opportunity to announce the gem merge and promote our work.

Assuming we target this date for the next release, what else should we include? Two things that come to mind immediately (in addition to merging ruby-atomic) are new actors and C extensions.

GitHub Organization

@headius has also suggested we create an organization within GitHub for this gem. As much as I love having my name in the URL, this has become a true team effort. The gem wouldn't be where it is today had I worked alone so I'm happy to move the repo from my personal account to an organization. If we do that, however, we need a name for the organization. Any suggestions?

One Gem, or a Family of Gems

Right now we have about 3,700 lines of code, not counting tests. We plan to extend actors significantly and also merge in up to two additional gems. Has this gem grown too large? The gem composition seems "right" to me because we have a very robust toolkit that is very nicely layered from high-level abstractions (like Async and Future) through mid-level tools (like IVar) down to low-level, optimized utilities (like AtomicFixnum). But we also have a few groups of related functionality that could be pulled out into separate gems. I'm not sure how I feel about keeping everything in one gem versus creating a small family of related gems. My only concern is excessive fragmentation. We could easily do ourselves a disservice if we create too many. One possible way to create multiple gems would be:

• Atomics
• Actors (later, once they grow in scope)
• Everything else

External Dependencies

In another thread we discusses the inclusion of external gem dependencies. We currently have none. My preference is to remain that way. My impression of that discussion is that the majority of the team agree. One result of this discussion was the offer to merge ruby-atomic into our gem. Moving forward we should have consensus on this issue.

Native C and Java Extensions

We've already added several JRuby optimizations and we recently began experimenting with C extensions. We have not made a decision regarding C extensions by my preference is to use them when they make sense. If we merge ruby-atomic into our gem we've answered our question because it has a native C implementation.

One of the greatest strengths of this gem, I believe, is that we have always been interpreter-agnostic. We have pure Ruby implementations of everything. We only use Java to optimize. Our audience is the entire Ruby community, not a subset of it. I believe we should continue with this goal and very diligently follow it.

I do not know if ruby-atomic and thread_safe have pure Ruby implementations. If they do not and we choose to merge them I believe we should update the implementations to follow the "pure Ruby with native optimizations" approach that's served us so well this far.

[chrisseaton]

What would people think about a policy of being an 'unopinionted' gem? As @jdantonio says we're a toolbox for concurrency, and in some cases that means some overlap between tools, and some aliases so people can find the method they're used to, but I think that's good. For example people can come to us for both promises and futures, without being lectured about which is better or why.

I think not having dependencies is part of being unopinionted: you don't have to use anything else to use this gem, and it's not a stack or a framework.

So I'm in favour of unopinionted and of no dependencies.

I'm also of favour of one big tool box at the moment - one gem. If we start dividing up I fear we'll keep going and end up with one gem per construct and a terrible nest of dependencies between them.

[lucasallan]

I like that the gem doesn't have dependencies, I think that so far it makes sense. But we should also keep our mind open in case we need to add dependencies - we don't want to recreate everything, right?

[mighe]

One Gem, or a Family of Gems

I like the idea of extracting actors from the lower level gem. I feel actors are an higher level abstraction that can be built over atomics-ivar-futures-etc and they are also maintained by a slightly different team (usually @chrisseaton and I are working on the lowest levels, but not on actors, for example)

External Dependencies

I agree with @chrisseaton and @lucasallan: we should avoid external dependencies. If we keep our gem well focused, after the @headius merge, we probably will not need anything else.
If in future we will create something requiring external gems, it will probably be another gem (like the actor in my previous point)

Native C and Java Extensions

I agree with you @jdantonio, our gem should be for everyone and Java or C extensions are only a way to optimize the code

GitHub Organization

What about a simple name like "concurrent-ruby"?

[jdantonio]

I love the idea of being explicitly 'unopinionted'. For all the reasons @chrisseaton stated. If other developers want to build opinionated gems the wealth of tools we provide will make their job easier. We'll just keep building cool tools.

I agree with @mighe that if we were to pull anything out into a separate gem, it would be actors. Because there is no single canonical implementation of the actor model, any actor implementation is inherently opinionated. But I also think there is value in providing a simple actor implementation as part of a robust and varied toolkit. My vote is to keep actors for now and reconsider pulling that out at a later date if our implementation grows sufficiently large.

I'm fine with https://github.com/concurrent-ruby/concurrent-ruby for the organization/gem name if everyone else is OK with it. I know the name is fairly bland, but I think that's part of its charm. We aren't trying to be cute or edgy or hipster. To @chrisseaton's point, we're simply trying to create a robust, unopionated set of tools that, as @mighe stated, is for all Rubyists. The name may be unexciting but it perfectly states what we do and what we provide. It's unambiguous and unpretentious.

[headius]

Ok a few notes.

• At some point I tucked away https://github.com/ruby-concurrency for a rainy day. If we want it, I'll turn it over to the group. Otherwise I'll probably delete it.
• I like unopinionated too. java.util.concurrent can be confusing with many overlapping tools (Semaphore? Lock? CountdownLatch? SynchronousQueue?).
• I have not provided an abstract for my Steel City talk yet. My usual talk this year has been on concurrency, trying to promote concurrent-ruby and other libraries, as well as talking about the why of concurrency. I'm guessing @jdantonio submitted a talk on concurrent-ruby? Perhaps if your talk is accepted, we can figure out how mine can dovetail with it like maybe I give people the why and then direct them to your talk for the how).

Once we have an organization set up I'll move the atomic and thread_safe repositories in prep for merging. Once merged, we should do a final atomic release that just depends on concurrent-ruby, perhaps with an install-time message letting people know about the merge.

[schmurfy]

I also like unopinionated as much as possible but concerning the "no dependencies" I can only repeat what I said on the other issue: If having no dependencies means swallowing everything that could be a depdency inside this gem then I think that's an horrible idea.

I think everything that may be used separately should stay outside (without going to the extremes), it's not like there much overhead to load two gems instead of one in any project and that way the basic pieces can still be used separately.

Part of my current work is development on small linux devices (not quite embbeded but still limited) and it's incredible the size your gems end up using if you are not careful and since, sadely, many gems have what I consider "stupid" dependencies you end up having to fork them or extract the code you need to keep a low profile.

As I said, just look at activesupport, there is a lot of useful things I won't deny that but when you use it in your project it feels like you use a nuclear reactor to light a candle... Is it really the path you want this gem to take ?

[jdantonio]

The plan, then, is:

• Update the README and wiki with our goal of remaining "unopinionated"
• Update the README and wiki with our goal of remaining dependency-free, but reconsider in the future if the need arises
• Release 0.6.0 ASAP without the ActorContext spike.
• Create a "concurrent-ruby" organization in GitHub and move the repo (probably immediately after the 0.6.0 release)
• Give @headius commit access to the new organization so he can move his gems whenever it is convenient for him
• Create a roadmap for the 0.7.0 release, targeting the week before Steel City Ruby
• Plan the following updates for 0.7.0
  • New actor implementation
  • Pure-Ruby implementation of atomic references that matches the API of ruby-atomic
  • Merge of ruby-atomic that follows our current practice of "pure Ruby + platform-specific optimizations"
  • Merge of the experimental CRuby extensions
  • Gemspec configuration that is platform-aware and does not attempt to compile C extensions on unsupported platforms (based on the C Ruby experimental branches and the setup from ruby-atomic)
  • Update all Ruby mutex operations to follow the newly-agreed upon technique of explicit lock/unlock with ensure clause
  • Any other cool stuff we have time to work on between now and then

Is there anything I've missed?

[chrisseaton]

Sorry FJ probably won't make it this week - I have a lot of stuff at work now.

[pitr-ch]

• https://github.com/concurrent-ruby/concurrent-ruby and https://github.com/ruby-concurrency sounds good to me, concurrent-ruby/concurrent-ruby may be easier to remember though
• 👍 keeping one gem for now with keeping in mind possible extraction of Actors if it grows too big
• 👍 for being conservative about adding a dependency to this gem, but not totally opposed to it.
• 👍 "our gem should be for everyone and Java or C extensions are only a way to optimize the code"
• ruby-atomic has native Ruby implementation, afaik it does not need any changes before merging
• "lock/unlock with ensure clause" is in master

[jdantonio]

@mighe @chrisseaton @lucasallan @pitr-ch @headius @schmurfy Did one of you create this organization: https://github.com/concurrent-ruby? It's the organization I was planning to create for this gem by it already existed.

[pitr-ch]

hm it was not there just few days back :/

[mighe]

I did not create it... is there any way to know who is the owner? I could not find anything in github help.

[jdantonio]

I've submit a help request to GitHub.

[chrisseaton]

That wasn't me, I'm afraid.

[jdantonio]

The GitHub team has agreed to pass a polite message from me to the owner of that organization. Hopefully my request will be received in the spirit that was intended and we'll receive a positive response.

[jdantonio]

I've created a to-do list of the 0.7.0 release at issue #98 0.7.0 Release To-do. I'm closing this thread so we can move the discussion over there.

[lucasallan]

That wasn't me either.

[jdantonio]

@headius It's been two days and I haven't heard back from the owner of the concurrent-ruby organization. We may need a Plan B. Are you still willing to share the ruby-concurrency organization?

@mighe @chrisseaton @lucasallan @pitr-ch What do you all think? Assuming we can't get the concurrent-ruby organization (which looks to be the case) should we use the ruby-concurrency organization that @headius suggested, or does anyone have a suggestion for a different organization name? I'm leaning toward ruby-concurrency, but I'm open to other ideas.

[pitr-ch]

I am fine with waiting litter longer a weak or two to see if we can't really get concurrent-ruby. ruby-concurrency sounds as great alternative to me.

[mighe]

I completely agree with @pitr-ch 👍

[jdantonio]

OK. I'll follow-up with the support person at GitHub early next week.

[schmurfy]

it wasn't me either (just got back from vacation).
Both names are fine for me :)