intridea/multi_json

1.7.1: Breakage with ActiveRecord serialization

Closed this issue · 14 comments

purcell commented

I have an AR model with a serialized attribute:

class TwitterAccount < ActiveRecord::Base
  serialize :user_data, JSON
  ...
end

This works fine under Rails 3.2.12 using multi_json 1.6.1, but after upgrading to 1.7.1 I start seeing SystemStackErrors upon saving record instances, apparently from deep within AR's transaction error handling machinery. I initially encountered this error upon updating my gems for Rails 3.2.13, but have subsequently narrowed the issue down to multi_json.

Unfortunately the SystemStackErrors do not display a stacktrace, so I haven't succeeded in getting more detailed information about what is going wrong.

The user_data is passed the attrs of an object retrieved via the Twitter gem, and things work okay dumping that directly:

irb> JSON.dump(Twitter.user('pink').attrs)
=> "{\"id\":28706024,\"id_str\":\"28706024\",\"name\":\"P!nk\",\"screen_name\":\"Pink\",\"location\":\"los angeles\",\"description\":\"it's all happening\",\"url\":\"http://twitter.com/pink\",\"entities\":{\"url\":{\"urls\":[{\"url\":\"http://twitter.com/pink\",\"expanded_url\":null,\"indices\":[0,23]}]},\"description\":{\"urls\":[]}},\"protected\":false,\"followers_count\":14080965,\"friends_count\":230,\"listed_count\":61769,\"created_at\":\"Sat Apr 04 01:16:34 +0000 2009\",\"favourites_count\":64,\"utc_offset\":-28800,\"time_zone\":\"Pacific Time (US & Canada)\",\"geo_enabled\":false,\"verified\":true,\"statuses_count\":4243,\"lang\":\"en\",\"status\":{\"created_at\":\"Tue Mar 19 02:34:32 +0000 2013\",\"id\":313840884678545408,\"id_str\":\"313840884678545408\",\"text\":\"“@ladybusiness: @Pink thought you might enjoy this... http://t.co/sow4lxgekZ” hahahaaha amazing. I catch like that sometimes\",\"source\":\"<a href=\\\"http://twitter.com/download/iphone\\\" rel=\\\"nofollow\\\">Twitter for iPhone</a>\",\"truncated\":false,\"in_reply_to_status_id\":313832518820438017,\"in_reply_to_status_id_str\":\"313832518820438017\",\"in_reply_to_user_id\":21823648,\"in_reply_to_user_id_str\":\"21823648\",\"in_reply_to_screen_name\":\"ladybusiness\",\"geo\":null,\"coordinates\":null,\"place\":null,\"contributors\":null,\"retweet_count\":207,\"entities\":{\"hashtags\":[],\"urls\":[{\"url\":\"http://t.co/sow4lxgekZ\",\"expanded_url\":\"http://youtu.be/QYZfcFzcwxo\",\"display_url\":\"youtu.be/QYZfcFzcwxo\",\"indices\":[54,76]}],\"user_mentions\":[{\"screen_name\":\"ladybusiness\",\"name\":\"STVN MNZNO\",\"id\":21823648,\"id_str\":\"21823648\",\"indices\":[1,14]},{\"screen_name\":\"Pink\",\"name\":\"P!nk\",\"id\":28706024,\"id_str\":\"28706024\",\"indices\":[16,21]}]},\"favorited\":false,\"retweeted\":false,\"possibly_sensitive\":false,\"lang\":\"en\"},\"contributors_enabled\":false,\"is_translator\":false,\"profile_background_color\":\"DBE9ED\",\"profile_background_image_url\":\"http://a0.twimg.com/profile_background_images/178806023/grammys13.jpg\",\"profile_background_image_url_https\":\"https://si0.twimg.com/profile_background_images/178806023/grammys13.jpg\",\"profile_background_tile\":false,\"profile_image_url\":\"http://a0.twimg.com/profile_images/549375583/Pinkdeborahanderson_112_normal.jpg\",\"profile_image_url_https\":\"https://si0.twimg.com/profile_images/549375583/Pinkdeborahanderson_112_normal.jpg\",\"profile_link_color\":\"CC3366\",\"profile_sidebar_border_color\":\"DBE9ED\",\"profile_sidebar_fill_color\":\"E6F6F9\",\"profile_text_color\":\"333333\",\"profile_use_background_image\":true,\"default_profile\":false,\"default_profile_image\":false,\"following\":true,\"follow_request_sent\":false,\"notifications\":false}"

So something subtle must be going on... any ideas?

-Steve

rwz commented

That's strange because rails use ActiveSupport::JSON to serialize/deserialize data for models.

I couldn't reproduce your error using rails 3.2.13, MultiJson 1.7.1 and attributes from @pink twitter user.

Can you provide more info? The data sample causes the error, the MultiJson adapter you use and the versions of related adapter gems.

purcell commented

Thanks for taking a look. I've been continuing to investigate, so give me a while to rule out clashes with other gems, and hopefully provide a minimal reproducible case.

(It might be worth noting that the database is postgresql, and the column type for user_data is text -- which DB did you test with?)

-Steve

rwz commented

I've tested with sqlite3 since it was minimal and easiest test-case I could build.

purcell commented

Hmm, I can reproduce this trivially, even without touching the database. Starting with ruby 1.9.3 and Rails 3.2.13:

rails new foobar
cd foobar
echo 'gem "twitter"' >> Gemfile
bundler install

Then create a config/initializers/twitter.rb containing something like this:

Twitter.configure do |config|
  config.consumer_key = 'REPLACEME'
  config.consumer_secret = 'REPLACEME'
  config.oauth_token = "REPLACEME"
  config.oauth_token_secret = "REPLACEME"
end

Finally:

% rails console
irb(main):010:0> JSON.dump(Twitter.user('pink').attrs)
SystemStackError: stack level too deep
    from /Users/steve/.rbenv/versions/1.9.3-p327/lib/ruby/1.9.1/irb/workspace.rb:80
Maybe IRB bug!
rwz commented

Why are you doing JSON.dump? If it's MultiJson bug, you should do MultiJson.dump instead.

purcell commented

Ah, am I wrong in thinking that JSON is wrapping MultiJson in this case?

rwz commented

No. In that case JSON is not wrapping anything. In case of the model, JSON is actually ActiveSupport::JSON, which partly wraps MultiJson

rwz commented

Yes, it's complicated :)

purcell commented

Same issue, in fact, even with multi_json 1.6.1:

irb(main):009:0> MultiJson.dump(Twitter.user('pink').attrs)
SystemStackError: stack level too deep
    from /Users/steve/.rbenv/versions/1.9.3-p327/lib/ruby/1.9.1/irb/workspace.rb:80
Maybe IRB bug!

I'm thinking that Twitter might have just changed the structure of their responses, and so the Twitter gem objects have become unserializable...

rwz commented

Twitter.user('pink').attrs should be a Hash, so it is serializable.

Ok, now what if you add json gem into your Gemfile and run bundle install && bundle update json

Also, try ActiveSupport::JSON.encode(Twitter.user('pink').attrs)

purcell commented

Well I just tried a plain irb session, and loaded the 'multi_json' and 'twitter' gems manually: the result is that it works!

Can we then conclude that some Rails magic is at fault?

(BTW, the .attrs expression doesn't yield a plain Hash: one of its attributes is a #<Twitter::Tweet:0x007f86e3827e60 ...)

purcell commented

Looks like there's a corresponding twitter gem issue: sferik/twitter-ruby#368

I'll close this because there's now no clear indication that multi_json is at fault.

Thanks so much for your time and trouble helping me with this.

-Steve

rwz commented

Oh, I've used Twitter gem version 4.5.0 in my test and it actually outputs a simple Hash

purcell commented

Yeah, when I updated the gems for Rails 3.2.13, the new Twitter gem got pulled in too, then I failed to notice that in my testing. Feeling pretty stupid for not eliminating that sooner. Thanks again. :-)