Ruby and JRuby JSON Time with json, json_pure, and Merb + DataMapper extlib

After trying to run a super-simple Ruby script in JRuby that ran fine through the MRI, I found myself trying to debug a JSON generator exception.

The Ruby script required a RubyGem that had its own dependencies on the json gem and Merb + DataMapper extlib gem. I thought that I could simply rebuild the gem to use json_pure and run things on JRuby. Little did I know…

First, here are the versions of everything involved:

• Ruby 1.8.6
• JRuby 1.1.4
• rubygems 1.2.0
• json 1.1.3
• json_pure 1.1.3
• extlib 0.9.6

Let’s take a simple example of converting Time to JSON. My environment uses UTC as the timezone. Bear with me. The code samples are repetitive, but I had to go through them to figure things out.

require 'rubygems'
require 'json'

p [ JSON.parser, JSON.generator ]
p Time.now.to_json

We’ll run it using the MRI and JRuby.

MRI

[JSON::Ext::Parser, JSON::Ext::Generator]
"\"Fri Sep 12 20:20:06 +0000 2008\""

JRuby

[JSON::Pure::Parser, JSON::Pure::Generator]
"\"Fri Sep 12 20:20:55 +0000 2008\""

Seems like a reasonable conversion and in the same format. JRuby is using json_pure as expected. Good, it’s consistent. Note that the output is not in ISO format.

Now, let’s specify the the json_pure gem.

require 'rubygems'
require 'json/pure'

p [ JSON.parser, JSON.generator ]
p Time.now.to_json

Run it through the MRI and JRuby.

MRI

[JSON::Pure::Parser, JSON::Pure::Generator]
"\"Fri Sep 12 20:29:13 +0000 2008\""

JRuby

[JSON::Pure::Parser, JSON::Pure::Generator]
"\"Fri Sep 12 20:29:45 +0000 2008\""

Again, everything is consistent. Good news again.

Since things are going well, let’s try using extlib in the script so that the Time is in ISO format.

require 'rubygems'
require 'extlib'
require 'json'

p [ JSON.parser, JSON.generator ]
p Time.now.to_json

Run it through the MRI and JRuby.

MRI

[JSON::Ext::Parser, JSON::Ext::Generator]
"\"2008-09-12T20:44:06+00:00\""

JRuby

[JSON::Pure::Parser, JSON::Pure::Generator]
"\"2008-09-12T20:44:31+00:00\""

Excellent! Everything is working wonderfully. No problems at all!

At this point, let me do what I wanted to do in the first place and modify the script to be a basic variation of the problematic Ruby script that gave me a headache.

I’m going to place the time in a Hash, but I won’t include extlib yet.

require 'rubygems'
require 'json'

p [ JSON.parser, JSON.generator ]
p Time.now.to_json

h = {"created_on" => Time.now}
p h.to_json

Run it through the MRI and JRuby.

MRI

[JSON::Ext::Parser, JSON::Ext::Generator]
"\"Fri Sep 12 20:48:50 +0000 2008\""
"{\"created_on\":\"Fri Sep 12 20:48:50 +0000 2008\"}"

JRuby

[JSON::Pure::Parser, JSON::Pure::Generator]
"\"Fri Sep 12 20:49:26 +0000 2008\""
"{\"created_on\":\"Fri Sep 12 20:49:26 +0000 2008\"}"

Fantastic! It’s a useless script, but it’s going places: The Time is in a Hash that is being converted to JSON.

Ok, let’s add extlib so that we can get the JSON Time ISO format…and because our problematic Ruby script uses a particular gem (a very useful gem for the Ruby script) that happens to depend on extlib.

require 'rubygems'
require 'extlib'
require 'json'

p [ JSON.parser, JSON.generator ]
p Time.now.to_json

h = {"created_on" => Time.now}
p h.to_json

Run it through the MRI and JRuby.

MRI

[JSON::Ext::Parser, JSON::Ext::Generator]
"\"2008-09-12T20:51:43+00:00\""
"{\"created_on\":\"2008-09-12T20:51:43+00:00\"}"

JRuby

[JSON::Pure::Parser, JSON::Pure::Generator]
"\"2008-09-12T20:52:02+00:00\""
/home/share/storage/jruby-1.1.4/lib/ruby/gems/1.8/gems/json_pure-1.1.3/lib/json/pure/generator.rb:251:in `to_json': wrong # of arguments(2 for 0) (ArgumentError)
	from /home/share/storage/jruby-1.1.4/lib/ruby/gems/1.8/gems/json_pure-1.1.3/lib/json/pure/generator.rb:251:in `json_transform'
	from /home/share/storage/jruby-1.1.4/lib/ruby/gems/1.8/gems/json_pure-1.1.3/lib/json/pure/generator.rb:245:in `each'
	from /home/share/storage/jruby-1.1.4/lib/ruby/gems/1.8/gems/json_pure-1.1.3/lib/json/pure/generator.rb:245:in `map'
	from /home/share/storage/jruby-1.1.4/lib/ruby/gems/1.8/gems/json_pure-1.1.3/lib/json/pure/generator.rb:245:in `json_transform'
	from /home/share/storage/jruby-1.1.4/lib/ruby/gems/1.8/gems/json_pure-1.1.3/lib/json/pure/generator.rb:218:in `to_json'
	from time_extlib_to_json.rb:9

EH?! Now why in the world would it give me an error? Everything was fine up until this point. Why was it giving me a “wrong # of arguments(2 for 0) (ArgumentError)” exception?

Well, I had a great time with JRuby debugging sessions looking at the json_transform method in json_pure’s generator.rb.

Just like the exception said, I came to see that the Time.to_json method no longer accepted two arguments…even though the json_transform method was trying to pass them in through the s << value.to_json(state, depth + 1) line. When the exception was thrown, the Time instance, aka value, had a to_json method that took 0 arguments.

          def json_transform(state, depth)
            delim = ','
            delim << state.object_nl if state
            result = '{'
            result << state.object_nl if state
            result << map { |key,value|
              s = json_shift(state, depth + 1)
              s << key.to_s.to_json(state, depth + 1)
              s << state.space_before if state
              s << ':'
              s << state.space if state
              s << value.to_json(state, depth + 1)
            }.join(delim)
            result << state.object_nl if state
            result << json_shift(state, depth)
            result << '}'
            result
          end

But why was this only showing up after I put a Time instance into a Hash? Well, the json_transform method call in generator.rb is in the module Hash, and it’s responsible for calling to_json with two arguments, state and depth + 1, to convert a Hash key’s value.

module JSON
  module Pure
    module Generator
      module GeneratorMethods
        module Hash
          def json_transform(state, depth)

Nice. So now what? Well, I tried requiring ‘json/add/core’ and running it under JRuby…

require 'rubygems'
require 'extlib'
require 'json'
require 'json/add/core'

p [ JSON.parser, JSON.generator ]
p Time.now.to_json

h = {"created_on" => Time.now}
p h.to_json

It works because ‘json/add/core’ gives Time.to_json a variable number of arguments again. But it gave this JSON output:

JRuby

[JSON::Pure::Parser, JSON::Pure::Generator]
"{\"json_class\":\"Time\",\"s\":1221254881,\"n\":82427000}"
"{\"created_on\":{\"json_class\":\"Time\",\"s\":1221254881,\"n\":83467000}}"

I’m not into that Time format at all!

If I require ‘extlib’ after ‘json’ and ‘json/add/core’, I get the same error of course.

Maybe I’ll just drop using the gem that requires extlib and override Time.to_json myself…That doesn’t seem right since that gem has other functionality that I need.

Right now, I feel like this is both an extlib and json_pure issue. extlib’s Time.to_json doesn’t accept a variable number of arguments, and the ‘json/add/core’ time format isn’t what I was looking for.

Hmmm…

Update

Actually, I take back what I said about it being either an extlib or json_pure issue. This might just be a case of code coming together and clashing.

Update 2

I guess eating helps clear the head! It looks like it’s a json_pure issue since it happens on both JRuby and the MRI when the script requires ‘json/pure’. Here are the parts of the code in the C extension where it differs from Ruby:

static VALUE mHash_to_json(int argc, VALUE *argv, VALUE self)
{
    VALUE Vstate, Vdepth, result;
    long depth;

    rb_scan_args(argc, argv, "02", &Vstate, &Vdepth);
    depth = NIL_P(Vdepth) ? 0 : FIX2LONG(Vdepth);
    if (NIL_P(Vstate)) {
        long len = RHASH(self)->tbl->num_entries;
        result = rb_str_buf_new(len);
        rb_str_buf_cat2(result, "{");
        rb_hash_foreach(self, hash_to_json_i, result);
        rb_str_buf_cat2(result, "}");
    } else {
        GET_STATE(Vstate);
        check_max_nesting(state, depth);
        if (state->check_circular) {
            VALUE self_id = rb_obj_id(self);
            if (RTEST(rb_hash_aref(state->seen, self_id))) {
                rb_raise(eCircularDatastructure,
                        "circular data structures not supported!");
            }
            rb_hash_aset(state->seen, self_id, Qtrue);
            result = mHash_json_transfrom(self, Vstate, LONG2FIX(depth));
            rb_hash_delete(state->seen, self_id);
        } else {
            result = mHash_json_transfrom(self, Vstate, LONG2FIX(depth));
        }
    }
    OBJ_INFECT(result, self);
    return result;
}

static int hash_to_json_i(VALUE key, VALUE value, VALUE buf)
{
    VALUE tmp;

    if (key == Qundef) return ST_CONTINUE;
    if (RSTRING_LEN(buf) > 1) rb_str_buf_cat2(buf, ",");
    tmp = rb_funcall(rb_funcall(key, i_to_s, 0), i_to_json, 0);
    Check_Type(tmp, T_STRING);
    rb_str_buf_append(buf, tmp);
    OBJ_INFECT(buf, tmp);
    rb_str_buf_cat2(buf, ":");
    tmp = rb_funcall(value, i_to_json, 0);
    Check_Type(tmp, T_STRING);
    rb_str_buf_append(buf, tmp);
    OBJ_INFECT(buf, tmp);

    return ST_CONTINUE;
}

Update 3

Follow-up post:
json_pure 1.1.3 patch

Feedback is welcome!