Posterous theme by Cory Watilo

Filed under: datamapper

Ruby and JRuby JSON Time with json, json_pure, and Merb + DataMapper extlib

After trying to run a super-simple Ruby script in JRuby that ran fine through the MRI, I found myself trying to debug a JSON generator exception. The Ruby script required a RubyGem that had its own dependencies on the json gem and Merb + DataMapper extlib gem. I thought that I could simply rebuild the gem to use json_pure and run things on JRuby. Little did I know... First, here are the versions of everything involved:
  • Ruby 1.8.6
  • JRuby 1.1.4
  • rubygems 1.2.0
  • json 1.1.3
  • json_pure 1.1.3
  • extlib 0.9.6
Let's take a simple example of converting Time to JSON. My environment uses UTC as the timezone. Bear with me. The code samples are repetitive, but I had to go through them to figure things out. [sourcecode="ruby"] require 'rubygems' require 'json' p [ JSON.parser, JSON.generator ] p Time.now.to_json [/sourcecode] We'll run it using the MRI and JRuby. MRI [sourcecode="ruby"] [JSON::Ext::Parser, JSON::Ext::Generator] "\"Fri Sep 12 20:20:06 +0000 2008\"" [/sourcecode] JRuby [sourcecode="ruby"] [JSON::Pure::Parser, JSON::Pure::Generator] "\"Fri Sep 12 20:20:55 +0000 2008\"" [/sourcecode] Seems like a reasonable conversion and in the same format. JRuby is using json_pure as expected. Good, it's consistent. Note that the output is not in ISO format. Now, let's specify the the json_pure gem. [sourcecode="ruby"] require 'rubygems' require 'json/pure' p [ JSON.parser, JSON.generator ] p Time.now.to_json [/sourcecode] Run it through the MRI and JRuby. MRI [sourcecode="ruby"] [JSON::Pure::Parser, JSON::Pure::Generator] "\"Fri Sep 12 20:29:13 +0000 2008\"" [/sourcecode] JRuby [sourcecode="ruby"] [JSON::Pure::Parser, JSON::Pure::Generator] "\"Fri Sep 12 20:29:45 +0000 2008\"" [/sourcecode] Again, everything is consistent. Good news again. Since things are going well, let's try using extlib in the script so that the Time is in ISO format. [sourcecode="ruby"] require 'rubygems' require 'extlib' require 'json' p [ JSON.parser, JSON.generator ] p Time.now.to_json [/sourcecode] Run it through the MRI and JRuby. MRI [sourcecode="ruby"] [JSON::Ext::Parser, JSON::Ext::Generator] "\"2008-09-12T20:44:06+00:00\"" [/sourcecode] JRuby [sourcecode="ruby"] [JSON::Pure::Parser, JSON::Pure::Generator] "\"2008-09-12T20:44:31+00:00\"" [/sourcecode] Excellent! Everything is working wonderfully. No problems at all! At this point, let me do what I wanted to do in the first place and modify the script to be a basic variation of the problematic Ruby script that gave me a headache. I'm going to place the time in a Hash, but I won't include extlib yet. [sourcecode="ruby"] require 'rubygems' require 'json' p [ JSON.parser, JSON.generator ] p Time.now.to_json h = {"created_on" => Time.now} p h.to_json [/sourcecode] Run it through the MRI and JRuby. MRI [sourcecode="ruby"] [JSON::Ext::Parser, JSON::Ext::Generator] "\"Fri Sep 12 20:48:50 +0000 2008\"" "{\"created_on\":\"Fri Sep 12 20:48:50 +0000 2008\"}" [/sourcecode] JRuby [sourcecode="ruby"] [JSON::Pure::Parser, JSON::Pure::Generator] "\"Fri Sep 12 20:49:26 +0000 2008\"" "{\"created_on\":\"Fri Sep 12 20:49:26 +0000 2008\"}" [/sourcecode] Fantastic! It's a useless script, but it's going places: The Time is in a Hash that is being converted to JSON. Ok, let's add extlib so that we can get the JSON Time ISO format...and because our problematic Ruby script uses a particular gem (a very useful gem for the Ruby script) that happens to depend on extlib. [sourcecode="ruby"] require 'rubygems' require 'extlib' require 'json' p [ JSON.parser, JSON.generator ] p Time.now.to_json h = {"created_on" => Time.now} p h.to_json [/sourcecode] Run it through the MRI and JRuby. MRI [sourcecode="ruby"] [JSON::Ext::Parser, JSON::Ext::Generator] "\"2008-09-12T20:51:43+00:00\"" "{\"created_on\":\"2008-09-12T20:51:43+00:00\"}" [/sourcecode] JRuby [sourcecode="ruby"] [JSON::Pure::Parser, JSON::Pure::Generator] "\"2008-09-12T20:52:02+00:00\"" /home/share/storage/jruby-1.1.4/lib/ruby/gems/1.8/gems/json_pure-1.1.3/lib/json/pure/generator.rb:251:in `to_json': wrong # of arguments(2 for 0) (ArgumentError) from /home/share/storage/jruby-1.1.4/lib/ruby/gems/1.8/gems/json_pure-1.1.3/lib/json/pure/generator.rb:251:in `json_transform' from /home/share/storage/jruby-1.1.4/lib/ruby/gems/1.8/gems/json_pure-1.1.3/lib/json/pure/generator.rb:245:in `each' from /home/share/storage/jruby-1.1.4/lib/ruby/gems/1.8/gems/json_pure-1.1.3/lib/json/pure/generator.rb:245:in `map' from /home/share/storage/jruby-1.1.4/lib/ruby/gems/1.8/gems/json_pure-1.1.3/lib/json/pure/generator.rb:245:in `json_transform' from /home/share/storage/jruby-1.1.4/lib/ruby/gems/1.8/gems/json_pure-1.1.3/lib/json/pure/generator.rb:218:in `to_json' from time_extlib_to_json.rb:9 [/sourcecode] EH?! Now why in the world would it give me an error? Everything was fine up until this point. Why was it giving me a "wrong # of arguments(2 for 0) (ArgumentError)" exception? Well, I had a great time with JRuby debugging sessions looking at the json_transform method in json_pure's generator.rb. Just like the exception said, I came to see that the Time.to_json method no longer accepted two arguments...even though the json_transform method was trying to pass them in through the s << value.to_json(state, depth + 1) line. When the exception was thrown, the Time instance, aka value, had a to_json method that took 0 arguments. [sourcecode="ruby"] def json_transform(state, depth) delim = ',' delim json_transform method call in generator.rb is in the module Hash, and it's responsible for calling to_json with two arguments, state and depth + 1, to convert a Hash key's value. [sourcecode="ruby"] module JSON module Pure module Generator module GeneratorMethods module Hash def json_transform(state, depth) [/sourcecode] Nice. So now what? Well, I tried requiring 'json/add/core' and running it under JRuby... [sourcecode="ruby"] require 'rubygems' require 'extlib' require 'json' require 'json/add/core' p [ JSON.parser, JSON.generator ] p Time.now.to_json h = {"created_on" => Time.now} p h.to_json [/sourcecode] It works because 'json/add/core' gives Time.to_json a variable number of arguments again. But it gave this JSON output: JRuby [sourcecode="ruby"] [JSON::Pure::Parser, JSON::Pure::Generator] "{\"json_class\":\"Time\",\"s\":1221254881,\"n\":82427000}" "{\"created_on\":{\"json_class\":\"Time\",\"s\":1221254881,\"n\":83467000}}" [/sourcecode] I'm not into that Time format at all! If I require 'extlib' after 'json' and 'json/add/core', I get the same error of course. Maybe I'll just drop using the gem that requires extlib and override Time.to_json myself...That doesn't seem right since that gem has other functionality that I need. Right now, I feel like this is both an extlib and json_pure issue. extlib's Time.to_json doesn't accept a variable number of arguments, and the 'json/add/core' time format isn't what I was looking for. Hmmm...

Update

Actually, I take back what I said about it being either an extlib or json_pure issue. This might just be a case of code coming together and clashing. :-(

Update 2

I guess eating helps clear the head! It looks like it's a json_pure issue since it happens on both JRuby and the MRI when the script requires 'json/pure'. Here are the parts of the code in the C extension where it differs from Ruby:
static VALUE mHash_to_json(int argc, VALUE *argv, VALUE self)
{
    VALUE Vstate, Vdepth, result;
    long depth;

    rb_scan_args(argc, argv, "02", &Vstate, &Vdepth);
    depth = NIL_P(Vdepth) ? 0 : FIX2LONG(Vdepth);
    if (NIL_P(Vstate)) {
        long len = RHASH(self)->tbl->num_entries;
        result = rb_str_buf_new(len);
        rb_str_buf_cat2(result, "{");
        rb_hash_foreach(self, hash_to_json_i, result);
        rb_str_buf_cat2(result, "}");
    } else {
        GET_STATE(Vstate);
        check_max_nesting(state, depth);
        if (state->check_circular) {
            VALUE self_id = rb_obj_id(self);
            if (RTEST(rb_hash_aref(state->seen, self_id))) {
                rb_raise(eCircularDatastructure,
                        "circular data structures not supported!");
            }
            rb_hash_aset(state->seen, self_id, Qtrue);
            result = mHash_json_transfrom(self, Vstate, LONG2FIX(depth));
            rb_hash_delete(state->seen, self_id);
        } else {
            result = mHash_json_transfrom(self, Vstate, LONG2FIX(depth));
        }
    }
    OBJ_INFECT(result, self);
    return result;
}

static int hash_to_json_i(VALUE key, VALUE value, VALUE buf)
{
    VALUE tmp;

    if (key == Qundef) return ST_CONTINUE;
    if (RSTRING_LEN(buf) > 1) rb_str_buf_cat2(buf, ",");
    tmp = rb_funcall(rb_funcall(key, i_to_s, 0), i_to_json, 0);
    Check_Type(tmp, T_STRING);
    rb_str_buf_append(buf, tmp);
    OBJ_INFECT(buf, tmp);
    rb_str_buf_cat2(buf, ":");
    tmp = rb_funcall(value, i_to_json, 0);
    Check_Type(tmp, T_STRING);
    rb_str_buf_append(buf, tmp);
    OBJ_INFECT(buf, tmp);

    return ST_CONTINUE;
}

Update 3

Follow-up post: json_pure 1.1.3 patch Feedback is welcome!

merb 0.9.5 + haml + datamapper + sqlite3

Since Merb is constantly changing prior to its 1.0 release, a lot of the tutorials did not work exactly the way I expected them to work. I was looking for a tutorial that covered a simple Merb + HAML + DataMapper + SQLite application. I found that a good candidate to tweak was the simple chat wall tutorial that was prominently linked on the Merb wiki. The following are some code updates to get it working with Merb 0.9.5, HAML, DataMapper, and SQLite. Feedback is welcome.

merb-gen app

merb-gen app allows you to specify the template engine, ORM, and testing framework directly on the command line: merb-gen app slapp --template-engine haml --orm datamapper --testing-framework rspec

init.rb

DataMapper dependencies should be specified in init.rb to get counts/aggregates, automatic created_at timestamps, and validations. Datamapper can also handle session storage. And don't forget to set the :session_id_key! init.rb contents [sourcecode="ruby"] dependency "dm-aggregates" dependency "dm-timestamps" dependency "dm-validations" dependency "merb_helpers" ... use_orm :datamapper ... use_template_engine :haml ... c[:session_id_key] = 'chat_wall_session_id' ... c[:session_store] = 'datamapper' ... [/sourcecode]

SQLite database configuration and setup

DataMapper offers a rake task to create a sample database.yml file: rake dm:db:database_yaml Here's what a database.yml file for development and test SQLite databases looks like: database.yml contents [sourcecode="ruby"] --- :development: &defaults :adapter: sqlite3 :database: db/dev.db :test: sqlite db/dev.db sqlite db/test.db Database sessions migration: rake dm:sessions:create Database migration: rake dm:db:automigrate

Post model

Model validations didn't seem to be working properly... post.rb contents [sourcecode="ruby"] class Post include DataMapper::Resource property :id, Serial property :body, String property :created_at, DateTime validates_length :body, :minimum => 2 end [/sourcecode]

HAML

I had problems with the Merb form helper in the index HAML file... app/views/posts/index.html.haml contents [sourcecode="ruby"] !!! Loose %html{html_attrs{'en-us'}} %body %h1 Welcome to Slapp %h2 A simple chat wall %p Recent Posts: .container{:id => 'posts'}= partial("shared/post", :with => @posts) %div Post Something: - #form_for(:post, :action => url(:posts) ) do %form{ :action => '/posts/create', :method => :post} = text_field(:name => "body", :size => 40) = submit "Post Message!" [/sourcecode]The individual post HAML file... app/views/shared/_post.html.haml contents [sourcecode="ruby"] .post{:id => 'post-' + post.id.to_s} .body= h(post.body) .created= relative_date(post.created_at) [/sourcecode]

RSpec

Lastly, I had to change some of the CSS selectors and the deletion method for the index spec... spec/views/posts/index_spec.rb contents [sourcecode="ruby"] ... it "should have a containing div for the posts" do @body.should have_selector("div#posts.container") end it "should have a div for each individual post" do @posts.each do |post| @body.should have_selector("div#posts.container div#post-#{ post.id }.post") end end it "should have the contents of each post inside a div with an id and class" do @posts.each do |post| @body.should have_tag("div#posts") do with_tag(:div, :id => "post-#{ post.id }", :class => "post", :content => post.body) end end end it "should have a form to create new posts with a single input and submit button" do @body.should have_selector("form[@action='/posts/create']") @body.should have_selector("form[@action='/posts/create'] input[@name='body']") @body.should have_selector("form[@action='/posts/create'] input[@type='submit']") end after(:each) do Post.all.destroy! end ... [/sourcecode]