Blog
what did i learn today

Let me quickly introduce WiceGrid, if you do not know yet: it is a super-gem that will allow you to easily show a list/grid of items, and allow easy filtering/searching/pagination.

For rails there is, afaik, no better alternative. There are some javascript/jquery driven dynamic grids, but for me the big advantage is that with WiceGrid all work is done server-side, which is ideal when handling large sets of data.

Since you can just render html in any column, we do for instance the following for our KLIP platform :

Screen Shot 2016-06-05 at 21.08.43

In our first we show our internal identifier, and the external identifier. In code this looks like this:

g.column name: 'Id', in_csv: false do |pr|
  render 'title_with_info', plan_request: pr
end

and the partial title_with_info looks like

%h4
  = plan_request.ident
%p.text-muted.small
  = plan_request.maprequest_id

But now the problem is: how can we, when filtering, automatically look for both fields? WiceGrid automatically handles one field, but not both. Luckily, WiceGrid allows us to define custom filter types. What we want is:

  • we want the filter to just look like a standard string field
  • we want to build a query which will search for ident or maprequest_id.

Adding your own custom filter types is not entirely clear in the documentation, I had to take a look at the code to fully understand it. So that's why I decided to write it in detail here.

It takes three steps:

  • define a class to create the correct filter (a conditions generator)
  • define a custom filter_type inside WiceGrid, using your custom class
  • use the class in the column definition

Create Conditions Generator

Inside lib/wice/columns add a new file called conditions_generator_column_plan_request_identifier.rb and add the following content:

module Wice
  module Columns
    class ConditionsGeneratorColumnPlanRequestIdentifier < ConditionsGeneratorColumn  #:nodoc:

      def generate_conditions(table_alias, opts)   #:nodoc:
        if opts.kind_of? String
          string_fragment = opts
          negation = ''
        elsif (opts.kind_of? Hash) && opts.has_key?(:v)
          string_fragment = opts[:v]
          negation = opts[:n] == '1' ? 'NOT' : ''
        else
          Wice.log "invalid parameters for the grid string filter - must be a string: #{opts.inspect} or a Hash with keys :v and :n"
          return false
        end
        if string_fragment.empty?
          return false
        end

        table_name = @column_wrapper.alias_or_table_name(table_alias)
        op = ::Wice.get_string_matching_operators(@column_wrapper.model)
        search_value = "%#{string_fragment}%"

        [
            " #{negation}  (#{table_name}.ident #{op} ? OR #{table_name}.external_id #{op} ?)",
            search_value, search_value
        ]
      end
    end
  end
end

This class is actually almost copied from the standard column generator, except I generate a different condition at the end, where I compare with two fields with an OR operator. This way I will find results if either the ident or the external_id matches the search-value.

Define filter type in config

In config/wice_grid_config.rb add the following:

Wice::Defaults::ADDITIONAL_COLUMN_PROCESSORS = {
  plan_request_identifier_filter:  ['ViewColumnString', 'Wice::Columns::ConditionsGeneratorColumnPlanRequestIdentifier']
}

We just use the standard ViewColumnString to just show us a string filter.

Use the filter_type in the column

To enable the filter in the column, we just have to write the following:

g.column name: 'Id', attribute: 'ident', filter_type: :plan_request_identifier_filter, in_csv: false do |pr|
  render 'title_with_info', plan_request: pr
end
[oracle] avoiding SLOW sdo_aggr_union

There is this recurring problem we have in GIS: getting road-segments and wanting to show complete roads. The naive approach would we to do something like the following:

insert into street_geoms
select ro.rd_ro_ident, ro.rd_ro_name, ro.com_code, ssdo_aggr_union(sdoaggrtype(rd.ro_geometry, 0.005)) as geom
from rd_road ro, rd_ro_sec ros
where ros.rd_ro_ident = ro.rd_ro_ident
group by ro.rd_ro_ident, ro.rd_ro_name, ro.com_code;

For good measure: we have 45.000+ roads, with a total of 230.000+ road segments. So when that query starts running and starts taking a long time, I started googling. Apparently there are two faster alternatives: SDO_AGGR_CONCAT_LINES and SDO_AGGR_SET_UNION. While the first was really quick, completed in minutes, the result was completely wrong (complete segments were missing). The second might be quicker, but it was really hard to get an idea about any progress, and if it would fail, everything should be lost (rolled back).

So I decided to write a little script, and issue a sql statement for each single road, allowing me to track progress and added restartibility. For each road I issued a statement like:

insert into street_geoms
select ro.rd_ro_ident, ro.rd_ro_name, ro.com_code, sdo_aggr_set_union(CAST(COLLECT(ros.rd_ros_geometry) AS mdsys.SDO_Geometry_Array),0.005) as geom
from rd_road ro, rd_ro_sec ros
where ros.rd_ro_ident = ro.rd_ro_ident
  and ro.rd_ro_ident = 1895101 
group by ro.rd_ro_ident, ro.rd_ro_name, ro.com_code;

I added some ruby code around it, to make sure it tracked the progress and calculated the remaining time, just to have an idea. The first "large" road it stumbled upon literally took hours. It only had to join 39 segments. A simple query learned I had 150+ roads with more segments, and a maximum of 125 segments in the database. I could not just simply ignore them :) So this was not going to work either.

Why would this be so hard? I just wanted to throw all linestrings together into one geometry. How could I do that? Querying the geometries was really easy, so what if I joined the geometries outside of oracle? And wouldn't that be hard? But there is a simple solution: convert the strings to WKT, and join all LINESTRING in a MULTILINESTRING. This would just be simple string manipulation. I can do that ;)

I had some hiccups with this approach: handling the long strings proved a bit akward (use CLOB instead) and I had to regularly call GC.start to make sure the open cursors were released. And I had to make sure not to build a string literal which was too long (ORA-06550).

But in the end I was able to join the road-sections for the 45.000 + roads in approx 1.5h, which is not blindingly fast, but faster than 1 single SDO_AGGR_SET_UNION operation :) :)

For reference you can see the full code:

class StreetGeom < ActiveRecord::Base
  self.primary_key = 'rd_ro_ident'
end


def format_time (t)
  t = t.to_i
  sec = t % 60
  min  = (t / 60) % 60
  hour = t / 3600
  sprintf("% 3d:%02d:%02d", hour, min, sec)
end


def eta(count)
  if count == 0
    "ETA:  --:--:--"
  else
    elapsed = Time.now - @start_time
    # eta = elapsed * @total / count - elapsed;
    eta = (elapsed / count) * (@total - count)

    sprintf("ETA: %s", format_time(eta))
  end
end


all_roads = Road.count
geoms_to_calculate = all_roads - StreetGeom.count
@total = geoms_to_calculate

puts "Joining geometries for #{all_roads} roads [still #{geoms_to_calculate} to do]"


cntr = 1
@start_time = Time.now

done = 0


Road.order(:rd_ro_ident).each do |road|
  street_count = StreetGeom.where(rd_ro_ident: road.rd_ro_ident).count
  print "\rConverting #{cntr}/#{all_roads} [ #{eta(done)} ] "
  if street_count == 0
    print "..."
    $stdout.flush

    ## get all geometries in WKT format
    get_geoms_sql = <<-SQL
      select sdo_cs.make_2d(ros.rd_ros_geometry).get_wkt() as wkt_geom from rd_ro_sec ros where ros.rd_ro_ident = #{road.rd_ro_ident}
    SQL

    cursor = Road.connection.execute(get_geoms_sql)

    line_strings = []

    while row = cursor.fetch
      line_string = row[0].read.to_s
      line_strings << line_string[10..-1]
    end


    insert_sql = <<-SQL
      DECLARE
        wkt_str clob;
      BEGIN
        wkt_str := 'MULTILINESTRING(#{line_strings.join(", ';\nwkt_str := wkt_str || '")})';
        insert into street_geoms(rd_ro_ident, name, com_code, geom)
        values (#{road.rd_ro_ident}, q'[#{road.rd_ro_name}]', '#{road.com_code}',
             sdo_util.from_wktgeometry(to_clob(wkt_str)) );
      END;
    SQL

    Road.connection.execute(insert_sql)
    done += 1
  else
    print "_"
  end

  cntr += 1

  # periodically cleanup GC so we release open cursors ...
  # to avoid ORA-1000 errors
  if (cntr % 50) == 0
    GC.start
  end
end

print "\n"
puts "\n\nDone!"

and I run this script in the rails environment as follows: rails runner lib\tasks\join_road_geometries.rb.

Using render-anywhere gem with partials

Normally in rails, you can only render views inside of the controller. But what if you want to render a view somewhere else? For instance we wanted to generate xml-files using views. Haml can be used to describe xml just as well as plain html.

There is a gem called render_anywhere that allows just that. So how does this work, for example:

class Organisation < ActiveRecord::Base

  has_many :members

  include RenderAnywhere

  def to_xml
    render partial: "#{self.to_partial_path}", object: self, layout: 'my_xml_layout'
  end
end

We had a little problem when using partials though.

Normally if you type something like

= render @member

it will ask the partial path from the model (@member.to_partial_path), but somehow this always got prefixed with render_anywhere. The gem creates a dummy RenderingController in the RenderAnywhere namespace, so apparently it will look for the following view:

render_anywhere/members/member

In our case, I did not want to use the render_anywhere subfolder. It took me a while to figure out how to overrule this, but in essence it is pretty simple: rails uses the namespace of the rendering controller to prefix the path. Some deep googling proved that any controller has a method called _prefixes which lists all the prefixes for that class.

We can easily verify this in the rails console:

:001 > RenderAnywhere::RenderingController._prefixes
=> ["render_anywhere/rendering"]

So if we could overrule _prefixes to just return ["rendering"] ... Mmmmmm fork the code of render_anywhere? Or ...

There is another option: render_anywhere allows you to supply your own RenderingController and will use that instead if found in the context where the RenderAnywhere code is included.

So, if you write something like:

class Organisation < ActiveRecord::Base

  has_many :members

  include RenderAnywhere

  class RenderingController < RenderAnywhere::RenderingController

    def self._prefixes
      ["rendering"]
    end

  end

  def to_xml
    render partial: "#{self.to_partial_path}", object: self, layout: 'my_xml_layout'
  end
end

it will look for a view called members/member. Woot. To specify a different sub-folder you can adapt the _prefixes method as you wish :)

Developing rails websites with a geographic component we rely heavily on Postgis, so we use activerecord-postgis-adapter for the Postgis support, and I always use schema_plus because it allows me to define views. Until recently, I always had to use the structure.sql instead of the schema.rb because the geometric columns did not dump correctly.

But for a while now, activerecord-postgis-adapter handles this correctly and so we use the schema.rb file again. Only to discover a "new" error:

ActiveRecord::StatementInvalid: PG::DependentObjectsStillExist: ERROR:  cannot drop view geography_columns because extension postgis requires it
HINT:  You can drop extension postgis instead.
: DROP VIEW IF EXISTS "geography_columns"

Apparently specific Postgis views are also dumped in the schema file, and those views obviously cannot simply be re-created.

A very naive solution I kept using was to comment those create_view lines in our schema.rb file. But apparently there is a much better solution: you can configure which tables and views schema_plus should ignore.

So I added an initializer in /initializers/schema_dumper.rb with the following content:

ActiveRecord::SchemaDumper.ignore_tables = [
   "geography_columns", "geometry_columns", "spatial_ref_sys", "raster_columns", "raster_overviews"
]

And now my schema.rb is correct, and simple commands as rake db:setup or rake db:test:prepare just work. Hehe.

[rails] select distinct values of a column

The simplest way to select all distinct values of a column is, somewhat unintuitively:

Visit.uniq.pluck(:project)

this runs the query select distinct project from visits, and returns an array of strings. Exactly what you need, except ... I want it to be paginated. So we

Visit.uniq.pluck(:project).page(1)

... and that completely bombs: we now get an array of numbers?

So we try something else, and write:

Visit.select('distinct project') 

which runs the good query, but returns an array of Visit's with only the project filled in. I can live with that. And then pagination (using the kaminari gem) is again as expected:

Visit.select('distinct project').page(params[:page])

Nice :)

They default way in rails 4 to add foreign keys is simply

add_reference :people, :user

And this will add a column user_id to the people table, if the table exists. I have been looking in the rails code where this is handled, and it is really straightforward.

Note that standard rails does not really do anything for referential integrity, it creates the correct columns and an index if so specified.

But we use the schema_plus gem, for two reasons:

  • it does handle referential constraints correctly (on the database level)
  • it allows to specify the creation of views in a migration more cleanly

So, with schema_plus, if you write something like:

add_reference :people, :owner

and there is no owners table, this will give an error.

So instead you need to write :

add_reference :people, :owner, references: :users

This will correctly generate an owner_id which is a foreign key to users(id).

If you want to create a self-referential link, that is really easy. Just write

add_reference :people, :parent

This will create a self-referential link. Awesome :)

For completeness, the add_reference will add a reference columns to an already existing table. The same can be done when creating (or changing) a table with the following syntax:

create_table :people do |t| 
  t.references :owner, references: :users
  t.references :parent
end 

So, in short, if you were not using the schema_plus gem already, start doing so now :)

How to clean assets in rails 4

Gentle reminder, do not forget, in rails 4

rake assets:clean

seems to work, but actually does nothing. That is not entirely true: it only cleans the old assets, leaving the three most recent versions of the assets intact. So it is a like a mild cleaning, a throw-away-the-garbage-clean, a bring-those-old-clothes-you-never-wear-to-the-thriftstore-clean.

But sometimes that does not cut it. Sometimes, don't ask me why, building my assets does not seem to work, my code is just not being picked up. And then you need to do use brute force cleaning (throw everything out). Run

rake assets:clobber

to actually clean the assets. The logic or meaning is lost on my me (clobber?), but this works.

For my current job we have two 2.3.5 rails sites, of which I already succesfully upgraded one to rails 4. For the other we still need to start the migration, and we were asked to install new windows servers to run the rails servers on in the meantime (let's not digress why they choose windows, in a business environment Windows servers are still preferred, and let's be honest; they are easier to maintain and manage then *nix servers at first sight).

So whatever: I had to install the rails 2.3.5 on a new Windows 2012 server.

This proved problematic, since the new ruby 1.8.7 comes with the new rubygems, and this does not work nicely with rails 2.3.5.

So step 1: install the old ruby 1.8.7 (p302), the old gems and run the rails server. This worked.

But then I saw this one thing I really needed to improve. So I migrated the project to the latest 1.8.7 and rails 2.3.18, use bundler for gem dependencies. On my development box this worked like a charm (Macbook Pro). So then I deployed this back on the server, and then the following happened:

C:/Ruby187/lib/ruby/1.8/pathname.rb:290:in `[]': no implicit conversion from nil to integer (TypeError)
    from C:/Ruby187/lib/ruby/1.8/pathname.rb:290:in `chop_basename'
    from C:/Ruby187/lib/ruby/1.8/pathname.rb:343:in `cleanpath_aggressive'
    from C:/Ruby187/lib/ruby/1.8/pathname.rb:331:in `cleanpath'
    from C:/Ruby187/lib/ruby/gems/1.8/gems/rails-2.3.18/lib/rails/rack/log_tailer.rb:9:in `initialize'
    from C:/Ruby187/lib/ruby/gems/1.8/gems/rack-1.1.6/lib/rack/builder.rb:54:in `new'
    from C:/Ruby187/lib/ruby/gems/1.8/gems/rack-1.1.6/lib/rack/builder.rb:54:in `use'
    from C:/Ruby187/lib/ruby/gems/1.8/gems/rack-1.1.6/lib/rack/builder.rb:73:in `call'
    from C:/Ruby187/lib/ruby/gems/1.8/gems/rack-1.1.6/lib/rack/builder.rb:73:in `to_app'
    from C:/Ruby187/lib/ruby/gems/1.8/gems/rack-1.1.6/lib/rack/builder.rb:71:in `inject'
    from C:/Ruby187/lib/ruby/gems/1.8/gems/rack-1.1.6/lib/rack/builder.rb:73:in `each'
    from C:/Ruby187/lib/ruby/gems/1.8/gems/rack-1.1.6/lib/rack/builder.rb:73:in `inject'
    from C:/Ruby187/lib/ruby/gems/1.8/gems/rack-1.1.6/lib/rack/builder.rb:73:in `to_app'
    from C:/Ruby187/lib/ruby/gems/1.8/gems/rails-2.3.18/lib/commands/server.rb:95
    from script/server:3:in `require'
    from script/server:3

What was going on here? Apparently in the rails/rack/logtailer.rb is the following code:

class LogTailer
  def initialize(app, log = nil)
    @app = app

    path = Pathname.new(log || "#{::File.expand_path(Rails.root)}/log/#{Rails.env}.log").cleanpath

And the cleanpath is somehow crashing. I tried googling this, to no avail.

Of course:

  • rails 2.3.18: who uses that still?
  • ruby 1.8.7 is deprecated
  • and deploying on windows servers

So I was on my own. I tracked the error down to the cleanpath_agressive which called the chop_basename recursively to remove superfluous . and .. redirections.

I am guessing the problem is that on windows, a path starts with a drive-letter, like D:\ or C:\ which messes up the ending of the cleanpath_aggressive loop.

Instead of really diving in, the path handed down to cleanpath in my case, did not need any cleaning, and furthermore, an uncleaned path would still work.

So I added an initializer config\cleanpath_bug_fix.rb with the following code:

if RUBY_PLATFORM =~ /mingw/
  # on windows the Pathname.cleanpath crashes on windows paths, instead of fixing it thoroughly
  # just ignore it
  class Pathname
    def cleanpath_aggressive
      self
    end
  end
end

Now my rails 2.3.18, using ruby 1.8.7p374 runs on Windows Server 2012R2. Woot ;)

complete coverage with SimpleCov

When using SimpleCov in a very ill-covered project, I got amazingly good results: SimpleCov just did not count not-covered files. So files that were never used in our test-suite, were just simply being ignored. While I understand that approach, it did not feel good. I want to measure absolute progress, and I want to know how badly it really is.

So, on a mission to count all uncovered files with simplecov, I encountered an issue in their github repo. In itself it contained/mentioned two solutions:

  • in a three year old comment, a solution to set a starting base-line, and merge it after the tests. Unfortunately, it did not work completely: all files were added, but with complete coverage. Doh.
  • a pull request, claiming to fix it: while it did add all files, files that had before 100% coverage, where now no longer.

Let's get technical. The first solution added a baseline, with for all lines the value nil. The second added the value 0 for each line. The value nil is what SimpleCov uses internally to count for a never line: a line that never matters. Which is either an empty line, a comment line, or begin and end of a class.

The zero is line that is not covered (a 1 is a line that was covered). When merging lines for files that had coverage, I assume the base-line 0 takes precedence over the coverage-calculated nil and so we end up with a non-covered line in the merged result. Bummer.

So I added my spec_helper.rb as follows:

if ENV["COVERAGE"]
  require 'simplecov'
  SimpleCov.start 'rails'
  SimpleCov.coverage_dir 'coverage/rspec'

  all_files = Dir['**/*.rb']
  base_result = {}
  all_files.each do |file|
    absolute = File::expand_path(file)
    lines = File.readlines(absolute, :encoding => 'UTF-8')
    base_result[absolute] = lines.map do |l|
      l.strip!
      l.empty? || l =~ /^end$/ || l[0] == '#' ? nil : 0
    end
  end

  SimpleCov.at_exit do
    coverage_result = Coverage.result
    covered_files = coverage_result.keys
    covered_files.each do |covered_file|
      base_result.delete(covered_file)
    end
    merged = SimpleCov::Result.new(coverage_result).original_result.merge_resultset(base_result)
    result = SimpleCov::Result.new(merged)
    result.format!
  end
end

So, before the test runs, I create a "baseline" containing for all possible ruby files (might need to filter that later), a nil if the line is empty or the "ending end" of a file, and a zero otherwise. After the tests have run, I remove the covered files from the "baseline", and those are then merged with the result that get the final result.

Now maybe try to translate that into a pull request :)

[netzke] make action buttons toggle

I am currently converting my GIS application that needs two browser windows (one for the map, and one for the administrative data and functions), to a single window application using the netzke gems (which in turn rely on Sencha Ext js).

The netzke gems are extremely powerful, but there is a bit of learning curve for me, since both netzke and ext js are new. But netzke makes a lot of things a lot easier. It is not really rails-like, since it abstracts a lot away, and I am not yet completely convinced of the approach. But for my current goals it is ok.

I have a panel containing an openlayers map, and a toolbar with action buttons. The action buttons need to indicate state (drawing/measuring). So when clicked they have to be in a "pressed" state, and start with the action.

Although I found no example of this, achieving it was pretty easy. I defined my action buttons as follows:

action :measure_line do |c|
  c.id = "measure_line"
  c.icon = :ruler
  c.text = ""
end

action :measure_area do |c|
  c.id = "measure_area"
  c.icon = :surface
  c.text = ""
end

js_configure do |c|
  c.layout = :fit
  c.mixin
end

def configure(c)
  c.desc = "Contains the openlayers map"
  c.title = "Map"
  c.tbar = [:draw_point, :draw_line, :draw_polygon, :measure_line, :measure_area, :navigation_history]
  c.body_padding = 5
end   

I am only showing the actions for :measure_line and :measure_area because these are now the relevant ones I want to show. I need to give explicit id to the buttons, so that I can use Ext.ComponentManager.get to find them later on.

Nice thing to know is that the action handler by default receives the pressed button (yes!). This will allow us to just toggle it. So your mixed in javascript will look this:

{
    onMeasureLine: function(btn) {
      Map.toggleMeasuring(btn, 'line');
    },
    onMeasureArea: function(btn) {
      Map.toggleMeasuring(btn, 'polygon');
    },
}

Because I prefer to write coffeescript (not found how I can do that for the netzke mixins), I have a class Map in map.js.coffee containing all map-related functions. toggleMeasuring needs the button and the measurement-action to activate/deactivate

toggleMeasuring: (button, measurement_type) ->
  button.toggle()
  if 'pressed' in button.uiCls
    @measureControls[measurement_type].activate()
  else
    @measureControls[measurement_type].deactivate()

  # make sure the other measuring is automatically switched off
  for  key of @measureControls
    unless key == measurement_type
      @measureControls[key].deactivate()
      Ext.ComponentManager.get("measure_#{key}").toggle(false)

That is about the gist of it. I will be sharing more of my netzke experience and code soon. For the moment still impressed :)

generating pdf on heroku
I was investigating ways to generate pdf's in Ruby on Rails, but I had one enormous constraint: it had to deploy on heroku. There are two very different ways to generate pdf's in ruby:
  • use prawn: it is pure ruby, very powerful. It has it's own DSL, that unleaches all the power of building a PDF, but at the same time: it seems to be very hard and tedious.
  • use some sort of HTML to PDF conversion. In ruby there exists two gems: wicked_pdf and PDFKit, both use wkhtmltopdf under the covers. I dreamed of having a view magically converted to PDF.
I went for the second option. Furthermore, I choose wicked_pdf over PDFKit, because I felt the rails integration was better. It allowed my to just render a view which would automatically be downloaded as a PDF.

Setting up wicked_pdf in Rails to run on heroku

Luckily, getting it running on heroku proved to be incredibly easy:just including the correct gem with the binaries that work on heroku. In my Gemfile I added the following: [ruby] gem "wicked_pdf" gem "wkhtmltopdf-heroku", :git => 'git://github.com/camdez/wkhtmltopdf-heroku.git' [/ruby] And, then, inside a view you want to render as pdf, write something like [ruby] respond_to do |format| format.js format.pdf { render :pdf => "show", :header => { :font_size => '8', :right => '[page] of [toPage]' }, :footer => {:font_size => '8', :right => 'Generated by jottinx.com' } } end end [/ruby] Then, you will still have to create the view, show.pdf.erb. Just make sure your view renders HTML and it will be converted to PDF correctly. That is just awesome. Hope this helps.

The Problem

For jottinx I wrote a small piece of code that allowed to sort items using drag and drop. Of course, after writing it and making sure it works (manually), I want to make sure it keeps working. So I add a test, using cucumber. My scenario actually looks pretty straightforward: [ruby] @javascript Scenario: Dragging a book Given I authenticate as a "default user" And I add a new book with title "a-new-book" And I add a new book with title "another-book" And I add a new book with title "one-last-book" And I drag book "one-last-book" to the top Then book "one-last-book" is at the top of the list [/ruby] The difficult bit was: how do I implement the dragging?. Actually it seemed straightforward, because capybara has a method called drag_to. So I implemented the step like this: [ruby] When /^I drag book "([^"]*)" to the top$/ do |book_title| drop_place = page.find(:css, 'ul.sortable-books li:first') page.find(:xpath, "//a[@href='##{book_title.parameterize}']").drag_to(drop_place) end [/ruby] But, unfortunately, this did not work. I googled around a bit and found the following two similar questions: The short conclusion: it does not work, and it is a combination of how jquery implemented the sortable element, and the fact that the selenium driver does not support it yet. So, refraining to using the selenium driver directly does not help either. After some more googling, I found a similar question on stackoverflow, and there I found the solution.

The solution

Enter jquery.simulate.drag-sortable.js. It is a script that will allow you to simulate dragging in a sortable object by issuing a simple javascript command: [javascript] // drag item down one position in the list $('#itemToDrag').simulateDragSortable({ move: 1 }); [/javascript] If move parameter is negative, it will move up. And down if positive. If you include the js inside your project, you can easily test that out inside your Chrome console. It just works. Awesome piece of work. To use that in a step-definition, just write: [ruby] When /^I drag book "([^"]*)" to the top$/ do |book_title| page.execute_script %Q{ $('.sortable-books li:last').simulateDragSortable({move: -4}); } end [/ruby] Hope this helps :)
One recurring problem when doing Ruby on Rails development is a nested model form.
Nested Model Form: a single form that contains multiple, nested models.
For example a project with its tasks. A nested model form will allow you to create or edit, in 1 form, the project and each of its tasks. With the use of cocoon, this post will describe how to create a nested model form for some (all?) of the possible relations. All these examples start from the project, so we will just start with a dummy project as follows: [ruby] rails g scaffold Project name:string [/ruby] That's all we need to get started. In these examples i am using simple_form and slim (which looks a lot like haml, but is much faster).

The simple :has_many

The simple :has_many is the most occuring relation (unfortunately no scientific data to back that up). The simple :has_many is a relation where the child cannot exist without the parent, and where each child can have only one parent. A typical example is the project and his tasks. Each task is unique, each task only exists because of the project. If the project is gone, so are the tasks. First the models: [ruby] class Project < ActiveRecord::Base has_many :tasks accepts_nested_attributes_for :tasks, :reject_if => :all_blank, :allow_destroy => true end class Task < ActiveRecord::Base end [/ruby] The accepts_nested_attributes_for method makes sure that when posting the project-data, it will accept the attributes for the nested model. Secondly we add the views. Inside the project _form.html.slim we add [ruby] #tasks = f.simple_fields_for :tasks do |task| = render 'task_fields', :f => task .links = link_to_add_association 'add task', f, :tasks [/ruby] Create a partial called projects/_task_fields.html.slim which looks like: [ruby] .nested-fields = f.input :name = f.input :description = f.input :done, :as => :boolean = link_to_remove_association "remove task", f [/ruby]

The nested :has_many

Actually this is a simple extension of the previous example. For instance, a project with tasks, and each tasks has sub-tasks. Our models: [ruby] class Task < ActiveRecord::Base has_many :sub_tasks accepts_nested_attributes_for :sub_tasks, :reject_if => :all_blank, :allow_destroy => true end class SubTask < ActiveRecord::Base end [/ruby] Edit the projects\_task_fields.html.slim : [ruby] .nested-fields = f.input :name = f.input :description = f.input :done, :as => :boolean .sub_tasks = f.simple_fields_for :sub_tasks do |sub_task| = render 'sub_task_fields', :f => sub_task .links = link_to_add_association 'add sub-task', f, :sub_tasks = link_to_remove_association "remove task", f [/ruby] and add the view projects\_sub_tasks.html.slim (which bears a lot of resemblance to the previous tasks-partial): [ruby] .nested-fields = f.input :name = f.input :description = link_to_remove_association "remove sub-task", f [/ruby]

The look-up or create :belongs_to

A simple :belongs_to would mean that the parent would already exist, and a simple look-up (dropdown list) would suffice. If the parent is always unique, it can be solved in the same way as the :has_many. So, let's describe a solution for the case where we either select the item from a list, or create a new one. Our project has an owner (which is a person): [ruby] class Project < ActiveRecord::Base belongs_to :owner, :class_name => 'Person' accepts_nested_attributes_for :owner, :reject_if => :all_blank end class Person < ActiveRecord::Base end [/ruby] inside the _projects/_form.html.slim we add the following: [ruby] #owner #owner_from_list = f.association :owner, :collection => Person.all(:order => 'name'), :prompt => 'Choose an existing owner' = link_to_add_association 'add a new person as owner', f, :owner [/ruby] Here we use the built-in way from simple_form to select an association from a look-up list: f.association. But secondly, we place the link_to_add_association that will render the partial projects/_owner_fields.html.slim and create a new Person and link to him. The partial itself is pretty straightforward: [ruby] .nested-fields = f.input :name = f.input :role = f.input :description = link_to_remove_association "remove owner", f [/ruby] Now if you implement this, the form is functioning but not really user-friendly. What we want is that when clicking the add a new person as owner-link, that the drop-down box and the link itself disappear. When we choose to remove the to-be created owner, that the drop-down and add-link reappear. So we need to add a bit of extra javascript. Open up app\javascripts\projects.js and add: [javascript] $(document).ready(function() { $("#owner a.add_fields"). data("association-insertion-position", 'before'). data("association-insertion-node", 'this'); $('#owner').bind('insertion-callback', function() { $("#owner_from_list").hide(); $("#owner a.add_fields").hide(); }); $('#owner').bind("removal-callback", function() { $("#owner_from_list").show(); $("#owner a.add_fields").show(); }); }); [/javascript] The first two lines control where the new partial (the new owner) will appear. Upon insertion or removal, cocoon will trigger callbacks that are defined on the parent-container of the add-link. The last lines add those callbacks, and these will make sure that the link and dropdownbox will be hidden or shown again.

The :has_many :through relation

A classic example of this relation are tags. Something has tags, those can be chosen from an exisiting list of tags, one or more, or one could create new tags. This relation is, again, an extension of the previous solution. This solution is a bit more complicated, because there is a bit more javascript involved here. The models: [ruby] class Project < ActiveRecord::Base has_many :project_tags has_many :tags, :through => :project_tags, :class_name => 'Tag' accepts_nested_attributes_for :tags accepts_nested_attributes_for :project_tags end class ProjectTag < ActiveRecord::Base belongs_to :tag belongs_to :project accepts_nested_attributes_for :tag, :reject_if => :all_blank end class Tag < ActiveRecord::Base end [/ruby] inside the projects/_form.html.slim add: [ruby] #tags = f.simple_fields_for :project_tags do |project_tag| = render 'project_tag_fields', :f => project_tag = link_to_add_association 'add a tag', f, :project_tags [/ruby] Create a file projects/_project_tag_fields.html.slim containing: [ruby] .nested-fields.project-tag-fields #tag_from_list = f.association :tag, :collection => Tag.all(:order => 'name'), :prompt => 'Choose an existing tag' = link_to_add_association 'or create a new tag', f, :tag = link_to_remove_association "remove tag", f [/ruby] And create another file projects/_tag_fields.html.slim: [ruby] .nested-fields = f.input :name, :hint => 'how should it be tagged' [/ruby] When entering all this, it should be working, but it looks a bit confusing. The dropdown box does not disappear (as before). So we need the following bit of javascript to projects.js: [javascript] $("#tags a.add_fields"). data("association-insertion-position", 'before'). data("association-insertion-node", 'this'); $('#tags').on('cocoon:after-insert', function() { $(".project-tag-fields a.add_fields"). data("association-insertion-position", 'before'). data("association-insertion-node", 'this'); $('.project-tag-fields').on('cocoon:after-insert', function() { $(this).children("#tag_from_list").remove(); $(this).children("a.add_fields").hide(); }); }); [/javascript] While this javascript is doing almost the same as with the :belongs_to it is a bit more complicated, because the tag-partial is not yet inside in the page, so neither is the parent-div. Once that is clear, it is actually quite obvious what it does. I hope. Upon insertion of a new (project)tag, it also adds the callbacks for the insertion and removal of a new tag.

Example Code

All these examples are demonstrated in a working project: cocoon-simple-form-demo. If you can think of more relations you would like to see solved, or see an example of, please let me know. Hope this helps.
Take the following steps, by preference install this in a specific gemset [ruby] rvm gemset create rails31 rvm gemset use rails31 gem install bundler gem install rails --pre [/ruby] These steps will create a new gemset and start using it, then install bundler and rails 3.1 (which is still pre-release at the time of writing). Now in rails 3.1 there is a new requirement: you will need a javascript runtime. You can see the full list here. If you are on linux and using straight ruby then therubyracer is just fine. Type [ruby] gem install therubyracer [/ruby] Creating a new rails-project always poses the same problem: creating a new project, with your set of gems, which all have to installed, generators need to be ran ... Rails Wizard solves that. [ruby] gem install rails_wizard [/ruby] Type rails_wizard list to see all possible templates. To create a new rails 3.1 project using jquery, rspec, haml, devise you just need to write: [ruby] rails_wizard new your_new_application_name -r jquery rspec haml devise [/ruby] To go full out on options, you could write: [ruby] rails_wizard new your_new_application_name -r jquery rspec haml devise sass settingslogic git [/ruby]

Note

If you are using a specific gemset for your project, please remember to add your javascript runtime (e.g. I used therubyracer) to your Gemfile. ... and you are ready to go! Happy coding :)
configuring TeamCity to run rails projects
In our team we are very happy users of RubyMine, by JetBrains. Now JetBrains also has a continuous integration server, called TeamCity, and it is also capable to run rails rake tasks. Installing TeamCity is close to a non-operation, as described on their website. Just download the package, extract, and run bin/runAll.sh start. Then you can browse to http://localhost:8111 and you are up and running. Unfortunately, getting the build to actually run was a bit more complicated in our case. Our prerequisets:
  • we do not check in any configuration file (like config.yml, database.yml, ...) so each developer can have their own settings.
  • we use bundler to manage our gem dependencies
We use the rake runner, but as you might know: rake tasks will not run unless the bundle is up to date. Luckily we are not the first to encounter that problem, and the suggested solution is actually quite simple, although a bit backward: use a custom rakefile, that will first run the bundle install, then require the actual Rakefile from which you can run the tasks. To create the log folder, the css files (we use sass), the config-files, i had to create specific rake-tasks. We also had a problem that our database was always somehow corrupt. Upon thorough investigation, i was able to pin-point this on the rake db:test:prepare task, that actually does not load the database from the schema.rb, but tries to clone the development database! On our continous integration server there is no development database! So i cleared the db:test:prepare task. Our custom rakefile then looked as follows: [ruby] directory "log" task 'copy_rakefile' do cp 'Rakefile', 't_rakefile.rb', {:verbose => true} end task :bundle_install do sh 'bundle install' end task :sass do Sass::Plugin.update_stylesheets end task :default => [:bundle_install, 'copy_rakefile', 'log'] do RAILS_ENV = ENV['RAILS_ENV'] = 'test' require File.dirname(__FILE__) + '/t_rakefile' # launch rake tasks from the original Rakefile Rake::Task["log:clear"].invoke Rake::Task["teamcity:init_config"].invoke Rake::Task['db:test:prepare'].clear Rake::Task["db:reset"].invoke Rake::Task["environment"].invoke Rake::Task['sass'].invoke Rake::Task["test"].invoke Rake::Task["spec"].invoke Rake::Task["cucumber"].invoke end [/ruby] We had to copy the Rakefile, so we could later require it as a straight ruby-file. Note that to disable the db:test:prepare i used the following code: [ruby] Rake::Task['db:test:prepare'].clear [/ruby] Simple, isn't it :) It's also only cleared when running the teamcity:build, just as we want it. Now we got this working, it is time to enjoy the goodies TeamCity offers us, like spreading over different build agents.
Arrrrcamp 29/10/2010
The fourth and most international of all arrrrcamp editions. Also the first paying edition. As i said in my post about the previous edition, i have been to all editions, and it is awesome to see how it has grown. The schedule was amazing, with an amazing array of international speakers, of which for me especially Yehuda Katz i looked forward to. The biggest problem i had was that two of my collegues gave a talk at the same time, both of which i wanted to see. Choices choices choices :) Let me give a quick impression of the talks i did see:

Keynote by Carl Lerche:

To be frank, i had totally no idea who Carl was. But his talk was very nice, talking about the speed improvements Rails does, under the covers, unknown at least to me, to get your site visible as soon as possible. Also the speed improvements they will take in the next version 3.1 and 3.2 to improve this even further. In short, they look for ways to serve the HTML so that a browser can download the page and all the assets quicker. Very new to me, touched some very interesting subjects as automatic spriting, the cache manifest ... But in the end, there is one thing that Rails can't solve: latency. That is the core time it takes to reach a server and get back to the client. A mere ping. If you want to reduce latency, you either make sure your servers are really close to all clients, ... or ... you work in a different way, and let your client handle more stuff. In short: using a minimal HTML which is only downloaded once, using the cache manifest and lots of javascript: back to client side code? Hold that thought, we'll come back to that later.

Escaping the Sludgy Swamp of Slow Tests by Joseph Wilk

This is a problem we are getting into at work. We try to solve that by running our tests in parallel. Joseph explains the routes they had taken in their team.
  • First off: use a lot of EC2 instances to reduce time. They went from 4hours build time to 11minutes, cheering developers, but very angry boss :) EC2 is not cheap!
  • So then they introduced some kind of low-level caching, which will cache sequences of SQL commands and returned a cached result. This sounds very interesting as that is what you see most in tests: a lot of similar db-setup to actually test something small.
  • Thirdly a great idea was to divide your tests into different sets, and only run those that are most likely to fail. Those that always succeed you only run nightly, and the ones that sometimes fail you also seperate so you can more easily test those, check loggings, and hopefully fix them.
I also remembered the suggestions to use capybara together with env.js to be able to test your javascript pages quicker. Good tips, and i especially like the graphics (and humour) on the slides :D

Refactoring++ : Refactor Everything A.T.F.T by Alain Ravet

A talk by a collegue of mine (had to miss Elise's talk, bummmmer, but her talk is recorded so i hope to see that later). I know most of his refactoring tips, because we have paired extensively and i have learned a lot by doing that. Alain took a small piece of code from the internet, an example to solve the Towers of Hanoi: the code was small and compact and seemed to be ok. But then Alain was able to find and show a lot of improvements. That was great. Aside from the standard refactorings (from the book) Alain also stressed the scanability of the code. This is also a pet-pieve of mine. Code has to be formatted nicely and correctly. In teams it is preferred that classes follow a bit the same structure, e.g. inside rails it is easy to make agreements where to expect the filters, constructors, public methods, ... Also Alain stressed the point to know your tools and your language. Ruby is a very expressive and compact language: use that to your benefit! Know the tool you are using, whether it is Rubymine, Textmate or VIM. And practice. Do code kata's, download some code and refactor-refactor-refactor.

Making it an Anti-Pattern is Not Enough by Timothy Payton and Sebastian Roebke :

I saw these guys last time, and i was very impressed by them at that time. Probably because what they said struck a chord, and matched my own experiences. This time, i found them less persuasive. I think maybe because my expectations were too high, they re-iterated a lot they said last time (or so it felt), the title was not explained anywhere (or i missed that), and i guess they were also having troubles with the single microphone (being two speakers) that made them more uncomfortable. Still a very nice talk, and Xing seems to be a very nice place to work.

Lightning Talks

  • Friendly attributes: apparently, in mysql, adding a columns is a very expensive operation. So this guy looked for alternatives, and came up with friendly attributes. This could be a very nice tool for someone if you are forced to use MySql (e.g. by your hosting provider). Otherwise i would suggest to use Postgresql.
  • A postgres textsearch without any extra server: this seemed very interesting and promising, but also very non-existant at the moment. He was in the process of turning it into a gem. Wait and see.
  • Yehuda's 10 minutes about concurrency in MRI: inspired by Elise's talk about concurrency, Yehuda dug a bit deeper into MRI threads and possible advantages. Unfortunately none of the big servers (passenger or unicorn) make any use of that. Only thin --threaded would profit from that. This really was a lightning talk! A lot of information in a short span, very interesting!
  • @Defv's recipe for a good rails rumble: after his third Rails Rumble Jan gave us the recipe: 1) a knock-out idea but simple enough so any visitor/judge would get it, simple enough to implement in 48 hours, and 2) good design. With the next emphasis being: simple! simple! simple! Reminds me a bit of what 37signals always says.
  • Wicegrid: i have used this grid in a few big projects i did, and really liked it. Nice that Yuri showed it, hope some more people start using it.
  • Restart of BRUG: this was nice news too. I have seen that amsterdam.rb and rotterdam.rb exist, and was wondering why we don't have such a "working" thing in Belgium. But the guys from Openminds offered to revive it, they will support the first three sessions, and then other sponsors would need take over (volunteers from the public immediately jumped in). Meetings would take place in different cities, so i am very curious how this would work out. That would be very nice ;)

Closing Keynote by Yehuda Katz:

Yehuda recently switched jobs, and together with Carl Lerche they are now both actively working on a framework called Sproutcore. In short, Sproutcore (or SC) is a javascript MVC framework. So, MVC on the client. In the talk from Carl we learned that the big problem is latency, so one way to improve or remove that, is instead of developing your web-application in the Rails way, is to use a more client-centric way. Yehuda then showed us a few moving parts of Sproutcore, which he intends to extract in seperate libraries so you could use them more easily. They also stressed that Carl and Yehuda are aiming for a very tight integration with Rails in the future. Altough it is not entirely sure what or how that will be be. You would be using Rails to just serve json-data. Rails still has its very strong points to be used in the backend for something like that. His talk left me a bit baffled, to be honest. The big advantage for me of the Rails framework is that there is one language, one platform to develop in, to cover both server and client side. That is a big advantage. It is also a big advantage compared to using AIR or Silverlight on the client, because you remove the duplication of all the models on both sides (in Rails you only need them on server side). That said, i do believe the future lies in heavier clients, like gmail and twitter shows us. The bindings that Yehuda showed, reminded my very much of the bindings offered in WPF and Silverlight. Possible advantages of Silverlight: with Ironruby you have the option to run ruby in the client as well. But, as the video-section of arrrrcamp themselves show: Silverlight it is not ready for all platforms. It does not work on my ubuntu 10.10. I am thinking now would be a good time to revive the idea of a ruby vm on javascript. In no specific order:
  • Hotruby was very promising but seems dead. With the speed improvements in current javascript vm's it would seem awesome to get some ruby on javascript too.
  • rubyjs: compiles ruby to javascript to be run in the browser
  • Js.class could be an alternative approach: use the classes and paradigms of ruby in javascript.
  • red: writes like ruby and runs like javascript
  • coffeescript: not quite ruby, that compiles into javascript
I will dig into these alternatives in a later post :) The biggest advantage of sproutcore over extjs, AIR or Silverlight is that sproutcore is truly open source. Exciting times ahead :)

To conclude

As usual I really enjoyed arrrrcamp, the organisation was great, i enjoyed seeing and talking to a lot like-minded people, and the talks were really though-provoking this year. Setting sail to the next edition!
I am developing several rails projects. Some are still rails 2.3.9, and some are rails3. But using those at the same time, on the same machine can cause some trouble. I use bundles for all my projects, so the gem dependencies are managed. But still, somehow, when running my tests, i get into trouble. If i run spec, it complains that it is deprecated. Rubymine doesn't support running tests in my rails 2 project. The solution? Use gemsets! Gemsets are a feature from rvm, the Ruby Version Manager. But with gemsets rvm does more than just manage your ruby version! A gemset is a set of gems, which you can switch at will. So i am using a gemset per project. Creating a gemset is easy: [ruby] rvm gemset create some_name [/ruby] Using a gemset is equally simple: [ruby] rvm gemset use some_name [/ruby] Showing which gemset is in use: [ruby] rvm gemset name [/ruby] Showing the list of possible gemsets [ruby] rvm gemset list [/ruby] By default, you are using an unnamed gemset, which contains all the gems for your ruby-version. Per project you can create a new gemset. Per gemset you have to re-install your gems, but luckily, using bundler, that is as simple as doing bundle install. Now comes the good part:
  • Rubymine supports using gemsets, so you can select your gemset to work with, and each time your project is opened, it uses the correct gemset and the correct corresponding scripts (e.g. spec)
  • If you place a .rvmrc file in your project-folder containing rvm default@projecta, every time you cd into that folder it will select the default ruby and the projecta gemset. Awesome!
getting rcov working with rspec 2.0.0.rc
I had troubles to get rcov working with the latest rspec2 release (since beta23 and now 2.0.0.rc). I got the same error every time: [bash] `require': no such file to load -- spec_helper (LoadError) [/bash] But luckily, somebody found the problem and it is extremely easy to fix. Just add -Ispec to your rcov task. The rake spec:rcov does not work for me (as it needs to be fixed). So i added my own task (add this code to the end of Rakefile or add in a seperate file rcov.rake inside lib/tasks) : [ruby] desc "Run all specs with rcov" RSpec::Core::RakeTask.new("test_cov") do |t| t.rcov = true t.rcov_opts = %w{--rails --include views -Ispec --exclude gems\/,spec\/,features\/,seeds\/} end [/ruby] and then you can run the task by typing [bash] > rake test_cov [/bash] in your rails root folder.
I have created a Rails3 application, started with Haml/Sass and finding it awesome. I am also trying to do unobtrusive javascript. Before, in Rails 2.3, I would have expected a remote-form to have an :update attribute, where you could specify a selector where the response of the remote method would be rendered. Now it needs to be done differently: my controller function will render a ".js" view, which will do the necessary actions itself. This makes it more library agnostic, and i prefer jquery. So for instance i have a controller action "search", and i have a corresponding "search.js.erb" as follows: [ruby] $('.grid').replaceWith('<%= escape_javascript(render :partial => 'list') %>') [/ruby] This works. It will replace the html of the element with a class grid with the html from my rendered partial. I started out trying to achieve that in HAML, and failed at first. So first i created the above ERB code which worked. Now my task seemed simpler: translate this seemingly simple line to HAML. My first naive approach was to do the following: [ruby] = "$('.grid').replaceWith('#{escape_javascript(render :partial => 'list')}')" [/ruby] But this just places the html as a readable string inside my page. Even i do something simple like [ruby] = "$('.grid').replaceWith('<h3>Text</h3>') [/ruby] i see the actual characters [ruby] <h3>Text</h3> [/ruby] and not the markup. But i need to contain the javascript inside the double-quotes or haml will not recognise it, and can not interpolate my ruby there. After looking through the HAML reference, actually the solution was incredible simple. Once more. [ruby] != "$('.grid').replaceWith('#{escape_javascript(render :partial => 'list')}')" [/ruby] The != unescapes HTML (as opposed to the standard =). This is exactly what we need of course.
skipping slow tests in rspec2
For Test::Unit there is a module called skippy, that allows you to skip for instance slow tests. I was looking for a way to achieve the same in Rspec. I am currently developing against Rails 3.0.0.rc, and using rspec 2.0.0.beta.19 and could not find anything. So i asked on the rspec mailing list and got a great answer! It is actually present in rspec2, right out of the box, as a feature they needed for their own tests. Read the description by David himself here. He shows a lot of very good examples, even filtering tests based on ruby-version is now possible. Awesome. In short, you would do something like : [ruby] describe "something", :slow => true do it "should be blabla" end describe "something else" do it "should be a slow test", :slow => true do ... end it "should be a fast test" do .. end end [/ruby] then you can write the following to skip the slow tests : [ruby] # in spec/spec_helper.rb RSpec.configure do |c| c.filter_run_excluding :slow => true end [/ruby] and to run only the slow tests: [ruby] # in spec/spec_helper.rb RSpec.configure do |c| c.filter_run :slow => true end [/ruby] and to filter on slow tests, and run all tests if no slow tests are available: [ruby] # in spec/spec_helper.rb RSpec.configure do |c| c.filter_run :slow => true c.run_all_when_everything_filtered = true end [/ruby] Awesome! :) For the moment there is no command-line support, but in the spec/spec_helper.rb you could also check for any environment variables to do the slow or fast test-set. What i personally needed, was something that would either run all tests, or run all tests excluding a few (i have a test-suite for an sms-gateway and do not want to send sms-es all the time). So i handled that like this: [ruby] # in spec/spec_helper.rb RSpec.configure do |config| if ENV['SEND_REAL_SMS'] != "true" config.filter_run_excluding :send_sms => true end end [/ruby]