Design and Implementation of compleX Information Systems

Write

ruby on rails 34 ruby 26 rails3 17 rails 15 oracle 11 rspec 9 rspec2 7 jquery 7 ubuntu 5 javascript 5 windows 5 activerecord 3 refactoring 3 geoserver 3 gis 3 arrrrcamp 3 actionmailer 2 oracle spatial 2 tdd 2 postgis 2 routing 2 rvm 2 mongoid 2 csharp 2 thin 2 win32 2 gem 2 rails4 2 git 2 service 2 haml 2 cucumber 2 view testing 2 i18n 1 displaysleep 1 spatial 1 gemsets 1 wubi 1 oracle_enhanced_adapter 1 migrations 1 watchr 1 ci 1 plugins 1 coderetreat 1 ie8 1 ssl 1 oci 1 nested model form 1 wcf 1 11.04 1 jsonp 1 ruby-oci8 1 teamcity 1 engines 1 pgadmin 1 soap 1 content_for 1 word automation 1 plugin 1 capybara 1 xml 1 bootstrap 1 migrate to rails3 1 mvc 1 unity 1 rendering 1 word2007 1 x64 1 limited stock 1 fast tests 1 pl/sql 1 delayed_job 1 pdf 1 test coverage 1 optimization 1 processing 1 borland 1 method_missing 1 cross-browser 1 devise 1 schema_plus 1 mongo 1 mongrel 1 dual boot 1 usability 1 mongrel_service 1 dba 1 mission statement 1 model 1 metadata 1 rcov 1 exceptions 1 image_tag 1 attachments 1 bde 1 css 1 yield 1 ajax 1 generative art 1 rails-assets 1 coordinate systems 1 submodules 1 netzke 1 ora-01031 1 authlogic 1 postgresql 1 shopping cart 1 agile 1 fast_tagger 1 subjective 1 wice_grid 1 generators 1 nvidia 1 mongodb 1 etsyhacks 1 staleobjecterror 1 session 1 jeweler 1 wordpress hacked 1 jasmine 1 heroku 1 rjs 1 life 1 unobtrusive-javascript 1 render_anywhere 1 html5 1 rails31 1 json 1 cocoon 1 mingw32 1 observe_field 1 osx 1 actionwebservice 1 testing 1 debugging 1 strings 1
Technology
[windows] Configuring ruby/git to work behind a proxy

While I develop on OS X, and by preference deploy on Unix/Linux machines, for some of our clients we have to deploy on Windows servers. An IT-department will only allow machines/OSes they can manage themselves. While there are some drawbacks to this, most of them can be worked around perfectly.

One of this those "difficulties" is configuring a proxy. In general, on Windows configuring a proxy means: opening Internet Explorer/IE Edge, editing the connection details and adding/configuring the proxy (I find this completely baffling, did Microsoft not get some kind of anti-trust lawsuit because of the too tight coupling of Internet Explorer and Windows OS? anyways ... old news probably).

Git/ssh

To make sure git (using ssh) uses the proxy correctly, you have to open your $HOME/.ssh/config file, and add at the top:

 ProxyCommand /bin/connect.exe -H 10.52.111.111:80 %h %p
 
 Host github.com 
     [...snipped..]

you can also add the proxy configuration for each Host separately, but in my most cases you will want to use a shared/general proxy (I guess). Replace the shown IP-address with your proxy obviously.

Ruby

For ruby it is actually quite simple: you have to define the environment variable http_proxy (I define it as lower case, I think upper case should also work, but not entirely sure).

[add screenshot]

Javascript

We also run some puppeteer scripts from ruby, and to configure the proxy there, so in a puppeteer script, when creating the browser we have to hand down the proxy details. E.g. something like:

const browser = await puppeteer.launch({
    args: [
        '--proxy-server=10.52.111.111:80',
        '--proxy-bypass-list=localhost,10.200.*.*',
        '--no-sandbox',
        '--disable-setuid-sandbox',
        '--ignore-certificate-errors',
    ]
});
const page = await browser.newPage();
More ...
Technology ruby on rails rspec
[RSPEC] Handling the MockExpectionError "does not implement"

I was writing a test for my helper which would, if the user is authorised to perform an action, show the actual link, and otherwise show the title or nothing (depending on what is wanted).

I make use of the vigilante gem which I should still write an article about, but this gem allows to store authorisations in the database and can be context-specific (e.g. you you can be assigned different roles for different "contexts", which could translate to organisations, projects, ... whatever applies to your setup). This gem adds a helper method is_allowed_to? to determine if one has the permissions to perform a certain given action.

When I wanted to mock this in my helper, I assumed a simple

     expect(helper).to receive(:is_allowed_to?) { true }

would suffice. However, I got a very strange error:

RSpec::Mocks::MockExpectationError: #<#<Class:0x00007fe2dc84e010>:0x00007fe29c83dc78 ....snip some very long things ... >> does not implement: is_allowed_to?

This is actually a great feature from Rspec: it checks if the thing you want to mock/stub actually exists on the original object, but in this case a helper (or a view) in Rails can access a lot of helper methods which are implicitly loaded.

This behaviour can be controller by setting the RSpec::Mocks.configuration.verify_partial_doubles to false. Of course I do not want to disable this for my entire test-suite, but just locally for the single test or single spec file.

So in my spec I temporarily disable the checking for the existence of doubles, as follows

require 'rails_helper'

RSpec.describe MapHelper do
  before(:all) do
    RSpec::Mocks.configuration.verify_partial_doubles = false
  end

  after(:all) do
    RSpec::Mocks.configuration.verify_partial_doubles = true
  end

  it "has the correct method" do
    expect(helper.respond_to?(:map_checked_link_to)).to eq(true)
  end

  context "map_checked_link_to" do
    context "a reachable feature" do
      before do
        @pipe = FactoryBot.create(:pipe)
      end
      it "renders a normal link" do
        allow(helper).to receive(:is_allowed_to?).and_return("XXX00")
        expect(helper.map_checked_link_to("Pijpstuk 123", @pipe, {})).to eq("<a href=\"/pipes/208\">Pijpstuk 123</a>")
      end
    end
    context "a blocked feature" do
      before do
      end
      it "does not render anything by default" do
        allow(helper).to receive(:is_allowed_to?) { false }
        expect(helper.map_checked_link_to("Pijpstuk 123", "", {})).to eq("Pijpstuk 123")
      end
      it "renders the title if so specified" do
        allow(helper).to receive(:is_allowed_to?) { false }
        expect(helper.map_checked_link_to("Pijpstuk 123", "", {}, true)).to eq("Pijpstuk 123")
      end
    end
  end

end

I also stumbled upon a PR implementing a method without_verifying_partial_doubles which takes a block which would do exactly the same. So one would be able to write

it "does something weird with mocks" do
  without_verifying_partial_doubles do
    ...
  end
end

But it did not work for me. Not sure if that is because the rspec version I am using in this project is too old, or the example I found was outdated.

More ...
Technology ruby on rails
[rails] Using ActionCable in production when using passenger+apache2

So on the server for work we use apache2+passenger to host all our rails servers. We now have about 10-ish, not too much, not all are used heavily, all are long-living, most are still rails 4. On my own server, with a similar setup, the rails versions vary from 4 to 6.

Recently I started a new project, and of course used rails 6. And I had the awesome idea to try out stimulus-reflex as this seems an interesting addition in the js front-end world. It really is a very interesting and different approach, and since the new/upcoming version 3.3 includes morphs, actually well-suited for our use-case which almost always involves a leaflet map (so we definitely need control over which area of the page to refresh). It works! Great. I will write in more detail about that at some other time.

So I had a working part of the website, enriched using stimulus/stimulus-reflex, and now I wanted to deploy it.

Only to realise ... passenger does not support ActionCable on apache2. ActionCable is one of those new tools I still need to investigate in detail, but yes, stimulus-reflex uses ActionCable to get a highly efficient channel between the browser and the server.

So now I am presented with two options:

  • we replace apache2 on our server with nginx. Frankly: this does not feel like a bad thing to do, however I am a bit hesitant to do this, since this seems to be a larger operation and will cause some down-time for our other applications.
  • use passenger-standalone and make apache2 proxy to it ---ha this seems interesting!

So I had to perform a few steps:

  • configure action-cable for deployment
  • install and run passenger standalone
  • change my apache2 config for my site

Configure ActionCable

Actually, this is amazing, but in development everything just works. Of course in production there are apparently a lot of options to deploy it.

I choose for the simplest setup: let my rails process host the websockets (and thus passenger), and use postgresql as the backend (I am not using redis yet).

So in my config/routes.rb I had to make sure we mount the ActionCable server

   mount ActionCable.server => '/cable'

In my config/environment/production.rb I specified our allowed domains:

  config.action_cable.allowed_request_origins = [ 'http://geotrax.be', /http:\/\/geotrax.*/ ]

And to use postgresql instead of redis is actually really simple (we were already using postgresql as our db backend). In your config/cable.yml just write

 production:
   adapter: postgresql 

Install passenger-standalone

I imagined this was easy, but it was only until I got all the steps correctly it worked. In short:

First I added the passenger gem to my Gemfile in the production block and move the puma to :development/:test

then create a Passengerfile.json file specifying the environment, port, user and instance_registry_dir

{
  // Run the app in a production environment. The default value is "development".
  "environment": "production",
  // Run Passenger on the given port. In this example, we use port 80,
  // the standard HTTP port.
  "port": 4000,
  // Tell Passenger to daemonize into the background.
  "daemonize": true,
  // Tell Passenger to run the app as the given user. Only has effect
  // if Passenger was started with root privileges.
  "user": "geotrax",
  "instance_registry_dir": "/srv/geotrax/tmp"
}

Somehow we still had to start passenger with the correct instance-registry-dir (specifying it in the Passengerfile did not work?). So we start passenger as follows

RAILS_ENV=production bundle exec passenger start --instance-registry-dir=/srv/geotrax/tmp

Then we have a running/waiting passenger instance. Now all that remains ...

Configure apache2 correctly

This was actually the hardest part. Also the passenger document just glosses over the subject. It is easy to write an apache site configuration that passes all http requests to passenger, but the websockets ... ?

To start you need the correct modules

sudo a2enmod proxy proxy_http proxy_wstunnel rewrite

(in my case this ment just adding the proxy_wstunnel mod, the other were already installed/used)

Then I edited my site.conf as follows:

<VirtualHost *:80>
    DocumentRoot /srv/geotrax/applications/geotrax/current/public

    ServerName  www.geotrax.be
    ServerAlias geotrax.be

    ErrorLog  /srv/geotrax/applications/geotrax/current/log/error.log
    CustomLog /srv/geotrax/applications/geotrax/current/log/access.log combined

    RewriteEngine on
    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteCond %{HTTP:Connection} upgrade [NC]
    RewriteRule /(.*) "ws://127.0.0.1:4000/$1" [P,L]

    ProxyPass / http://127.0.0.1:4000/
    ProxyPassReverse / http://127.0.0.1:4000/
    ProxyRequests Off

</VirtualHost>

Since we are using both http and websockets from the same domain, we are using the rewrite rules as explained in mod_proxy_wstunnel documentation

This took me a #$@!#$$%@#@ while to figure all out, I hope this helps someone to get started a little quicker (even if it is me the next time).

Next step: switch out apache2 and use nginx (but for now it is a little less urgent).

More ...
Technology bootstrap ruby on rails
[Bootstrap 4] Closing a modal programmatically

One of the UX patterns I like/prefer is to have some kind of in-place editing. So not to send the user to a completely new page or context, but edit or change a feature in place. One of the tools I frequently use for this is opening a modal.

This modal shows a mini-form, and will post the information to the server. There are two ways to handle this: we just post HTML and let the server decide what is rendered, or ... we post using js. In rails this is easy: we add remote: true.

For instance, I have a small form to resend a mail, but I want my users to overrule the email-address if they want to. The view looks something like this:

= link_to "(Re)send Email", '#', class: 'btn btn-large btn-info', 'data-toggle' => "modal", 'data-target' => "#resend-plan-request-#{plan_request.id}-email"

.modal.resend-plan-request.fade{id: "resend-plan-request-#{plan_request.id}-email"}
  .modal-dialog{role: "document"}
    .modal-content
      .modal-header
        %h4 Re(send) email for PlanRequest #{plan_request.external_id}
        %button.close{type: "button", 'data-dismiss' => "modal", 'aria-label' => "Close"}
      .modal-body
        = form_tag resend_admin_plan_request_path(plan_request), class: 'form', method: :post, remote: true do |f|
          .alert.alert-info.small
            This will allow either re-sending mails that clients/users claim have not received, or just for testing.

          .form-group
            = label_tag :emails, "Emails", class: 'form-label'
            .form-field-with-hint
              = text_field_tag :emails, plan_request.delivery_email, class: 'form-control', placeholder: 'Enter emails ...'
              %small.form-hint
                List the intended recipients, separate multiple emails using comma's e.g.
                %br
                email@abc.com, email2@def.com
          .form-group
            = submit_tag "Send Email", class: 'btn btn-primary', data: {disable_with: 'Sending ...'}
            = link_to 'Close', '#', class: "btn btn-secondary", 'data-dismiss' => "modal"

So for clarity: we have a small form in the modal, that will allow users to resend an email, and overrule the emails if wanted. We post the contents of the form to the resend action of the PlanRequestsController, which will actually do whatever is needed to send the email, and provide feedback to the user. I do not want to take the user to a new page, I just want to close the modal using js. According to the documentation this is simple, just do something like $(".modal").modal('hide');

So we write this inside resend.js.haml view

:plain
  toastr.success('#{@success_message}');
  $(".modal").modal('hide');

However I noticed that this leaves a .modal-backdrop div covering the entire screen, rendering a lot (or everything) unworkable. What seemed to be the case:

  • opening a modal using a button, adds 2 .modal-backdrop and closing a modal using the close-button(s) removes both
  • opening a modal using .modal('show') only adds a single .modal-backdrop, which is removed by the .modal('hide')

Problems:

  • controlling the modal using only buttons work
  • controlling the modal using only js code works
  • mixing (opening with button and closing with js) does not

I have two buttons that can close my modal: the x at the top, and a close button in the form, so I figured naively this bug (behaviour) was caused by the second close button for some reason. Unfortunately this is easy to verify (refute): if I remove the close button, it still adds two .modal-backdrop divs. So that was not a solution (and BTW I like to have a very visible close/cancel option).

It makes sense to close the modal in the same way it was opened, so either I change the way to open the modal, but I then tried if I could mimick "closing" by button by just triggering a click event on the close button(s) and then hoping it works as expected:

So do something like

    $("button[data-dismiss=modal]:visible").eq(0).click();

So to explain this:

  • first select all buttons that allow to close my modal: $("button[data-dismiss=modal]:visible") (select the visible just in case you have multiple modals defined in your html)
  • the added .eq(0) selects the first element out of that set
  • and then trigger the click event

Unfortunately this does not work either. During my numerous tests for a while it seemed to work, but I was just confused. Doh. This doesn't work either. I am guessing I just do not quite understand how bootstrap actually handles this, and (while I probably should) not really inclined to dive into their code to see what is wrong.

So I just went with:

:plain
  toastr.success('#{@success_message}');
  $(".modal").modal('hide');
  $(".modal-backdrop").remove();

Just close the modal, and if there are any pending/lingering .modal-backdrop divs, just remove those too.

More ...
Technology optimization ruby on rails
[RAILS] optimizing a slow request

We identified a single slow request in our moderation module: retrieving the json containing the entries to be moderated, feeding our react app.

So in short our datamodel looks as follows:

Class Topic 
  have_many :entries
  have_many :dynamic_attributes

Class Entry
  belongs_to :topic
  has_many :entry_values

Class EntryValue
  belongs_to :entry
  belongs_to :dynamic_attribute 

So in short: topics have a set of (dynamic) attributes that can be entered. An entry_value is the value entered for a dynamic_attribute and those are grouped in a complete entry.

In the moderation, our moderators verify that an entry (collection of entry-values) is appropriate, and have to option to edit or add missing information.

So in our controller we do something like

@entries = @q.result(distinct: true).page params[:page]

We are using ransack to filter/search on our entries. So we know we need entry-values, and possibly their dynamic attributes when building the json, so what is the best approach here? Use eager_load or includes? So what better way to decide than actually test this? So I temporarily changed the controller code as follows:

        preload_option = params[:pre].try(:to_i)
        if preload_option == 0
          @entries = @q.result(distinct: true).page params[:page]
        elsif preload_option == 1
          @entries = @q.result(distinct: true).eager_load(:entry_values).page params[:page]
        else
          @entries = @q.result(distinct: true).includes(:entry_values => :dynamic_attribute).page params[:page]
        end

Note:

  • I can only eager_load one level deep
  • with includes I can immediately fetch all dynamic attributes for the entry-values.

But which will prove to be more beneficial?

So then I ran a small benchmark script:

require 'benchmark'
require 'rest-client'

n = 10
Benchmark.bm do |x|
  x.report("normal ") do
    n.times { RestClient.get("http://admin.lvh.me:3013/moderation/projects/11/entries.json", {}) }
  end
  x.report("eager  ") do
    n.times { RestClient.get("http://admin.lvh.me:3013/moderation/projects/11/entries.json?pre=1", {}) }
  end
  x.report("include") do
    n.times { RestClient.get("http://admin.lvh.me:3013/moderation/projects/11/entries.json?pre=2", {}) }
  end
end

(note: for testing purposes I also disabled the need to authenticate, so I could easily fetch the jsons and time and compare)

This first run gave me the following results:

    normal   0.016362   0.005723   0.022085 ( 35.440171)
    eager    0.010036   0.004336   0.014372 ( 28.632490)
    include  0.012550   0.004061   0.016611 ( 29.173778) 

Ok. Not the kind of improvement I had hoped. Also nice to notice that eager_load in this case is more efficient than using the includes (which seemed a little counter-intuitive maybe).

I had recently changed a small part of the code, because in the moderation we also wanted to be able to edit fields that were not entered, and before we only had to retrieve entered values (:entry_values) so I presume that maybe there I fucked up the performance. Before we called entry.valid_entry_values which looked like

  def valid_entry_values
    entry_values.sorted.select do |ev|
      da = ev.dynamic_attribute
      da.attribute_type != 'item' || (da.attribute_type == 'item' && !ev.item_content_type.nil?)
    end
  end

and I replaced it with the following, adding empty entry-values to be filled in:


  def entry_values_with_empty
    result = []
    self.topic.dynamic_attributes.each do |da|
      ee = entry_values.find_by(dynamic_attribute_id: da.id)
      if ee.nil? || (da.attribute_type == 'item' && ee.item_content_type.nil?)
        ee = entry_values.build(dynamic_attribute: da)
      end
      result << ee
    end
    # check if we have entry-values not yet in the list
    # (e.g. from another topic when the entry was moved, and add those too)
    self.entry_values.each do |entry_value|
      if result.select{|ee| ee.id == entry_value.id}.count == 0
        result << entry_value
      end
    end
   
    result
  end

So what happens if we switch back to the old valid_entry_values : how does that change performance?

I ran my small benchmark script again, and got the following results:

    normal   0.015264   0.005280   0.020544 ( 33.283069)
    eager    0.009901   0.004359   0.014260 ( 17.145350)
    include  0.013153   0.004032   0.017185 ( 17.621856)

Wow! Now the eager_load or includes really seem to pay off. Also: almost the same speed improvement. Ok.

So if we check the entry_values_with_empty more closely, the implementation is somewhat naive: for each dynamic-attribute it will attempt to find the corresponding entry-value, except ... we use a query each time for each dynamic attribute, for each entry ... Mmmmmm. Let's see if we can improve this:

  def entry_values_with_empty
    result = []
    self.topic.dynamic_attributes.each do |da|
      ee = entry_values.detect{|ev| ev.dynamic_attribute_id == da.id}
      if ee.nil? || (da.attribute_type == 'item' && ee.item_content_type.nil?)
        ee = entry_values.build(dynamic_attribute: da)
      end
      result << ee
    end
    # check if we have entry-values not yet in the list
    # (e.g. from another topic when the entry was moved, and add those too)
    self.entry_values.each do |entry_value|
      if result.select{|ee| ee.id == entry_value.id}.count == 0
        result << entry_value
      end
    end

    result
  end

Notice: we only changed one line, replacing the find_by with a detect. This will, instead of launching a new query, iterate over the already retrieved array of entry_values. But does this make any difference?

Launching my small test script now returns the following:

    normal   0.016051   0.005418   0.021469 ( 16.448649)
    eager    0.009454   0.003858   0.013312 ( 22.479142)
    include  0.012419   0.003872   0.016291 ( 14.236868)

NICE! **fireworks** Not what I expected to see at all. A little baffled that the normal case is improved that much, and that the eager_load does not improve it (on the contrary). We have now found our optimal combination: improving the entry_values_with_empty and adding the includes will give the best performance.

Is this what you would have expected? Bottomline remains: it helps to measure (in Dutch we say: meten is weten which rhymes)

More ...
Technology ruby on rails rspec
[RSPEC] Cleaning up orphaned attachments when running specs

So when running the specs we also create a lot of fake attachments, but they are never cleaned up. Which is probably obvious, because we never actually destroy the models (containing the attachments), but truncate the database or rollback the transactions.

So I tried to find a way to 1) automatically/more easily clean up those dummy attachments, and 2) make sure it works when using parallel:specs. And over my different projects, where in some I use different gems to manage my attachments.

In one project, I am using paperclip and there I took the following approach. In the initializer config/initializers/paperclip.rb I wrote

  Paperclip::UriAdapter.register
  if Rails.env.production?
    Paperclip::Attachment.default_options.merge!(
      hash_secret: ENV.fetch('SECRET_KEY_BASE'),
      s3_protocol: :https,
      url: ':s3_domain_url',
      path: "/:class/:attachment/:id/:style/:hash.:extension",
      storage: :s3,
      s3_credentials: { .. }
    )
  elsif Rails.env.development?
    Paperclip::Attachment.default_options.merge!(
      url: "/system/:class/:attachment/:id/:style/:hash.:extension",
      hash_secret: Rails.application.credentials.secret_key_base
    )
  elsif Rails.env.test? || Rails.env.cucumber?
    Paperclip::Attachment.default_options.merge!(
      url: "/spec_#{ENV['TEST_ENV_NUMBER']}/:class/:attachment/:id/:style/:hash.:extension",
      hash_secret: Rails.application.credentials.secret_key_base
    )
  end

and then in rspec rails_helper.rb I can add the following piece of code

  config.after(:suite) do
    FileUtils.remove_dir(File.join(Rails.root, 'public', "spec_#{ENV['TEST_ENV_NUMBER']}"), true)
  end

In another projects I am using carrier_wave and there it is a little more difficult, but it amounts to the same approach. In CarrierWave we create different uploaders, and each have their own configuration. In my project I first iterate over all uploaders in my own code-base, and explicitly require one uploader from our own shared gem (between different projects). So we add an initializer config/carrierwave_clean_spec_attachments.rb (or whatever name you prefer) to override the path when in test mode:

if Rails.env.test? || Rails.env.cucumber?
  Dir["#{Rails.root}/app/uploaders/*.rb"].each { |file| require file }
  require 'document_uploader'

  CarrierWave::Uploader::Base.descendants.each do |klass|
    next if klass.anonymous?
    klass.class_eval do
      def cache_dir
        "#{Rails.root}/spec/support/uploads_#{ENV['TEST_ENV_NUMBER']}/tmp"
      end

      def store_dir
        "#{Rails.root}/spec/support/uploads_#{ENV['TEST_ENV_NUMBER']}/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
      end
    end
  end
end

and then in my rails_helper.rb I can then add the following statement:

config.after(:suite) do
  FileUtils.rm_rf(Dir["#{Rails.root}/spec/support/uploads_#{ENV['TEST_ENV_NUMBER']}"])
end

How do you do this? Do you use another gem for storage/attachments and how do you solve it? E.g. when using ActiveStorage ?

More ...

Create

show some portfolio and link to