Just a quick note about the downtime that we experienced this week …
We ran into a strange interaction between Ubuntu 07.04, RubyGems and Rails. I kicked off this chain-reaction train wreck by updating my code to use the new Rails 2.0 feature for gem dependency checking in environments.rb:
Rails::Initializer.run do |config| # Gem Dependencies config.gem 'icalendar', :version => '>= 1.0.2' #any version >= to 2.0.1 config.gem 'mislav-will_paginate', :lib => 'will_paginate', :version => '>=2.3.6' config.gem 'RedCloth', :version => '>= 4.1.1' config.gem 'timcharper-CalendarDateSelect', :lib => 'calendar_date_select' , :version => '>=1.11.1' end
This allows me to more easily verify that I have the correct gems and gem versions on my production box. And it works. At least it worked fine on my MacBook in development. So I went ahead and deployed it into production with my capistrano script.
And everything came crashing down.
Well, not really crashing down, but certainly to a complete halt. That is, Rails wouldn’t start because RubyGems was complaining about “too many arguments” in it’s load routines. A few minutes of debugging and comparison with my MacBook showed that I was several versions back on RubyGems in production (version 1.2 instead of 1.3.1). After much gnashing of teeth and RTFG’s I discovered the following nugget of wisdom (#293 gem dependencies broken in 2.1.0 – Ruby on Rails – rails):
“What I discovered running an upgrade from ubuntu 7.04 to 8.04 was that there was two versions of rubygems installed somehow. Rails ended up choosing and older version of Gem::DependencyList. This was causing the wrong number of arguments error for me.
To solve this, I removed the old version of rubygems from apt-get, reinstalled rubygems from source, and updated my path. Once I did this, the newer version of DependencyList was loaded, and I didn’t get any more errors.”
Of course, this was much easier said than down. In the end, it required uninstalling the the apt-get gems package, reinstalling gems from source and then eliminating the rogue gems library directory with prejudice. However, it certainly fixed the problem and forced me to look harder at maintaining the production environment.
For any of you following at home, the website is currently running revision 235 (“re-education through labor”) which boasts some slight performance improvements, a few bug fixes for the entry forms and some new functionality for widgets (both dashboard and blogs).