Baserock/ guides/ import-tool/ Using the Baserock Import tool for RubyGems

Using Baserock

Community

Background

Using the Baserock Import tool for RubyGems

NOTE: Details in this tutorial are out of date, so it will be difficult to follow without also asking questions on #baserock on IRC. If you just want to build a straightforward gem as a test of the tool, see the quickstart guide, which deals with the gem 'Sinatra'. This tutorial is still here because it explains the tool in more detail, and solutions here may be useful if you encounter tricky errors with other gems.

The Baserock Import tool exists to make your life easier if you want to integrate things in Baserock that are already packaged in a 'foreign' packaging system. Please read the import tool quickstart page first, for an overview of the tool.

This tutorial shows how you can use the tool to import Ruby on Rails, which is a popular web framework. Like most Ruby libraries, there is packaging for Rails in the RubyGems packaging system. The Baserock Import tool is extensible and supports several packaging systems, driven by the same interface. So hopefully this tutorial is useful for all users of the tool even if you are looking at importing something other than Ruby code.

As well as this tutorial, there is documentation in the 'README' files in the Baserock Import Git repository. Please read the main README first, to understand what the tool does and what it does not do.

The Baserock Import tool is available in Baserock 'devel' systems. See the using baserock page for how to set up a 'devel' system. Note that the downloadable images we provide are 'build' systems, not 'devel' systems: they don't contain the import tool, but they can be used to build a system that does.

NOTE: This document is very new, as is the import tool, and is likely to change. Please report any lies, inconsistencies, and pain points (ideally with patches to fix them!) to the mailing list. For swifter responses, ask in #baserock on irc.freenode.net.

Chapter 0: Preliminaries

  1. Installing and updating:

See the import tool quickstart page.

The old version of the RubyGems importer only accepted one argument, so if you are using the old version you may hit ERROR: Please pass the name of a RubyGem on the commandline. errors. Updating should fix the problem.

Chapter 1: Doing an import

We need to start by clearing a space among yesterday's dishes in which to work. Clone the definitions repository with the following command:

mkdir /src/baserock-import
cd /src/baserock-import
git clone git://git.baserock.org/baserock/baserock/definitions

You could replace 'me' with your actual name, or your initials. Inside that 'baserock' folder is another 'baserock' folder, inside which is a clone of the Baserock reference definitions repository. Let's look inside the 'strata' folder:

$ ls definitions/strata/

Lots of strata exist, including one for 'ruby', but there's no 'rails' stratum! We should amend that. Run the baserock-import tool as follows (from now on, assume that anything following a $ sign is a command for you to run, and other text in codeblocks is the expected output of the command).

$ baserock-import rubygems rails --definitions-dir=definitions --log=import.log
12/01/14 15:54:20 UTC: Import of rubygems rails started
Not updating existing Git checkouts or existing definitions
rails: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/rails/rails
rails master: using checkouts/ruby-gems_rails ref master (commit 07786c5e75a7b0afdf318063510af6b475e3e04c)
rails master: calling rubygems.to_chunk to generate chunk morph
activesupport: calling rubygems.to_lorry to generate lorry
activesupport 4.2.0.beta4: using checkouts/ruby-gems_rails ref v4.2.0.beta4 (commit 4daebedcc41e35079c47d5f130f5c7ad12db8bbb)
activesupport 4.2.0.beta4: calling rubygems.to_chunk to generate chunk morph
minitest: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/seattlerb/minitest
Could not find ref for minitest-5.4.3, required by: activesupport-4.2.0.beta4.
thread_safe: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/headius/thread_safe
thread_safe 0.3.4: using checkouts/ruby-gems_thread_safe ref v0.3.4 (commit 21ea387dce966f3a6a38752ab6e199bb9a2d710b)
json: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/flori/json
json 1.8.1: using checkouts/ruby-gems_json ref v1.8.1 (commit 92a96dea2b24b9c68856004d69491f46aedd0925)
tzinfo: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/tzinfo/tzinfo
tzinfo 1.2.2: using checkouts/ruby-gems_tzinfo ref v1.2.2 (commit f1c8830e584d08689ed2ac11c54c457dc8fd10d6)
i18n: calling rubygems.to_lorry to generate lorry
Lorrying http://github.com/svenfuchs/i18n
i18n 0.7.0.beta1: using checkouts/ruby-gems_i18n ref v0.7.0.beta1 (commit 4612e6ebf357fba5f90f879ff8b753f8cf8415ff)
i18n 0.7.0.beta1: calling rubygems.to_chunk to generate chunk morph
railties: calling rubygems.to_lorry to generate lorry
railties 4.2.0.beta4: using checkouts/ruby-gems_rails ref v4.2.0.beta4 (commit 4daebedcc41e35079c47d5f130f5c7ad12db8bbb)
railties 4.2.0.beta4: calling rubygems.to_chunk to generate chunk morph
thor: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/wycats/thor
thor 0.19.1: using checkouts/ruby-gems_thor ref v0.19.1 (commit c2da2f5b26f976510965adda01e4893d1eae7f7d)
actionpack: calling rubygems.to_lorry to generate lorry
actionpack 4.2.0.beta4: using checkouts/ruby-gems_rails ref v4.2.0.beta4 (commit 4daebedcc41e35079c47d5f130f5c7ad12db8bbb)
actionpack 4.2.0.beta4: calling rubygems.to_chunk to generate chunk morph
rack-test: calling rubygems.to_lorry to generate lorry
Lorrying http://github.com/brynary/rack-test
rack-test 0.6.2: using checkouts/ruby-gems_rack-test ref v0.6.2 (commit 5080d1e99de825bdd3b2b7bba34e5ac61cedd876)
rack: calling rubygems.to_lorry to generate lorry
Lorrying http://github.com/rack/rack
rack 1.5.2: using checkouts/ruby-gems_rack ref 1.5.2 (commit ac590d055c936bb9a618e955a690dc836c625211)
rack 1.6.0.beta: using checkouts/ruby-gems_rack ref 1.6.0.beta (commit e4e4c397e89c026f9c23500cf7fc14ccdb756010)
rack 1.6.0.beta: calling rubygems.to_chunk to generate chunk morph
rails-dom-testing: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/kaspth/rails-dom-testing
rails-dom-testing 1.0.4: using checkouts/ruby-gems_rails-dom-testing ref v1.0.4 (commit 6bbed9a153c24075eb9662469e5e080414e61c4f)
rails-dom-testing 1.0.4: calling rubygems.to_chunk to generate chunk morph
rails-deprecated_sanitizer: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/rails/rails-deprecated_sanitizer
rails-deprecated_sanitizer 1.0.3: using checkouts/ruby-gems_rails-deprecated_sanitizer ref v1.0.3 (commit 1a806136265ba6aba3ccdeaad0d686c3e1607bd7)
rails-deprecated_sanitizer 1.0.3: calling rubygems.to_chunk to generate chunk morph
nokogiri: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/sparklemotion/nokogiri
nokogiri 1.6.4.1: using checkouts/ruby-gems_nokogiri ref v1.6.4.1 (commit 594bf2f79dbd899d8cb77ad813bff734139a9cb2)
nokogiri 1.6.4.1: calling rubygems.to_chunk to generate chunk morph
rubygems.to_chunk failed with code 1: Did not find any files matching nokogiri.gemspec within /src/baserock-import/checkouts/ruby-gems_nokogiri.
rails-html-sanitizer: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/rafaelfranca/rails-html-sanitizer
rails-html-sanitizer 1.0.1: using checkouts/ruby-gems_rails-html-sanitizer ref v1.0.1 (commit 6b14d6a9e11b58253337df95f2b699665cf8b463)
rails-html-sanitizer 1.0.1: calling rubygems.to_chunk to generate chunk morph
loofah: calling rubygems.to_lorry to generate lorry
Lorrying http://github.com/flavorjones/loofah
loofah 2.0.1: using checkouts/ruby-gems_loofah ref v2.0.1 (commit 1e1637cafd32e98957c04550bfb8662b67e3fd6d)
loofah 2.0.1: calling rubygems.to_chunk to generate chunk morph
rubygems.to_chunk failed with code 1: Source in /src/baserock-import/checkouts/ruby-gems_loofah produces loofah-1.2.1.20130718080038, but the expected version was 2.0.1.
actionmailer: calling rubygems.to_lorry to generate lorry
actionmailer 4.2.0.beta4: using checkouts/ruby-gems_rails ref v4.2.0.beta4 (commit 4daebedcc41e35079c47d5f130f5c7ad12db8bbb)
actionmailer 4.2.0.beta4: calling rubygems.to_chunk to generate chunk morph
mail: calling rubygems.to_lorry to generate lorry
Lorrying http://github.com/mikel/mail/
Could not find ref for mail-2.6.3, required by: actionmailer-4.2.0.beta4.
activejob: calling rubygems.to_lorry to generate lorry
activejob 4.2.0.beta4: using checkouts/ruby-gems_rails ref v4.2.0.beta4 (commit 4daebedcc41e35079c47d5f130f5c7ad12db8bbb)
activejob 4.2.0.beta4: calling rubygems.to_chunk to generate chunk morph
globalid: calling rubygems.to_lorry to generate lorry
rubygems.to_lorry failed with code 1: ERROR: Gem metadata for 'globalid' does not point to its source code repository.
sprockets-rails: calling rubygems.to_lorry to generate lorry
Lorrying http://github.com/rails/sprockets-rails
sprockets-rails 3.0.0.beta1: using checkouts/ruby-gems_sprockets-rails ref v3.0.0.beta1 (commit 40cd1d4c1966e8920242a8786f834e425ecacbbb)
sprockets-rails 3.0.0.beta1: calling rubygems.to_chunk to generate chunk morph
activesupport 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
i18n 0.6.11: using checkouts/ruby-gems_i18n ref v0.6.11 (commit 9eadb6bb71cf13554514d2d6f22a0884891ebee0)
actionpack 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
actionview: calling rubygems.to_lorry to generate lorry
actionview 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
erubis: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/kwatch/erubis
Could not find ref for erubis-2.7.0, required by: actionview-4.1.8.
builder: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/jimweirich/builder
Could not find ref for builder-3.2.2, required by: actionview-4.1.8.
sprockets: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/sstephenson/sprockets
sprockets 2.12.3: using checkouts/ruby-gems_sprockets ref v2.12.3 (commit 55a53fcdd352bc906ec04945245920aedf105a02)
tilt: calling rubygems.to_lorry to generate lorry
Lorrying http://github.com/rtomayko/tilt
tilt 1.4.1: using checkouts/ruby-gems_tilt ref 1.4.1 (commit d67b24d14c84b6a877fbd2b5466028ecba6981c3)
multi_json: calling rubygems.to_lorry to generate lorry
Lorrying http://github.com/intridea/multi_json
multi_json 1.10.1: using checkouts/ruby-gems_multi_json ref v1.10.1 (commit 0c600e2a3274f4383fb02368635af9bab4cf893b)
hike: calling rubygems.to_lorry to generate lorry
Lorrying http://github.com/sstephenson/hike
hike 1.2.3: using checkouts/ruby-gems_hike ref v1.2.3 (commit f313ea1fd44caf2cae004fd45c67017b744bbb7a)
bundler: calling rubygems.to_lorry to generate lorry
Lorrying http://github.com/carlhuda/bundler/
bundler 1.7.6: using checkouts/ruby-gems_bundler ref v1.7.6 (commit 8cbeb529ee12db21c861748fc4a858fd7d27bacc)
bundler 1.7.6: calling rubygems.to_chunk to generate chunk morph
activerecord: calling rubygems.to_lorry to generate lorry
activerecord 4.2.0.beta4: using checkouts/ruby-gems_rails ref v4.2.0.beta4 (commit 4daebedcc41e35079c47d5f130f5c7ad12db8bbb)
activerecord 4.2.0.beta4: calling rubygems.to_chunk to generate chunk morph
arel: calling rubygems.to_lorry to generate lorry
Lorrying http://github.com/rails/arel
arel 6.0.0.beta2: using checkouts/ruby-gems_arel ref v6.0.0.beta2 (commit 8ac47c4580e55a41c5a159fe7fe74c732214ccec)
arel 6.0.0.beta2: calling rubygems.to_chunk to generate chunk morph
activemodel: calling rubygems.to_lorry to generate lorry
activemodel 4.2.0.beta4: using checkouts/ruby-gems_rails ref v4.2.0.beta4 (commit 4daebedcc41e35079c47d5f130f5c7ad12db8bbb)
activemodel 4.2.0.beta4: calling rubygems.to_chunk to generate chunk morph
actionview 4.2.0.beta4: using checkouts/ruby-gems_rails ref v4.2.0.beta4 (commit 4daebedcc41e35079c47d5f130f5c7ad12db8bbb)
actionview 4.2.0.beta4: calling rubygems.to_chunk to generate chunk morph

Errors encountered, not generating a stratum morphology.
See the README files for guidance.
12/01/14 15:58:34 UTC: Import of rubygems rails ended (took 253 seconds)

You might notice that your results differ a bit from what's listed here. It's a safe bet that there were some errors encountered, though. Look at the 5th line of the output:

rails master: using checkouts/ruby-gems_rails ref master (commit b0f44c9ff0be84ebc42760029adeeb9dd954655f)

What we tried to do here is integrate the latest, unstable version of Ruby on Rails. How far did it get? Try this:

$ ls definitions/strata/rails/*.morph

We do now have a directory for the 'rails' chunks. Perhaps some chunk morphs were generated too! But usually, importing the latest unstable version of a project will fail. For examplem when I ran the tool I got multiple versions of some things (which ended up causing a conflict further down):

definitions/strata/rails/actionpack-4.1.8.morph
definitions/strata/rails/actionpack-4.2.0.beta4.morph
definitions/strata/rails/actionview-4.1.8.morph
definitions/strata/rails/actionview-4.2.0.beta4.morph

This sort of thing is common when you try to import an unstable version of a RubyGem: some .gemspec files will have been updated to depend on development versions of components, others will still point at the last stable release.

Let's avoid this complexity by giving a specific version of Rails to import. At the time of writing, the latest stable is 4.1.8. Before running the tool again you should delete the morphologies that were generated for 'master' of Rails. The tool will work even if you don't do that, but it saves you from having to later work out which chunks are for Rails 'master' and which are for Rails 4.1.8 and which are for both.

$ rm definitions/strata/rails/*
$ baserock-import rubygems rails 4.1.8 \
    --definitions-dir=definitions --log=import.log
12/01/14 16:08:57 UTC: Import of rubygems rails started
Not updating existing Git checkouts or existing definitions
rails 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
rails 4.1.8: calling rubygems.to_chunk to generate chunk morph
rails 4.1.8: calling rubygems.find_deps to calculate dependencies
activesupport 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
activesupport 4.1.8: calling rubygems.to_chunk to generate chunk morph
activesupport 4.1.8: calling rubygems.find_deps to calculate dependencies
Could not find ref for minitest-5.4.3, required by: activesupport-4.1.8.
thread_safe 0.3.4: using checkouts/ruby-gems_thread_safe ref v0.3.4 (commit 21ea387dce966f3a6a38752ab6e199bb9a2d710b)
thread_safe 0.3.4: calling rubygems.to_chunk to generate chunk morph
thread_safe 0.3.4: calling rubygems.find_deps to calculate dependencies
json 1.8.1: using checkouts/ruby-gems_json ref v1.8.1 (commit 92a96dea2b24b9c68856004d69491f46aedd0925)
json 1.8.1: calling rubygems.to_chunk to generate chunk morph
json 1.8.1: calling rubygems.find_deps to calculate dependencies
tzinfo 1.2.2: using checkouts/ruby-gems_tzinfo ref v1.2.2 (commit f1c8830e584d08689ed2ac11c54c457dc8fd10d6)
tzinfo 1.2.2: calling rubygems.to_chunk to generate chunk morph
tzinfo 1.2.2: calling rubygems.find_deps to calculate dependencies
i18n 0.6.11: using checkouts/ruby-gems_i18n ref v0.6.11 (commit 9eadb6bb71cf13554514d2d6f22a0884891ebee0)
i18n 0.6.11: calling rubygems.to_chunk to generate chunk morph
i18n 0.6.11: calling rubygems.find_deps to calculate dependencies
railties 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
railties 4.1.8: calling rubygems.to_chunk to generate chunk morph
railties 4.1.8: calling rubygems.find_deps to calculate dependencies
thor 0.19.1: using checkouts/ruby-gems_thor ref v0.19.1 (commit c2da2f5b26f976510965adda01e4893d1eae7f7d)
thor 0.19.1: calling rubygems.to_chunk to generate chunk morph
thor 0.19.1: calling rubygems.find_deps to calculate dependencies
actionview 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
actionview 4.1.8: calling rubygems.to_chunk to generate chunk morph
actionview 4.1.8: calling rubygems.find_deps to calculate dependencies
Could not find ref for erubis-2.7.0, required by: actionview-4.1.8.
Could not find ref for builder-3.2.2, required by: actionview-4.1.8.
actionmailer 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
actionmailer 4.1.8: calling rubygems.to_chunk to generate chunk morph
actionmailer 4.1.8: calling rubygems.find_deps to calculate dependencies
Could not find ref for mail-2.6.3, required by: actionmailer-4.1.8.
sprockets-rails 2.2.2: using checkouts/ruby-gems_sprockets-rails ref v2.2.2 (commit b54af68acbdfaf3bb2bd5940ef4bc2d5a41a38f4)
sprockets-rails 2.2.2: calling rubygems.to_chunk to generate chunk morph
sprockets-rails 2.2.2: calling rubygems.find_deps to calculate dependencies
sprockets 2.12.3: using checkouts/ruby-gems_sprockets ref v2.12.3 (commit 55a53fcdd352bc906ec04945245920aedf105a02)
sprockets 2.12.3: calling rubygems.to_chunk to generate chunk morph
sprockets 2.12.3: calling rubygems.find_deps to calculate dependencies
tilt 1.4.1: using checkouts/ruby-gems_tilt ref 1.4.1 (commit d67b24d14c84b6a877fbd2b5466028ecba6981c3)
tilt 1.4.1: calling rubygems.to_chunk to generate chunk morph
tilt 1.4.1: calling rubygems.find_deps to calculate dependencies
rack 1.5.2: using checkouts/ruby-gems_rack ref 1.5.2 (commit ac590d055c936bb9a618e955a690dc836c625211)
rack 1.5.2: calling rubygems.to_chunk to generate chunk morph
rack 1.5.2: calling rubygems.find_deps to calculate dependencies
multi_json 1.10.1: using checkouts/ruby-gems_multi_json ref v1.10.1 (commit 0c600e2a3274f4383fb02368635af9bab4cf893b)
multi_json 1.10.1: calling rubygems.to_chunk to generate chunk morph
multi_json 1.10.1: calling rubygems.find_deps to calculate dependencies
hike 1.2.3: using checkouts/ruby-gems_hike ref v1.2.3 (commit f313ea1fd44caf2cae004fd45c67017b744bbb7a)
hike 1.2.3: calling rubygems.to_chunk to generate chunk morph
hike 1.2.3: calling rubygems.find_deps to calculate dependencies
activerecord 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
activerecord 4.1.8: calling rubygems.to_chunk to generate chunk morph
activerecord 4.1.8: calling rubygems.find_deps to calculate dependencies
Could not find ref for arel-5.0.1.20140414130214, required by: activerecord-4.1.8.
activemodel 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
activemodel 4.1.8: calling rubygems.to_chunk to generate chunk morph
activemodel 4.1.8: calling rubygems.find_deps to calculate dependencies
actionpack 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
actionpack 4.1.8: calling rubygems.to_chunk to generate chunk morph
actionpack 4.1.8: calling rubygems.find_deps to calculate dependencies
rack-test 0.6.2: using checkouts/ruby-gems_rack-test ref v0.6.2 (commit 5080d1e99de825bdd3b2b7bba34e5ac61cedd876)
rack-test 0.6.2: calling rubygems.to_chunk to generate chunk morph
rack-test 0.6.2: calling rubygems.find_deps to calculate dependencies

Errors encountered, not generating a stratum morphology.
See the README files for guidance.
12/01/14 16:10:44 UTC: Import of rubygems rails ended (took 106 seconds)

That was quicker than last time! The tool keeps quite a lot of state around between runs, actually. That way you can iterate quickly when doing the manual bits of the import. By default all the stuff goes in subdirectories of the current working directory (although there are configuration flags to put it elsewhere):

$ ls
definitions                checkouts                  lorries
lorry-working-dir          rubygems_api_cache.sqlite

One thing to note is that once baserock-import has lorried something (that is, imported or cloned its source code into a local Git repo inside the lorry-working-dir subdirectory), baserock-import won't update the clone when you run it again. This is because running git remote update inside dozens of Git repositories is not fast! If you do want it to update the repos as it goes (maybe you started an import a while ago, forgot about it, and in the meantime there's been a new release) you can use the --update-existing flag. This will cause it to overwrite any existing .morph files that were generated, too.

If you look inside definitions you should see a reasonable set of chunk morphs:

$ ls definitions/strata/rails/*.morph
definitions/strata/rails/actionmailer-4.1.8.morph
definitions/strata/rails/actionpack-4.1.8.morph
definitions/strata/rails/actionview-4.1.8.morph
definitions/strata/rails/activemodel-4.1.8.morph
definitions/strata/rails/activerecord-4.1.8.morph
definitions/strata/rails/activesupport-4.1.8.morph
definitions/strata/rails/hike-1.2.3.morph
definitions/strata/rails/i18n-0.6.11.morph
definitions/strata/rails/json-1.8.1.morph
definitions/strata/rails/multi_json-1.10.1.morph
definitions/strata/rails/rack-1.5.2.morph
definitions/strata/rails/rack-test-0.6.2.morph
definitions/strata/rails/rails-4.1.8.morph
definitions/strata/rails/railties-4.1.8.morph
definitions/strata/rails/sprockets-2.12.3.morph
definitions/strata/rails/sprockets-rails-2.2.2.morph
definitions/strata/rails/thor-0.19.1.morph
definitions/strata/rails/thread_safe-0.3.4.morph
definitions/strata/rails/tilt-1.4.1.morph
definitions/strata/rails/tzinfo-1.2.2.morph

Good news! Quite a few things seem to have imported automatically. In the next section I'll go through the various errors we encountered and how to fix them.

Chapter 2: Lorrying and missing refs

Here are the error messages extracted from the output we got above for importing Rails 4.1.8.

Could not find ref for minitest-5.4.3, required by: activesupport-4.1.8.
Could not find ref for mail-2.6.3, required by: actionmailer-4.1.8.
Could not find ref for arel-5.0.1.20140414130214, required by: activerecord-4.1.8.
Could not find ref for builder-3.2.2, required by: activemodel-4.1.8.
Could not find ref for erubis-2.7.0, required by: actionview-4.1.8.

There's a pattern here. Let's start with the 'minitest' repo. The Baserock Import tool checks out the source code of each project in the checkouts directory (unless configured otherwise), and the directories inside are named corresponding to the name in the .lorry file created for that project.

The .lorry file created for that project? Let's take a step back. Lorry is a tool which pulls source from various version control systems into Git. Baserock uses Git for all source control so Lorry comes in pretty handy. Another thing you probably already knew about Baserock is we like to mirror all of the source used in our systems, rather than depending on the generosity and reliability of the upstream projects. A Trove system such as http://git.baserock.org/ looks after that, and each Trove has a set of .lorry files describing the various upstream repositories that it should pull source from.

The baserock-import tool has created a .lorry file for you too!

$ ls lorries/
ruby-gems.lorry

If you look in 'ruby-gems.lorry' you'll see entries for each of the repos it found for the RubyGems we're importing. They've been put into a "ruby-gems/" namespace.

The code in the tool can create a separate .lorry for each component, either. Policy on this is up to the owner of the Trove; for git.baserock.org we seem to put all packages from one 'family' into the same .lorry file right now. Currently you'll need to edit the data/rubygems.yaml file in the baserockimport/ source tree to change this behaviour.

If you're planning on using a Trove to mirror all these sources, you 'll want to add the .lorry files to go in the local-config/lorries.git repo for that Trove. The best way to do this is to rerun the tool, and point it at a checkout of the local-config/lorries.git repo from the Trove. If I wanted to push the Rails lorries to a branch on git.baserock.org, I'd do this:

$ rm -R lorries
$ git clone git://git.baserock.org/baserock/local-config/lorries.git
$ cd lorries
$ git checkout -b baserock/me/rails-lorries
$ cd ..
$ baserock-import rubygems rails 4.1.8 \
    --definitions-dir=definitions \
    --lorries-dir=lorries/open-source-lorries \
    --log=import.log
...
11/20/14 18:23:43 UTC: Import of rubygems rails ended (took 6 seconds)

You'll find lorries/open-source-lorries/ruby-gems.lorry now contains all the lorry entries for Rails, plus anything that was already there.

There are still some errors though! Let's get back to the problem at hand. Inside the checkouts/ directory you'll find the repo for 'minitest': it's called ruby-gems_minitest (because the name is based on its name in the .lorry file).

$ cd checkouts/ruby-gems_minitest/
$ git tag

It should be fairly clear now why the tool couldn't work out which commit corresponded to version 5.4.3 of 'minitest': releases are not tagged in this repository. Reading git log makes me suspect that the repo listed for 'minitest' in the rubygems.org metadata is actually a mirror of a Perforce repository, which may explain why there are no tags.

This is a bit of a pain, but at the time of writing the top commit of 'minitest' was as follows:

$ git log -p | head -n 15
commit 32dad2e92a4953a052ece1ed689ee62a17b3abd8
Author: Ryan Davis <ryand@zenspider.com>
Date:   Tue Nov 11 15:44:00 2014 -0800

    prepped for release
    [git-p4: depot-paths = "//src/minitest/dev/": change = 9827]

diff --git a/History.txt b/History.txt
index a0e0e10..a10600c 100644
--- a/History.txt
+++ b/History.txt
@@ -1,3 +1,10 @@
+=== 5.4.3 / 2014-11-11
+
+* 2 bug fixes:

It's safe to assume then that commit 32dad2e92a4953a052ece1ed689ee62a17b3abd8 corresponds to v5.4.3. In order to communicate this to the tool, just create a tag yourself named 'v5.4.3' pointing to this commit:

$ git tag v5.4.3 -a 32dad2e92a4953a052ece1ed689ee62a17b3abd8 \
    -m "Release tag created for baserock-import"

I did this for the other 4 repos, too. I created the following tags:

$ cd ../ruby-gems_arel
$ git tag v5.0.1.20140414130214 -a v5.0.1 \
    -m "Release tag created for baserock-import"
$ cd ../ruby-gems_builder
$ git tag v3.2.2 -a 38c8f83d27670fee9ce057a2e51759b8f62a0b38 \
    -m "Release tag created for baserock-import"
$ cd ../ruby-gems_erubis
$ git tag v2.7.0 -a 14d3eab57fbc361312c8f3af350cbf9a5bafce17 \
    -m "Release tag created for baserock-import"
$ cd ../ruby-gems_mail
$ git tag v2.6.3 -a ab034a4d3d6ef1149cf6989437ec043561a1217e \
    -m "Release tag created for baserock-import"

The 'builder' and 'mail' repos seemed to have tags for older releases, but not the latest ones. These tags may already exist upstream by the time you read this! 'Arel' seems to be a dependency on a specific patched version. I took a punt on the fact that 5.0.1 would work just as well and created a 'v5.0.1.20140414130214' tag pointed at the same commit as the 5.0.1 tag.

$ cd ../..
$ baserock-import rubygems rails 4.1.8 \
    --definitions-dir=definitions \
    --lorries-dir=lorries/open-source-lorries \
    --log=./import.log
12/01/14 16:28:35 UTC: Import of rubygems rails started
Not updating existing Git checkouts or existing definitions
rails: calling rubygems.to_lorry to generate lorry
rails 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
activesupport: calling rubygems.to_lorry to generate lorry
activesupport 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
minitest: calling rubygems.to_lorry to generate lorry
minitest 5.4.3: using checkouts/ruby-gems_minitest ref v5.4.3 (commit 32dad2e92a4953a052ece1ed689ee62a17b3abd8)
minitest 5.4.3: calling rubygems.to_chunk to generate chunk morph
rubygems.to_chunk failed with code 1: Did not find any files matching minitest.gemspec within /src/baserock-import/checkouts/ruby-gems_minitest.
thread_safe: calling rubygems.to_lorry to generate lorry
thread_safe 0.3.4: using checkouts/ruby-gems_thread_safe ref v0.3.4 (commit 21ea387dce966f3a6a38752ab6e199bb9a2d710b)
json: calling rubygems.to_lorry to generate lorry
json 1.8.1: using checkouts/ruby-gems_json ref v1.8.1 (commit 92a96dea2b24b9c68856004d69491f46aedd0925)
tzinfo: calling rubygems.to_lorry to generate lorry
tzinfo 1.2.2: using checkouts/ruby-gems_tzinfo ref v1.2.2 (commit f1c8830e584d08689ed2ac11c54c457dc8fd10d6)
i18n: calling rubygems.to_lorry to generate lorry
i18n 0.6.11: using checkouts/ruby-gems_i18n ref v0.6.11 (commit 9eadb6bb71cf13554514d2d6f22a0884891ebee0)
railties: calling rubygems.to_lorry to generate lorry
railties 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
thor: calling rubygems.to_lorry to generate lorry
thor 0.19.1: using checkouts/ruby-gems_thor ref v0.19.1 (commit c2da2f5b26f976510965adda01e4893d1eae7f7d)
actionpack: calling rubygems.to_lorry to generate lorry
actionpack 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
rack-test: calling rubygems.to_lorry to generate lorry
rack-test 0.6.2: using checkouts/ruby-gems_rack-test ref v0.6.2 (commit 5080d1e99de825bdd3b2b7bba34e5ac61cedd876)
rack: calling rubygems.to_lorry to generate lorry
rack 1.5.2: using checkouts/ruby-gems_rack ref 1.5.2 (commit ac590d055c936bb9a618e955a690dc836c625211)
actionmailer: calling rubygems.to_lorry to generate lorry
actionmailer 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
mail: calling rubygems.to_lorry to generate lorry
mail 2.6.3: using checkouts/ruby-gems_mail ref v2.6.3 (commit ab034a4d3d6ef1149cf6989437ec043561a1217e)
mail 2.6.3: calling rubygems.to_chunk to generate chunk morph
mail 2.6.3: calling rubygems.find_deps to calculate dependencies
mime-types: calling rubygems.to_lorry to generate lorry
Lorrying https://github.com/halostatue/mime-types/
mime-types 2.4.3: using checkouts/ruby-gems_mime-types ref v2.4.3 (commit fbdde28897b51c9de928fcc82b96709a74f848d8)
mime-types 2.4.3: calling rubygems.to_chunk to generate chunk morph
rubygems.to_chunk failed with code 1: Source in /src/baserock-import/checkouts/ruby-gems_mime-types produces mime-types-2.4.1, but the expected version was 2.4.3.
sprockets-rails: calling rubygems.to_lorry to generate lorry
sprockets-rails 2.2.2: using checkouts/ruby-gems_sprockets-rails ref v2.2.2 (commit b54af68acbdfaf3bb2bd5940ef4bc2d5a41a38f4)
sprockets: calling rubygems.to_lorry to generate lorry
sprockets 2.12.3: using checkouts/ruby-gems_sprockets ref v2.12.3 (commit 55a53fcdd352bc906ec04945245920aedf105a02)
tilt: calling rubygems.to_lorry to generate lorry
tilt 1.4.1: using checkouts/ruby-gems_tilt ref 1.4.1 (commit d67b24d14c84b6a877fbd2b5466028ecba6981c3)
multi_json: calling rubygems.to_lorry to generate lorry
multi_json 1.10.1: using checkouts/ruby-gems_multi_json ref v1.10.1 (commit 0c600e2a3274f4383fb02368635af9bab4cf893b)
hike: calling rubygems.to_lorry to generate lorry
hike 1.2.3: using checkouts/ruby-gems_hike ref v1.2.3 (commit f313ea1fd44caf2cae004fd45c67017b744bbb7a)
activerecord: calling rubygems.to_lorry to generate lorry
activerecord 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
arel: calling rubygems.to_lorry to generate lorry
arel 5.0.1.20140414130214: using checkouts/ruby-gems_arel ref v5.0.1.20140414130214 (commit cef5bbd43064f608bb2724543e3138fd03ffb84a)
arel 5.0.1.20140414130214: calling rubygems.to_chunk to generate chunk morph
arel 5.0.1.20140414130214: calling rubygems.find_deps to calculate dependencies
activemodel: calling rubygems.to_lorry to generate lorry
activemodel 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
builder: calling rubygems.to_lorry to generate lorry
builder 3.2.2: using checkouts/ruby-gems_builder ref v3.2.2 (commit 38c8f83d27670fee9ce057a2e51759b8f62a0b38)
builder 3.2.2: calling rubygems.to_chunk to generate chunk morph
rubygems.to_chunk failed with code 1: Did not find any files matching builder.gemspec within /src/baserock-import/checkouts/ruby-gems_builder.
actionview: calling rubygems.to_lorry to generate lorry
actionview 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
erubis: calling rubygems.to_lorry to generate lorry
erubis 2.7.0: using checkouts/ruby-gems_erubis ref v2.7.0 (commit 14d3eab57fbc361312c8f3af350cbf9a5bafce17)
erubis 2.7.0: calling rubygems.to_chunk to generate chunk morph
rubygems.to_chunk failed with code 1: /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler.rb:394:in `rescue in eval_gemspec': There was a ArgumentError while loading erubis.gemspec:  (Bundler::GemspecError)
Malformed version number string $Release$ from
/src/baserock-import/checkouts/ruby-gems_erubis/erubis.gemspec:15:in `block in <main>'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler.rb:383:in `eval_gemspec'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler.rb:354:in `block in load_gemspec_uncached'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler/shared_helpers.rb:52:in `chdir'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler/shared_helpers.rb:52:in `chdir'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler.rb:349:in `load_gemspec_uncached'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler.rb:339:in `load_gemspec'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler/dsl.rb:53:in `gemspec'
    from /src/import/baserockimport/exts/importer_bundler_extensions.rb:61:in `create_bundler_definition_for_gemspec'
    from /src/import/baserockimport/exts/rubygems.to_chunk:161:in `block in run'
    from /src/import/baserockimport/exts/rubygems.to_chunk:156:in `chdir'
    from /src/import/baserockimport/exts/rubygems.to_chunk:156:in `run'
    from /src/import/baserockimport/exts/rubygems.to_chunk:173:in `<main>'

Errors encountered, not generating a stratum morphology.
See the README files for guidance.
12/01/14 16:29:02 UTC: Import of rubygems rails ended (took 27 seconds)

More errors! That's great news. Now I can write another section on some of the fun things that can go wrong with .gemspec files.

Chapter 3: Generated chunk morphologies

The Import Tool's Git repo contains file named README.rubygems, which has more information on generated chunk morphologies (and other stuff).

Take a quick moment to look through what the Import Tool has generated for one of the Gems.

name: rack-1.5.2
kind: chunk
description: Automatically generated by rubygems.to_chunk
products:
- artifact: rack-1.5.2-doc
include:
- usr/lib/ruby/gems/\d[\w.]*/doc/.*
build-commands:
- gem build rack.gemspec
install-commands:
- mkdir -p "$DESTDIR/$(gem environment home)"
- gem install --install-dir "$DESTDIR/$(gem environment home)" --bindir "$DESTDIR/$PREFIX/bin"
    --ignore-dependencies --local ./rack-1.5.2.gem

There are a couple of things in here that should, in future, be unnecessary because they should be handled by Morph's defaults.

The gem build, gem install command sequence is almost the same for all projects that ship conventional .gemspec files. There is a slight difference in each one though: the name of the Gem. This isn't a built-in 'buildsystem' type in Morph yet because there'd be no way to pass in the name of the Gem.

Also, it manually describes a -doc artifact. This is done in case you want to produce a system which doesn't contain the documentation for every Gem, which can total hundreds of megabytes of content. Morph has default chunk splitting rules that should be extended to handle this.

Chapter 4: Non-standard projects

The design of the Import Tool conflicts a bit with common practice in the Ruby community. Most Gem authors consider http://www.rubygems.org the canonical source for their project. The upstream source repository is an 'implementation detail', which is used to publish the source package on rubygems.org, but shouldn't necessarily be relied upon to follow any specific rules.

The majority of Gems do follow conventions like tagging releases and maintaining a .gemspec file, so using their source control repositories directly isn't a problem, and getting the source from a Git repo rather than from a source package fits better with the Baserock tooling. But some projects don't follow the conventions we expect, for various reasons.

Here's all of the output for the packages that raised errors in the last run:

minitest 5.4.3: using checkouts/ruby-gems_minitest ref v5.4.3 (commit 32dad2e92a4953a052ece1ed689ee62a17b3abd8)
minitest 5.4.3: calling rubygems.to_chunk to generate chunk morph
rubygems.to_chunk failed with code 1: Did not find any files matching minitest.gemspec within /src/baserock-import/checkouts/ruby-gems_minitest.
Lorrying https://github.com/halostatue/mime-types/
mime-types 2.4.3: using checkouts/ruby-gems_mime-types ref v2.4.3 (commit fbdde28897b51c9de928fcc82b96709a74f848d8)
mime-types 2.4.3: calling rubygems.to_chunk to generate chunk morph
mime-types 2.4.3: rubygems.to_chunk failed with code 1: Source in /src/baserock-import/checkouts/ruby-gems_mime-types produces mime-types-2.4.1, but the expected version was 2.4.3.
builder 3.2.2: using checkouts/ruby-gems_builder ref 3.2.2 (commit 38c8f83d27670fee9ce057a2e51759b8f62a0b38)
builder 3.2.2: calling rubygems.to_chunk to generate chunk morph
rubygems.to_chunk failed with code 1: Did not find any files matching builder.gemspec within /src/baserock-import/checkouts/ruby-gems_builder.
erubis 2.7.0: using checkouts/ruby-gems_erubis ref 2.7.0 (commit 14d3eab57fbc361312c8f3af350cbf9a5bafce17)
erubis 2.7.0: calling rubygems.to_chunk to generate chunk morph
rubygems.to_chunk failed with code 1: /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler.rb:394:in `rescue in eval_gemspec': There was a ArgumentError while loading erubis.gemspec:  (Bundler::GemspecError)
Malformed version number string $Release$ from
/src/baserock-import/checkouts/ruby-gems_erubis/erubis.gemspec:15:in `block in <main>'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler.rb:383:in `eval_gemspec'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler.rb:354:in `block in load_gemspec_uncached'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler/shared_helpers.rb:52:in `chdir'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler/shared_helpers.rb:52:in `chdir'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler.rb:349:in `load_gemspec_uncached'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler.rb:339:in `load_gemspec'
    from /usr/lib/ruby/gems/2.0.0/gems/bundler-1.7.6/lib/bundler/dsl.rb:53:in `gemspec'
    from /usr/lib/python2.7/site-packages/baserockimport/exts/importer_bundler_extensions.rb:61:in `create_bundler_definition_for_gemspec'
    from /usr/lib/python2.7/site-packages/baserockimport/exts/rubygems.to_chunk:161:in `block in run'
    from /usr/lib/python2.7/site-packages/baserockimport/exts/rubygems.to_chunk:156:in `chdir'
    from /usr/lib/python2.7/site-packages/baserockimport/exts/rubygems.to_chunk:156:in `run'
    from /usr/lib/python2.7/site-packages/baserockimport/exts/rubygems.to_chunk:173:in `<main>'

minitest

rubygems.to_chunk failed with code 1: Did not find any files matching
minitest.gemspec within
/baserock-import/checkouts/ruby-gems_minitest.

Minitest uses a tool called Hoe. Hoe extends Rake (the Ruby equivalent of Make) to allow various things to be automatically generated. A reasonable number of projects use Hoe, but the RubyGems 'to_chunk' program has not yet been extended to understand how Hoe works. This means you need to manually write a chunk .morph file for Minitest.

There's no .gemspec provided in the minitest.git repo, so we can't use gem build. But we can use rake gem, as the Hoe author intended. Save this text into a file called definitions/strata/rails/minitest-5.4.3.morph:

name: minitest-5.4.3
kind: chunk
build-system: manual
products:
- artifact: minitest-5.4.3-doc
  include:
  - usr/lib/ruby/gems/\d[\w.]*/doc/.*
build-commands:
- rake gem
install-commands:
- mkdir -p "$DESTDIR/$(gem environment home)"
- gem install --install-dir "$DESTDIR/$(gem environment home)" --bindir "$DESTDIR/$PREFIX/bin"
    --ignore-dependencies --local pkg/minitest-5.4.3.gem

This works because Hoe is in the Baserock Ruby stratum and so it will be present at built time (and is present in your devel system). If Hoe wasn't available, you'd see a rake aborted! LoadError: cannot load such file -- hoe error.

You might be wondering why we don't default to running rake gem. Firstly, the exact command that needs to be run varies from Gem to Gem (although rake gem often works). There are also other tools which extend Rake, such as Jeweler, which require lots of other Gems to be present for rake gem to work. In contrast, gem build can be used without requiring any extra dependencies.

If you ran the tool again now:

baserock-import rubygems rails 4.1.8 \
--definitions-dir=definitions \
--lorries-dir=lorries/open-source-lorries \
--log=import.log

You'd notice a new error:

minitest 5.4.3: using checkouts/ruby-gems_minitest ref v5.4.3 (commit 32dad2e92a4953a052ece1ed689ee62a17b3abd8)
minitest 5.4.3: calling rubygems.find_deps to calculate dependencies
rubygems.find_deps failed with code 1: Did not find any files matching minitest.gemspec within /src/baserock-import/checkouts/ruby-gems_minitest.

Remember the procedure described in the README? After the tool runs its rubygems.to_chunk program to create the chunk .morph file, it runs rubygems.find_deps which analyses the .gemspec file to find what other Gems this one depends on. It writes that information to a .foreign-dependencies file in the same directory as the chunk morph.

If there's no .gemspec file then this will break too. But we can generate a temporary .gemspec with the following trick, run in the checkouts/ruby-gems_minitest directory:

$ rake gem
$ gem spec pkg/minitest-5.4.3.gem > minitest.gemspec

mime-types

You will also get this error:

mime-types 2.4.3: rubygems.to_chunk failed with code 1: Source in /src/baserock-import/checkouts/ruby-gems_mime-types produces mime-types-2.4.1, but the expected version was 2.4.3.

The mime-types Gem uses Hoe, like Minitest. It seems that at the time of writing they had a .gemspec file committed to their Git repository but it was slightly out of date.

Including a .gemspec is optional (some projects specifically avoid it), so it's not surprising that they can sometimes get out of date.

We can create a chunk morph for 'mime-types' just like the one for Minitest. Save this as definitions/strata/rails/mime-types-2.4.3.morph.

name: mime-types-2.4.3
kind: chunk
build-system: manual
products:
- artifact: mime-types-2.4.3-doc
  include:
  - usr/lib/ruby/gems/\d[\w.]*/doc/.*
build-commands:
- rake gem
install-commands:
- mkdir -p "$DESTDIR/$(gem environment home)"
- gem install --install-dir "$DESTDIR/$(gem environment home)" --bindir "$DESTDIR/$PREFIX/bin"
    --ignore-dependencies --local pkg/mime-types-2.4.3.gem

Also edit the .gemspec file so that the version number is correct, so that the rubygems.find_deps extension can run.

builder

The Builder Gem doesn't ship any .gemspec. You will also notice if you try to run rake then it doesn't work, unless you happen to have installed RVM:

$ rake --tasks
rake aborted!
No such file or directory - rvm
/src/baserock-import/checkouts/ruby-gems_builder/rakelib/tags.rake:15:in ``'
/src/baserock-import/checkouts/ruby-gems_builder/rakelib/tags.rake:15:in `<module:Tags>'
/src/baserock-import/checkouts/ruby-gems_builder/rakelib/tags.rake:3:in `<top (required)>'
(See full trace by running task with --trace)

The functionality of RVM should be mostly redundant if you are using Baserock, so it's not included by default. A quick dive into the Builder source code shows that the issue is simply an oversight in the Rake tasks. This patch fixed the build for me: at the time of writing you'll need to apply this manually to the working tree in checkouts/ruby-gems_builder, to fix the rakelib/tags.rake so that RVM is optional.

After applying this patch we can now run rake --tasks and see that in fact there is no rake gem task, but there is a rake builder:gem task which produces a .gem file!

Thus, create the file definitions/strata/rails/builder-3.2.2.morph with the following contents:

name: builder-3.2.2
kind: chunk
build-system: manual
products:
- artifact: builder-3.2.2-doc
  include:
  - usr/lib/ruby/gems/\d[\w.]*/doc/.*
build-commands:
- rake builder:gem
install-commands:
- mkdir -p "$DESTDIR/$(gem environment home)"
- gem install --install-dir "$DESTDIR/$(gem environment home)" --bindir "$DESTDIR/$PREFIX/bin"
    --ignore-dependencies --local pkg/builder-3.2.2.gem

Also, create a .gemspec file so that the import tool can read the dependencies from it:

$ cd checkouts/ruby-gems_builder/
$ rake builder:gem
$ gem spec pkg/builder-3.2.2.gem > builder.gemspec
$ cd ../../

erubis

Malformed version number string $Release$ from
(...)

The Erubis Gem uses a templating system named 'Rook'. Rook was written by the same person as Erubis. The only version that seems to exist is a .gem file from 2006, marked 'obsolete', listed at the bottom of this page.

The erubis.gemspec file isn't valid Ruby, due to the $Release$ template marker, so the tool can't process it directly. Since Rook seems to be used only in Erubis, it's not worth teaching the Import Tool about it: instead, we can create a chunk .morph for Erubis manually, and use sed to do the necessary substitution.

Save this as definitions/strata/rails/erubis-2.7.0.morph:

name: erubis-2.7.0
kind: chunk
build-system: manual
products:
- artifact: erubis-2.7.0-doc
  include:
  - usr/lib/ruby/gems/\d[\w.]*/doc/.*
configure-commands:
# Manually do what it seems like the 'rook' build system would do.
# Values taken from 'Rookbook.props'.
- find -type f -exec sed -e 's/\$Release\$/2.7.0/g' -i \{} \;
- find -type f -exec sed -e 's/\$Copyright\$/copyright(c) 2006-2011 kuwata-lab.com all rights reserved./g' -i \{} \;
build-commands:
- gem build erubis.gemspec
install-commands:
- mkdir -p "$DESTDIR/$(gem environment home)"
- gem install --install-dir "$DESTDIR/$(gem environment home)" --bindir "$DESTDIR/$PREFIX/bin"
    --ignore-dependencies --local erubis-2.7.0.gem

Manually fix the .gemspec file too so that the tool can read the dependency information from it:

cd checkouts/ruby-gems_erubis
find -type f -exec sed -e 's/\$Release\$/2.7.0/g' -i \{} \;
cd ../..

Now that you've manually created a chunk .morph for all of the Gems where it couldn't be done automatically, try running the tool again.

$ baserock-import rubygems rails 4.1.8 \
    --definitions-dir=definitions \
    --lorries-dir=lorries/open-source-lorries \
    --log=import.log
12/02/14 18:20:29 UTC: Import of rubygems rails started
Not updating existing Git checkouts or existing definitions
rails 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
activesupport 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
minitest 5.4.3: using checkouts/ruby-gems_minitest ref v5.4.3 (commit 32dad2e92a4953a052ece1ed689ee62a17b3abd8)
thread_safe 0.3.4: using checkouts/ruby-gems_thread_safe ref v0.3.4 (commit 21ea387dce966f3a6a38752ab6e199bb9a2d710b)
json 1.8.1: using checkouts/ruby-gems_json ref v1.8.1 (commit 92a96dea2b24b9c68856004d69491f46aedd0925)
tzinfo 1.2.2: using checkouts/ruby-gems_tzinfo ref v1.2.2 (commit f1c8830e584d08689ed2ac11c54c457dc8fd10d6)
i18n 0.6.11: using checkouts/ruby-gems_i18n ref v0.6.11 (commit 9eadb6bb71cf13554514d2d6f22a0884891ebee0)
railties 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
thor 0.19.1: using checkouts/ruby-gems_thor ref v0.19.1 (commit c2da2f5b26f976510965adda01e4893d1eae7f7d)
actionpack 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
rack-test 0.6.2: using checkouts/ruby-gems_rack-test ref v0.6.2 (commit 5080d1e99de825bdd3b2b7bba34e5ac61cedd876)
rack 1.5.2: using checkouts/ruby-gems_rack ref 1.5.2 (commit ac590d055c936bb9a618e955a690dc836c625211)
actionmailer 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
mail 2.6.3: using checkouts/ruby-gems_mail ref v2.6.3 (commit ab034a4d3d6ef1149cf6989437ec043561a1217e)
mime-types 2.4.3: using checkouts/ruby-gems_mime-types ref v2.4.3 (commit fbdde28897b51c9de928fcc82b96709a74f848d8)
mime-types 2.4.3: calling rubygems.find_deps to calculate dependencies
sprockets-rails 2.2.2: using checkouts/ruby-gems_sprockets-rails ref v2.2.2 (commit b54af68acbdfaf3bb2bd5940ef4bc2d5a41a38f4)
sprockets 2.12.3: using checkouts/ruby-gems_sprockets ref v2.12.3 (commit 55a53fcdd352bc906ec04945245920aedf105a02)
tilt 1.4.1: using checkouts/ruby-gems_tilt ref 1.4.1 (commit d67b24d14c84b6a877fbd2b5466028ecba6981c3)
multi_json 1.10.1: using checkouts/ruby-gems_multi_json ref v1.10.1 (commit 0c600e2a3274f4383fb02368635af9bab4cf893b)
hike 1.2.3: using checkouts/ruby-gems_hike ref v1.2.3 (commit f313ea1fd44caf2cae004fd45c67017b744bbb7a)
activerecord 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
arel 5.0.1.20140414130214: using checkouts/ruby-gems_arel ref v5.0.1.20140414130214 (commit cef5bbd43064f608bb2724543e3138fd03ffb84a)
activemodel 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
builder 3.2.2: using checkouts/ruby-gems_builder ref v3.2.2 (commit 38c8f83d27670fee9ce057a2e51759b8f62a0b38)
actionview 4.1.8: using checkouts/ruby-gems_rails ref v4.1.8 (commit b32babc4b0ff8f830933f25375ce9dbfbb356601)
erubis 2.7.0: using checkouts/ruby-gems_erubis ref v2.7.0 (commit 14d3eab57fbc361312c8f3af350cbf9a5bafce17)
Generating stratum morph for rails
12/02/14 18:20:37 UTC: Import of rubygems rails ended (took 8 seconds)

I'd like to think that we both got the same output. In practice, it took me a number of attempts to get to this point, and I doubt you got all of the fiddly steps above correct straight away either. The baserock-import tool is designed on the assumption that, when importing packages created by humans, there will always be some that don't work properly. That's why baserock-import is not afraid to show you a backtrace from one of its subprocesses now and again. And that's why you should be able to easily run baserock-import many times while fixing up the non-standard projects you need to import, without the tool getting in your way. If you find that it does get in your way, please consider it a bug and report it to us.

Chapter 5: Cooking up a system

A stratum exists for Rails now! Have a look at it: it should be in definitions/strata/rails.morph.

If you're familiar with Baserock already you might have picked up on the fact that the 'repo' field for each chunk is set to stuff like 'upstream:ruby-gems/newly-imported-gem'. You'll know that this is a keyed URL, which will expand to a URL on whatever Trove (Git server) you have configured Morph to use. If you're using the default Morph configuration it'll expand like this: git://git.baserock.org/delta/ruby-gems/newly-imported-gem. That URL doesn't work.

How do we get our newly-imported Gems into the Trove so that we can build a system? The answer is: I'll tell you later, because for now you don't need to. The 'repo' field can take normal URLs as well as keyed URLs, and Morph and Git understand the file:/// protocol, so there's nothing stopping us building the local repos that the Import Tool made available for us in the checkouts/ subdirectory.

The Import Tool accepts a --use-local-sources option, which causes it to generate a stratum with the appropriate file:/// URLs.

Note that it won't overwrite the stratum it created, to avoid losing any changes that you might have made to it since, so you'll need to remove it first (or pass --force-stratum-generation to the tool).

$ rm definitions/strata/rails.morph
$ baserock-import rubygems rails 4.1.8 \
    --use-local-sources --definitions-dir=definitions \
    --lorries-dir=lorries/open-source-lorries \
    --log=import.log

Let's try and build it. Seems the example definitions/systems/web-system-x86_64.morph system would be a good environment to test Rails, so add the following entry to the list of strata in that file, then try to build it.

- name: rails
  morph: strata/rails.morph

To build:

$ morph build systems/web-system-x86_64-generic.morph

You might have to build some unrelated components of the 'web-system' to begin with. This should only happen once.

Fixing 'builder'

How far did it get? I got to here:

2014-12-04 11:10:12 [Build 163/184] [builder-3.2.2] Building chunk builder-3.2.2
2014-12-04 11:10:12 [Build 163/184] [builder-3.2.2] Creating staging area
2014-12-04 11:10:14 [Build 163/184] [builder-3.2.2] Starting actual build: builder-3.2.2 38c8f83
2014-12-04 11:10:14 [Build 163/184] [builder-3.2.2] Extracting file:///src/baserock-import/checkouts/ruby-gems_builder into /var/tmp/staging/tmpxejXSA/builder-3.2.2.build
2014-12-04 11:10:14 [Build 163/184] [builder-3.2.2] Running build-commands
build failed
# build
# # rake builder:gem
rake aborted!
No such file or directory - rvm
/builder-3.2.2.build/rakelib/tags.rake:15:in ``'
/builder-3.2.2.build/rakelib/tags.rake:15:in `<module:Tags>'
/builder-3.2.2.build/rakelib/tags.rake:3:in `<top (required)>'
(See full trace by running task with --trace)
ERROR: In staging area /var/tmp/staging/tmpxejXSA: running command 'sh -c rake builder:gem' failed.

This looks like a familiar error! If you look in the 'rails.morph' stratum you'll see the following entry:

- name: builder-3.2.2
  morph: strata/rails/builder-3.2.2.morph
  repo: file:///src/baserock-import/checkouts/ruby-gems_builder
  ref: 38c8f83d27670fee9ce057a2e51759b8f62a0b38
  unpetrify-ref: v3.2.2
  build-depends: []

Above, we patched 'builder' in the checkouts/ subdirectory so that the Rakefile didn't run RVM. If you commit it on a branch you can then use that ref in the stratum. If you later add this repo to a Trove or another Git server you can push the branch to that remote, too.

Change the stratum to use the commit you just made (your SHA1 may be different to mine, due to being commited at a different time or for other reasons).

- name: builder-3.2.2
  morph: strata/rails/builder-3.2.2.morph
  repo: file:///src/baserock-import/checkouts/ruby-gems_builder
  ref: 2d29046f12180c00f66539a99b0328140d9fa711
  unpetrify-ref: baserock/me/fix-rakefile
  build-depends: []

I'd prefer if it were less work to patch a chunk during import: there is a TODO item about it.

Fixing 'mime-types'

Run morph build systems/web-system-x86_64-generic.morph again. Now it gets to here:

2014-12-04 11:28:28 [Build 169/184] [mime-types-2.4.3] Building chunk mime-types-2.4.3
2014-12-04 11:28:28 [Build 169/184] [mime-types-2.4.3] Creating staging area
2014-12-04 11:28:30 [Build 169/184] [mime-types-2.4.3] Starting actual build: mime-types-2.4.3 fbdde28
2014-12-04 11:28:30 [Build 169/184] [mime-types-2.4.3] Extracting file:///src/baserock-import/checkouts/ruby-gems_mime-types into /var/tmp/staging/tmpNUGBOD/mime-types-2.4.3.build
2014-12-04 11:28:30 [Build 169/184] [mime-types-2.4.3] Running build-commands
build failed
# build
# # rake gem
rake aborted!
Don't know how to build task 'travis'
/mime-types-2.4.3.build/Rakefile:147:in `<top (required)>'
(See full trace by running task with --trace)
ERROR: In staging area /var/tmp/staging/tmpNUGBOD: running command 'sh -c rake gem' failed.

The problem here is the following line in the Rakefile:

Rake::Task['travis'].prerequisites.replace(%w(test:coveralls))

Removing this line prevents the rake travis task from working, but we don't need it anyway. Since this is a single line change, we can do it using sed to avoid having to carry a patch against upstream. It would be worth discussing ways to fix the issue upstream, but I don't have time to investigate what the best solution would be right now.

Add these lines to the definitions/strata/rails/mime-types-2.4.3.morph file:

configure-commands:
- sed -e "/Rake::Task\['travis'\]/ d" -i Rakefile

Then run morph build systems/web-system-x86_64-generic.morph again.

Testing out the system

Now you should have a built system! It might take a little time to assemble the final artifact -- the web-system is a 'batteries included' sort of system and totals 1.7GB at the time of writing.

In order to use the system you need to deploy it somewhere. There are various ways to deploy a system, see morph help deploy and morph help-extensions for the various options. Personally, I tested my system by deploying it to a .tar file:

$ cat > definitions/tar.morph <<'EOF'
name: tar
kind: cluster
systems:
- morph: systems/web-system-x86_64-generic.morph
  deploy:
    default:
      type: extensions/tar
      location: web-system.tar
EOF
$ morph deploy tar.morph

I then copied the web-system.tar file onto another machine, imported it into Docker and started a container from it.

$ docker import - test-rails-system < definitions/web-system.tar
$ docker run -i -t --rm test-web-system /bin/sh
$ cd /tmp

Trying out a Rails project

Let's make a new Rails project and see if it works.

$ rails new /tmp/testproject --skip-bundle
...
$ cd /tmp/testproject
$ rake --tasks
Could not find gem 'spring (>= 0) ruby' in the gems available on this machine.
Run `bundle install` to install missing gems.

Oh! First, notice that I used the --skip-bundle option, to force the use of the system versions of the Gems that we've so carefully integrated. This goes against the common practices of the Ruby community, but the common practice seems to have developed to make it easy to deploy the same application in various open-ended environments. If you're looking at Baserock, you've probably moved past this idea already: Baserock guides to towards making controlled, closed environments for deployment, making tools like Bundler unnecessary.

But it seems that we're lacking some Gems in our system. Why is that? Look in the Gemfile that the rails new command has generated for you: there are a bunch of things listed there. This list includes 'rails', plus a bunch of Gems which are kind of optional dependencies, but are actually needed when you want to run your application, like sqlite3 (which is often as a database during development, but not in production).

This makes sense if you think about it. It's a bit disappointing that we can't use the 'rails' stratum we just created to run the test app, but it makes sense that the Rails Gem only lists the essential dependencies and not every optional dependency you might choose to use.

So how do we proceed from here? Well, there's nothing stopping you using Bundler while we're prototyping:

$ bundle install --system
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
Fetching gem metadata from https://rubygems.org/..........
Resolving dependencies...

Gem::RemoteFetcher::FetchError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://rubygems.org/gems/rake-10.4.2.gem)
An error occurred while installing rake (10.4.2), and Bundler cannot continue.
Make sure that `gem install rake -v '10.4.2'` succeeds before bundling.

Okay, there is one thing stopping us. You've hit a bug that's present in Baserock right now where update-ca-certificates isn't run at the correct time during morph build. If you run it manually this error will go away:

$ update-ca-certificates
Updating certificates in /etc/ssl/certs... 168 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....run-parts: /etc/ca-certificates/update.d: No such file or directory
done.
$ bundle install --system
...
make: /usr/bin/mkdir: Command not found
make: *** [.RUBYARCHDIR.time] Error 127


Gem files will remain installed in /usr/lib/ruby/gems/2.0.0/gems/sqlite3-1.3.10 for inspection.
Results logged to /usr/lib/ruby/gems/2.0.0/gems/sqlite3-1.3.10/ext/sqlite3/gem_make.out
An error occurred while installing sqlite3 (1.3.10), and Bundler cannot
continue.
Make sure that `gem install sqlite3 -v '1.3.10'` succeeds before bundling.

There seems to be a bug in the 'sqlite3' Gem where it assumes /usr/bin/mkdir exists. In Baserock we only have/bin/mkdir`. I'd love to see this fixed, but for the time being I've just created a symlink:

$ ln -s /bin/mkdir /usr/bin/mkdir
$ bundle install --system
...

You've now got all the stuff you need to play with a test Rails application.

I hope the fact that we've had to install some Gems to actually play with Rails isn't too disappointing. If you want to import an existing Ruby on Rails application into Baserock you should use the tool to import the application, rather than Rails itself.

Note that the tool can only import Gems (i.e. projects which have a corresponding .gemspec file and are known to http://rubygems.org/) at the moment. The code uses Bundler internally, and could be extended to read dependencies from other Gem sources or from any Git repo that contains a Gemfile. For now, this is left as an exercise to the reader (it is described further in TODO.rubygems).

Chapter 6: Next steps

We made a prototype system with enough of Rails that the rails new command worked (although we couldn't make use of the project it created without installing more Gems in the system). But the imported repos and the chunks only exist on the local machine. If you're working on a project with Baserock you might be using a Trove to mirror upstream projects and store your source code.

Getting the Trove to Lorry the projects

The Import Tool uses Lorry in your local system to fetch source code. Trove contains Lorry too and will run it regularly, keeping a mirror of the source code up to date.

The .lorry file or files created by the tool are directly usable in a Trove. Just send them for review using whatever review process you have, and merge them to 'master' of the Trove's local-config/lorries.git repo.

If you've been following this tutorial to the letter, you'll have used the --lorries-dir flag to instruct the tool to place the .lorry files into a clone of your local-config/lorries.git repo. Note that the default behaviour of the tool for RubyGem lorries is to write them all to one file, named 'ruby-gems.lorry'. This behaviour can be configured in the file baserockimport/data/rubygems.yml in the tool's source code: remove the lorry-prefix line if you want each Gem to get its own .lorry file.

Merging the morphologies into your project's definitions repo

Once all the repos are available in your Trove, you can use them in your stratum. Remember above when we used the --use-local-sources option of the tool so that file:/// URLs were used? Delete the rails.morph file and run the tool again without that option to generate a version that uses 'upstream:' URLs.

Now you have a stratum ready for merging into your definitions (throw away the .foreign-dependencies files, they can be regenerated easily enough).

In the Baserock project's definitions.git, chunks have names like 'linux' rather than 'linux-3.17.4'. The Import Tool needs to be changed to support this convention (in a way that doesn't break it when multiple versions of the same project appear in a stratum). In the meantime, here's an ugly way of removing the version numbers from the morphologies, if you want to do that. I recommend committing the .morph files to Git first, then running this command, so you can use git diff to check that it did what you wanted.

name_with_version='\(.\+\)-[0-9]\+\(\.[0-9a-zA-Z]\+\)*'
for f in *.morph; do \
    sed -e "s/name: ${name_with_version}/name: \\1/" \
        -e "s/artifact: ${name_with_version}-doc/artifact: \\1-doc/" \
        -i "$f" \
done

Thanks for reading this tutorial, I hope it more or less made sense and I hope that you find it nicer to use than working through all of the dependencies you need by hand! Please send any feedback to the mailing list. Most of the tool and most of this tutorial was written by Sam Thursfield.