Declarative Definitions

We want to establish a standard syntax for describing software components and systems. Learning from the experiences we have had with morph files, we have established some general principles:

• We create descriptions of software components, systems, and sets of systems. We call the descriptions Definitions.
• Software tools can use Definitions to create and operate on these components and systems. By adopting the Definitions approach we are aiming for traceability, reproducibility and ultimately deterministic behaviour of the Defined systems and components.
• As far as possible Definitions contain all the information necessary for build, deployment, instantiation etc.
• Definitions are a a set of text files in a directory tree.Typically the files are stored in a single Git repo
• Definitions describe Components. Each Definition file may describe one or more Components.
• A Component may consist of
  • a single software project e.g. systemd, or
  • a set of software projects, e.g. GNOME, or
  • an integrated software system, e.g. a web server appliance, or
  • a set or cluster of software systems, e.g. a ci controller with several worker nodes
• Components may be nested and contain other components. The information which specifies a Component may be split across multiple Definition files.
• We have currently chosen YAML as the format for representing data relationships in Definitions.
• Most Definitions contain YAML, but some definitions contain shell scripts or other instructions to be executed as part of a configuration, build, installation, deployment, instantiation, upgrade or other process which tools can apply to Components or systems.

VERSION

We have evolved morph files through many states over several years, without providing a mechanism for versioning as the syntax has evolved. As of the meetup we have decided to adopt a versioning scheme, broadly as follows:

• The Definitions directory tree contains a file called VERSION. If this file is not present, tooling may assume definitions-V0 or fail gracefully
• The first line of the VERSION file must contain version: followed by an integer. The integer uniquely defines the version of the Definitions syntax
• Later lines of the VERSION file are expected to contain YAML describing default information which tooling such as Morph may require. This information could include, for example
  • default PATH
  • locations for git server and artifact server
  • default ARCH target etc.
• By specifying defaults here we aim to reduce complexity, hidden logic and implicit assumptions in tools such as morph. The overall objectives for this are
  • running baserock tools on a set of Definitions should give predictable and reproducible results - if it works for Aananth on his machine it should work for Brenda on hers too
  • it should be possible for new tooling to instrument, analyse and make operations on the information contained in Definitions, with confidence that there are no hidden tricks performed by existing tools (e.g. anything that has to be done differently for ARMv7 should be explicitly declared in Definitions, not as conditional logic in the build tool itself)
• We expect that the version number will rise relatively quickly, so
  • tooling should expect and handle changes gracefully
  • if something is expected to break between VERSION N and VERSION N+1, a migration script must be provided. We see this as similar to ruby on rails migrations, and hope to standardize this process over the first few VERSION iterations

Coming soon...

Here are some changes we expect to achieve over the first few VERSION iterations

1. s/.morph/.def/
2. Move build-system definitions to definitions.git
3. Drop support for automatic build-system detection
4. Move the definitions of per-architecture flags to definitions.git out of morph
5. Move all configuration/write extensions back into definitions.git
6. Change the format of configuration/write extensions to be of declarative form, as a yaml document that describes its inputs and either in-lines the code involved in the configuration/write extension, or provides instruction on how to run the code, referred to elsewhere in the document.
7. Move repository alias resolution configuration/logic to definitions.git out of local morph configuration files.

To be considered...

Discussion on irc highlighted the need for being able to include or exclude testing (both the building of tests, and the running of the built tests) depending on use-case. We should consider how to address this via both definitions and the tooling which operates on them