The cleopatra Build Process

1. Overview
2. The init phase
- 2.1. Tangling Generation Processes
- 2.2. Generating the Dependency Graph
3. The postbuild phase

In this chapter, we aim to explain how cleopatra works. And since cleopatra is far from being the single available build tool (it is merely an overlay on an existing one, actually!), we avoid ambiguity by defining a small, yet consistent terminology throughout this book.

Root

The directory which contains the cleopatra.toml file, and supposedly the root of the source tree of the software that cleopatra is expected to build.

Workspace

The execution environment set up by cleopara prior to starting a build. In particular, this workspace consists in

The current working directory is the root of the project
The ROOT environment variable has been set with the root of the project
The CLEOPATRA_DIRECTORY environment variable has been set with ${ROOT}/.cleopatra
The CLEOPATRA_GENERATION_PROCESSES environment variable has been set with the value of the generation_processes entry in cleopatra.toml

Generation Process

An Org document describing a set of build recipes, along with the auxiliary files used by these rules.

Build Process

How cleopatra leverages the generation processes supplied by her users to actually build their projects.

We now describe the build process enacted by cleopatra, which allows us to describe how a typical generation process has to be structured in order to work.

1 Overview

As quickly mentioned, cleopatra is designed as an overlay on an existing build system, not a build system implemented from scratch. Currently, only make is supported, it would be pretty easy to generalize our approach to others (e.g., ninja comes to mind).

When users call cleopatra within the source tree of a given project, it searches for a cleopatra.toml file. If it finds one, it set up its workspace, then starts a four stage build process. These four distinct, sequential phases are init, prebuild, build, and postbuild; they are implemented as Makefile recipes in ${CLEOPATRA_DIRECTORY/boot.mk}, whose content is the following:

ARTIFACTS := build.log
CONFIGURE := .cleopatra

PROCS := $(wildcard ${CLEOPATRA_GENERATION_PROCESSES}/*.org)

init :
        @cleopatra echo "Exporting" "generation process"
        @cleopatra-run-elisp "${CLEOPATRA_DIRECTORY}/emacs.d/cleopatra-gen-proc.el" \
              >build.log 2>&1
        @cleopatra-gen-deps ${PROCS}

-include ${CLEOPATRA_DIRECTORY}/deps.mk

prebuild :
build : prebuild
postbuild : build

postbuild :
        @cleopatra echo "Updating" ".gitignore"
        @cleopatra-update-gitignore $(sort ${CONFIGURE} ${ARTIFACTS})
        @rm ${CLEOPATRA_DIRECTORY}/deps.mk

clean :
        @cleopatra echo Cleaning "Build artifacts"
        @rm -rf ${ARTIFACTS}
        @rm -rf .cleopatra/emacs.d/cache

cleanall : clean
        rm -rf ${CONFIGURE}

.PHONY : init prebuild build postbuild clean cleanall

The init phase is reserved to cleopatra itself: it uses it to tangles the generation processes supplied by the user. These generation processes then populates the prebuild, build and postbuild phases with their own recipes. More precisely, a file proc.org within the ${CLEOPATRA_GENERATION_PROCESSES} directory defined a generation process called proc. proc can defined three recipes: proc-prebuild, proc-build, and proc-postbuild, whose dependencies are the following:

In practice, the build process of cleopatra consists in two make invocation: make init, then make postbuild. By transitivity, postbuild pulls build, which pulls prebuild, according to the following Makefile rules.

prebuild :
build : prebuild
postbuild : build

We then introduce two variables to list the output of the generation processes, with two purposes in mind: keeping the .gitignore up-to-date automatically, and providing rules to remove them.

ARTIFACTS: Short-term artifacts which can be removed frequently without too much hassle. They will be removed by make clean.
CONFIGURE: Long-term artifacts whose generation can be time consuming. They will only be removed by make cleanall.

ARTIFACTS := build.log
CONFIGURE := .cleopatra

clean :
        @cleopatra echo Cleaning "Build artifacts"
        @rm -rf ${ARTIFACTS}
        @rm -rf .cleopatra/emacs.d/cache

cleanall : clean
        rm -rf ${CONFIGURE}

2 The `init` phase

The goal of the init phase is twofold. Firstly, cleopatra tangles the generation processes present in the CLEOPATRA_GENERATION_PROCESSES directory at that time. Secondly, it generates several files whose purpose is to encode the dependency graph introduced in the previous section.

We first recall the init phase is implemented in the ${CLEOPATRA_DIRECTORY}/boot.mk as follows.

PROCS := $(wildcard ${CLEOPATRA_GENERATION_PROCESSES}/*.org)

init :
        @cleopatra echo "Exporting" "generation process"
        @cleopatra-run-elisp "${CLEOPATRA_DIRECTORY}/emacs.d/cleopatra-gen-proc.el" \
              >build.log 2>&1
        @cleopatra-gen-deps ${PROCS}

That is, the init phase relies on two auxiliary scripts.

2.1 Tangling Generation Processes

cleopatra-gen-proc.el is implemented in Emacs script, and basically consists in using org-publish to tangle the generation processes. We use a dedicated publishing function we call gen-processes-tangle-publish.

gen-processes-tangle-publish proceeds by calling cleopatra:tangle-publish to tangle a file, and processes its output (that is, the list of file produces by Org to tangle filename) to generates a dependency file for the generation process. To generate a file in Emacs lisp, we use the with-temp-buffer to create a new buffer which we modify using insert, and save using write-file.

(defun gen-processes-tangle-publish (conf filename pub-dir)
  (let ((tangled (cleopatra:tangle-publish conf filename pub-dir))
        (proc (file-name-sans-extension (file-name-nondirectory  filename))))
    (with-temp-buffer
      (insert
       (format "include %s.mk\n" proc)
       (format "CONFIGURE += %s\n" (mapconcat 'identity tangled " "))
       (format "prebuild : %s-prebuild\nbuild : %s-build\npostbuild : %s-postbuild\n"
               proc proc proc)
       (format "%s-build : %s-prebuild\n%s-postbuild : %s-build\n"
               proc proc proc proc)
       (format ".PHONY : %s-prebuild %s-build %s-postbuild\n"
               proc proc proc proc))
      (write-file (format "%s/%s.deps.mk" (getenv "CLEOPATRA_DIRECTORY") proc)))))

For a generation process proc, gen-processes-tangle-publish will generates a file ${CLEOPATRA_DIRECTORY}/proc.deps.mk which contains

include proc.mk
CONFIGURE += <list of files produced when tangling proc.org>
prebuild : proc-prebuild
build : proc-build
postbuild : proc-postbuild
proc-build : proc-prebuild
proc-postbuild : proc-build
.PHONY : proc-prebuild proc-build proc-postbuild

In addition to defining gen-processes-tangle-publish, we configure Org and Babel by

Using cleopatra:configure
Adding shell to the list of language that can be evaluated by Babel
Setting the org-publish-project-alist to tangle the generation processes in ${CLEOPATRA_GENERATION_PROCESSES} to the root directory using the gen-processes-tangle-publish function.

(cleopatra:configure)

(org-babel-do-load-languages
 'org-babel-load-languages
 '((shell . t)))

(setq org-publish-project-alist
      `(("cleopatra-gen-proc"
         :base-directory ,(getenv "CLEOPATRA_GENERATION_PROCESSES")
         :publishing-directory "."
         :publishing-function gen-processes-tangle-publish)))

Beyond this configuration phases, the script solely consists in org-publish-all, which processes the org-publish-project-alist.

(org-publish-all)

2.2 Generating the Dependency Graph

In the previous section, we have detailed how cleopatra a ${CLEOPATRA_DIRECTORY}/${proc}.deps.mk for each generation process in the ${CLEOPATRA_GENERATION_PROCESSES} directory.

As-is, these files are not used by Makefile. To address this, cleopatra systematically generates ${CLEOPATRA_DIRECTORY}/deps.mk, a Makefile whose only purpose is to include each ${proc}.deps.mk file.

This file is generated by cleopatra-gen-deps, a small Bash script whose only interesting point is to use basename to remove the extension of the generation processes.

out="${CLEOPATRA_DIRECTORY}/deps.mk"

rm -f "${out}"
touch "${out}"

for proc in "$@"; do
    proc_name=$(basename ${proc} ".org")
    echo "include \${CLEOPATRA_DIRECTORY}/${proc_name}.deps.mk" >> "${out}"
done

cleopatra generates the deps.mk file at the beginning of each build, to ensure it is consistent with the content of the ${CLEOPATRA_GENERATION_PROCESSES} directory. However, it is more than likely that this file does not exists the first time cleopatra invokes make Therefore, we include it using -include and not include, so that make knows this file is optional (and therefore does not fail if it is missing).

-include ${CLEOPATRA_DIRECTORY}/deps.mk

This is the main reason why the build process of cleopatra consists in invoking make init (to generate, among other thing, deps.mk), then make postbuild (to leverage it now that it exists).

3 The `postbuild` phase

Contrary to prebuild and build, which are left empty, cleopatra makes use of the postbuild phase itself.

postbuild :
        @cleopatra echo "Updating" ".gitignore"
        @cleopatra-update-gitignore $(sort ${CONFIGURE} ${ARTIFACTS})
        @rm ${CLEOPATRA_DIRECTORY}/deps.mk

cleopatra leverages the postbuild phase to delete the deps.mk file previously generated during the init phase. We do that to prevent the following scenario from happening: one of the generation processes tangles an invalid Makefile (e.g., it contains spaces in place of tabs), which means make exits without trying to achieve anything… preventing a fixed generation process to be ever tangled. Since a new deps.mk file is generated anyway, keeping it between two builds does not make any sense in any case.

Besides, it keep the .gitignore file up-to-date, based on the ARTIFACTS and CONFIGURE variables (sorted with the sort function) which have been populated by the generation processes. To that end, we implement the cleopata-update-gitignore script, which leverages some neat feature of sed I will never be able to write myself ever again.

BEGIN_MARKER="# begin generated files"
END_MARKER="# end generated files"

# remove the previous list of generated files to ignore
sed -i -e "/${BEGIN_MARKER}/,/${END_MARKER}/d" .gitignore
# remove trailing empty lines
sed -i -e :a -e '/^\n*$/{$d;N;};/\n$/ba' .gitignore

# output the list of files to ignore
echo "" >> .gitignore
echo ${BEGIN_MARKER} >> .gitignore
for f in $@; do
    echo "${f}" >> .gitignore
done
echo ${END_MARKER} >> .gitignore

The cleopatra Build Process

Table of Contents

1 Overview

2 The init phase

2.1 Tangling Generation Processes

2.2 Generating the Dependency Graph

3 The postbuild phase

2 The `init` phase

3 The `postbuild` phase