The cleopatra Build Process
Table of Contents
In this chapter, we aim to explain how cleopatra
works. And since
cleopatra
is far from being the single available build tool (it is merely
an overlay on an existing one, actually!), we avoid ambiguity by defining a
small, yet consistent terminology throughout this book.
- Root
- The directory which contains the
cleopatra.toml
file, and supposedly the root of the source tree of the software thatcleopatra
is expected to build. - Workspace
- The execution environment set up by
cleopara
prior to starting a build. In particular, this workspace consists in- The current working directory is the root of the project
- The
ROOT
environment variable has been set with the root of the project - The
CLEOPATRA_DIRECTORY
environment variable has been set with${ROOT}/.cleopatra
- The
CLEOPATRA_GENERATION_PROCESSES
environment variable has been set with the value of thegeneration_processes
entry incleopatra.toml
- Generation Process
- An Org document describing a set of build recipes, along with the auxiliary files used by these rules.
- Build Process
- How
cleopatra
leverages the generation processes supplied by her users to actually build their projects.
We now describe the build process enacted by cleopatra
, which allows us to
describe how a typical generation process has to be structured in order to
work.
1 Overview
As quickly mentioned, cleopatra
is designed as an overlay on an
existing build system, not a build system implemented from scratch. Currently,
only make
is supported, it would be pretty easy to generalize our approach to
others (e.g., ninja
comes to mind).
When users call cleopatra
within the source tree of a given project, it
searches for a cleopatra.toml
file. If it finds one, it set up its workspace,
then starts a four stage build process. These four distinct, sequential phases
are init
, prebuild
, build
, and postbuild
; they are implemented as
Makefile recipes in ${CLEOPATRA_DIRECTORY/boot.mk}
, whose content is the
following:
ARTIFACTS := build.log CONFIGURE := .cleopatra PROCS := $(wildcard ${CLEOPATRA_GENERATION_PROCESSES}/*.org) init : @cleopatra echo "Exporting" "generation process" @cleopatra-run-elisp "${CLEOPATRA_DIRECTORY}/emacs.d/cleopatra-gen-proc.el" \ >build.log 2>&1 @cleopatra-gen-deps ${PROCS} -include ${CLEOPATRA_DIRECTORY}/deps.mk prebuild : build : prebuild postbuild : build postbuild : @cleopatra echo "Updating" ".gitignore" @cleopatra-update-gitignore $(sort ${CONFIGURE} ${ARTIFACTS}) @rm ${CLEOPATRA_DIRECTORY}/deps.mk clean : @cleopatra echo Cleaning "Build artifacts" @rm -rf ${ARTIFACTS} @rm -rf .cleopatra/emacs.d/cache cleanall : clean rm -rf ${CONFIGURE} .PHONY : init prebuild build postbuild clean cleanall
The init
phase is reserved
to cleopatra
itself: it uses it to tangles the generation processes supplied
by the user. These generation processes then populates the prebuild
, build
and postbuild
phases with their own recipes. More precisely, a file
proc.org
within the ${CLEOPATRA_GENERATION_PROCESSES}
directory defined a
generation process called proc
. proc
can defined three recipes:
proc-prebuild
, proc-build
, and proc-postbuild
, whose dependencies are the
following:
In practice, the build process of cleopatra
consists in two make
invocation: make init
, then make postbuild
. By transitivity, postbuild
pulls build
, which pulls prebuild
, according to the following Makefile
rules.
prebuild : build : prebuild postbuild : build
We then introduce two variables to list the output of the generation processes,
with two purposes in mind: keeping the .gitignore
up-to-date automatically,
and providing rules to remove them.
ARTIFACTS
- Short-term artifacts which can be removed frequently without too much
hassle. They will be removed by
make clean
. CONFIGURE
- Long-term artifacts whose generation can be time consuming. They will only be
removed by
make cleanall
.
ARTIFACTS := build.log CONFIGURE := .cleopatra
clean : @cleopatra echo Cleaning "Build artifacts" @rm -rf ${ARTIFACTS} @rm -rf .cleopatra/emacs.d/cache cleanall : clean rm -rf ${CONFIGURE}
2 The init
phase
The goal of the init
phase is twofold. Firstly, cleopatra
tangles the
generation processes present in the CLEOPATRA_GENERATION_PROCESSES
directory
at that time. Secondly, it generates several files whose purpose is to encode
the dependency graph introduced in the previous section.
We first recall the init
phase is implemented in the
${CLEOPATRA_DIRECTORY}/boot.mk
as follows.
PROCS := $(wildcard ${CLEOPATRA_GENERATION_PROCESSES}/*.org) init : @cleopatra echo "Exporting" "generation process" @cleopatra-run-elisp "${CLEOPATRA_DIRECTORY}/emacs.d/cleopatra-gen-proc.el" \ >build.log 2>&1 @cleopatra-gen-deps ${PROCS}
That is, the init
phase relies on two auxiliary scripts.
2.1 Tangling Generation Processes
cleopatra-gen-proc.el
is implemented in Emacs script, and basically consists
in using org-publish
to tangle
the generation processes. We use a dedicated publishing function we call
gen-processes-tangle-publish
.
gen-processes-tangle-publish
proceeds by calling
cleopatra:tangle-publish
to tangle a file, and
processes its output (that is, the list of file produces by Org to tangle
filename
) to generates a dependency file for the generation process. To
generate a file in Emacs lisp, we use the with-temp-buffer
to create a new
buffer which we modify using insert
, and save using write-file
.
(defun gen-processes-tangle-publish (conf filename pub-dir) (let ((tangled (cleopatra:tangle-publish conf filename pub-dir)) (proc (file-name-sans-extension (file-name-nondirectory filename)))) (with-temp-buffer (insert (format "include %s.mk\n" proc) (format "CONFIGURE += %s\n" (mapconcat 'identity tangled " ")) (format "prebuild : %s-prebuild\nbuild : %s-build\npostbuild : %s-postbuild\n" proc proc proc) (format "%s-build : %s-prebuild\n%s-postbuild : %s-build\n" proc proc proc proc) (format ".PHONY : %s-prebuild %s-build %s-postbuild\n" proc proc proc proc)) (write-file (format "%s/%s.deps.mk" (getenv "CLEOPATRA_DIRECTORY") proc)))))
For a generation process proc
, gen-processes-tangle-publish
will generates
a file ${CLEOPATRA_DIRECTORY}/proc.deps.mk
which contains
include proc.mk CONFIGURE += <list of files produced when tangling proc.org> prebuild : proc-prebuild build : proc-build postbuild : proc-postbuild proc-build : proc-prebuild proc-postbuild : proc-build .PHONY : proc-prebuild proc-build proc-postbuild
In addition to defining gen-processes-tangle-publish
, we configure Org and
Babel by
- Using
cleopatra:configure
- Adding
shell
to the list of language that can be evaluated by Babel - Setting the
org-publish-project-alist
to tangle the generation processes in${CLEOPATRA_GENERATION_PROCESSES}
to the root directory using thegen-processes-tangle-publish
function.
(cleopatra:configure) (org-babel-do-load-languages 'org-babel-load-languages '((shell . t))) (setq org-publish-project-alist `(("cleopatra-gen-proc" :base-directory ,(getenv "CLEOPATRA_GENERATION_PROCESSES") :publishing-directory "." :publishing-function gen-processes-tangle-publish)))
Beyond this configuration phases, the script solely consists in
org-publish-all
, which processes the org-publish-project-alist
.
(org-publish-all)
2.2 Generating the Dependency Graph
In the previous section, we have detailed how cleopatra
a
${CLEOPATRA_DIRECTORY}/${proc}.deps.mk
for each generation process in the
${CLEOPATRA_GENERATION_PROCESSES}
directory.
As-is, these files are not used by Makefile. To address this, cleopatra
systematically generates ${CLEOPATRA_DIRECTORY}/deps.mk
, a Makefile whose only
purpose is to include each ${proc}.deps.mk
file.
This file is generated by cleopatra-gen-deps
, a small Bash script whose only
interesting point is to use basename
to remove the extension of the generation
processes.
out="${CLEOPATRA_DIRECTORY}/deps.mk" rm -f "${out}" touch "${out}" for proc in "$@"; do proc_name=$(basename ${proc} ".org") echo "include \${CLEOPATRA_DIRECTORY}/${proc_name}.deps.mk" >> "${out}" done
cleopatra
generates the deps.mk
file at the beginning of each build, to
ensure it is consistent with the content of the
${CLEOPATRA_GENERATION_PROCESSES}
directory. However, it is more than likely
that this file does not exists the first time cleopatra
invokes make
Therefore, we include it using -include
and not include
, so that make
knows this file is optional (and therefore does not fail if it is missing).
-include ${CLEOPATRA_DIRECTORY}/deps.mk
This is the main reason why the build process of cleopatra
consists in
invoking make init
(to generate, among other thing, deps.mk
), then make
postbuild
(to leverage it now that it exists).
3 The postbuild
phase
Contrary to prebuild
and build
, which are left empty, cleopatra
makes
use of the postbuild
phase itself.
postbuild : @cleopatra echo "Updating" ".gitignore" @cleopatra-update-gitignore $(sort ${CONFIGURE} ${ARTIFACTS}) @rm ${CLEOPATRA_DIRECTORY}/deps.mk
cleopatra
leverages the postbuild
phase to delete the deps.mk
file
previously generated during the init
phase. We do that to prevent the
following scenario from happening: one of the generation processes tangles an
invalid Makefile (e.g., it contains spaces in place of tabs), which means
make
exits without trying to achieve anything… preventing a fixed generation
process to be ever tangled. Since a new deps.mk
file is generated anyway,
keeping it between two builds does not make any sense in any case.
Besides, it keep the .gitignore
file up-to-date, based on the
ARTIFACTS
and CONFIGURE
variables (sorted with the sort
function) which have been populated by the generation processes. To
that end, we implement the cleopata-update-gitignore
script, which
leverages some neat feature of sed
I will never be able to write
myself ever again.
BEGIN_MARKER="# begin generated files" END_MARKER="# end generated files" # remove the previous list of generated files to ignore sed -i -e "/${BEGIN_MARKER}/,/${END_MARKER}/d" .gitignore # remove trailing empty lines sed -i -e :a -e '/^\n*$/{$d;N;};/\n$/ba' .gitignore # output the list of files to ignore echo "" >> .gitignore echo ${BEGIN_MARKER} >> .gitignore for f in $@; do echo "${f}" >> .gitignore done echo ${END_MARKER} >> .gitignore