Git and Make: not just for code

Don Marti


basic stuff

not much Ledger

off message

probably broken stuff







(many more)

pandoc --template templates/default.html \
--to=html5 --smart --standalone --toc 

Other tools





Working on the command line

Remember command syntax?

Remember sequences of commands?

Tired of rowing through stuff?

Use the same spelling dictionary for both?

Professional Marketing White Paper

Gritty Noir Novel

"executable notebook" for commands needed to build stuff


Basic rules

Makefile has tabs, never spaces

$$ is $

# Comments

\ to break long commands (passed to shell)

Structure of a rule

# quiet comment
target : prerequisite
    # noisy comment

This presentation's Makefile

index.html : template.kittenslides
    pandoc --section-divs -t html5 \
    --template template.kittenslides -s -o $@ $< 

clean :
    rm -f index.html

.PHONY : clean


Patterns with %




%.html : template.kittenslides
    pandoc --section-divs -t html5 \
    --template template.kittenslides -s -o $@ $< 

clean :
    rm -f $(PRESENTATIONS)

.PHONY : clean

Special target


Empty files as targets

deploy :
    # deploy your site here...
    touch deploy

print :
    enscript # obscure enscript arguments here
    touch print

That's great when you're working with a known list of files...

But what about NaNoWriMo?

"shell" and "foreach"

CHAPTERS=$(shell find . -name '??-*' | sort) : $(CHAPTERS) 
    rm -f $@
    $(foreach c,$(CHAPTERS),cat $(c) >> $@; \
    echo "<hr>" >> $@;)
    echo "The End" >> $@

Fix our aspell problem

# Create the project-specific dictionary
mynovel.pws : 
        echo "personal_ws-1.1 en 0" > $@

# Spell check using the project-specific 
# dictionary
spell : $(CHAPTERS) mynovel.pws
        $(foreach c,$(CHAPTERS), aspell \
        --personal=./mynovel.pws \
        check $(c);)

NaNoWriMo is all about the count

count : $(CHAPTERS)
        wc -w $(CHAPTERS)


"git gets easier once you get the basic idea that branches are homeomorphic endofunctors mapping submanifolds of a Hilbert space." —Isaac Wolkerstorfer

picture of kitten
picture of kitten

Git tutorials are pure evil

Map and compass
Map and compass

Git data
Git data

Read this
Read this

Scratch repositories are your friend





Push to deploy or make deploy?

Hooks are not centrally managed

and people are lazy and break things


Use make

hooks : .git/hooks/pre-commit

.git/hooks/% : Makefile
        echo "#!/bin/sh" > $@
        echo "make `basename $@`" >> $@
        chmod 755 $@

pre-commit :
        git diff-index --check HEAD

Use make

hooks : .git/hooks/pre-commit

.git/hooks/% : Makefile
        echo "#!/bin/sh" > $@
        echo "make `basename $@`" >> $@
        chmod 755 $@

pre-commit : spell

Check that a ledger balances before committing

pre-commit : company.dat
    ledger -f $< bal > \

Smudge and clean

Work in MS-Word or Markdown?

smudge and clean scripts must run as filters: read from stdin, write to stdout

$ head secret-document.docx 
Ideas for a SCaLE talk on Git and Make {.western}

Two of the most common programming tools are way too useful to be left
to the programmers. We'll cover how to use the Git version control
system and the Make utility to get more accurate and faster results on a
variety of common tasks.

“But what about the people who use MS-Word or Libreoffice? Don't they
want to stick with their tools of choice?” you might ask.

In this case what's actually checked into Git is real .DOCX

# smudge-docx script

TMPDIR=$(mktemp -d)
trap "rm -rf $TMPDIR" EXIT

cat > $TMPDIR/index.docx
unoconv -f html --output $TMPDIR/index.html \
pandoc --to=markdown $TMPDIR/index.html

# clean-docx script

pandoc --to=docx --output=/proc/self/fd/1 

git config filter.docx.smudge smudge-docx

git config filter.docx.clean clean-docx

$ cat .gitattributes
*.docx filter=docx

Like all Git configuration this is repo-specific, so you can mangle DOCX in one repository and leave it alone in another one.

Or you could make the Markdown the "real" version and use smudge to put a temporary .DOCX in Dropbox...

Don't keep typing "make"


make $* # Pass all arguments to make
while true; do
    fl=$(prereq_files $*)
    ev=$(inotifywait --quiet --format %e $fl)
    if [ "xOPEN" != "x$ev" ]; then
        sleep 1
        make $*

make_prereqs() {
    # Make "make" figure out what files
    # it's interested in.
    echo "Makefile"
    make -dnr $* | tr ' ' '\n' | \
        grep ".*'.$" | grep -o '\w.*\b'

prereq_files() {
    # prerequisites mentioned in a Makefile
    # that are extant files
    for f in `make_prereqs $* | sort -u`; do
        [ -e $f ] && echo -n "$f ";

Replicating Git is remarkably easy*

Raft by Diego Ongaro and John Osterhout
Raft by Diego Ongaro and John Osterhout

Really big* Git deployments


update hook

post-update hook

Update hook: before moving the ref

Atomic test and set in etcd

Post-update hook

Flood the update out to other git repositories

Since etcd is already current, they accept it


Conflicting pushes: one fails, the other succeeds

Out of date repo: push to it fails, then it tries to pull from other members of repogroup

Git pushes already fail, so users can just deal

Meteor strike 1: you get out with your laptop

Meteor strike 2: laptop and home Git server destroyed right after update but before all objects copied


Use Makefiles to remember complex commands.

Never believe a Git tutorial.

People are lazy and like to break things.

Smudge and clean can help with format issues.

You don't have to keep typing "make" if you don't want to.

Git is almost meteor-proof because Computer Science.

Further reading

GNU Make manual

Butterick's Practical Typography

Pandoc User's Guide

Raft Consensus Algorithm

piehole: simple highly fault-tolerant Git replication

Creative Commons photos

Kitten: Christian Holmér

Map and compass: Audriusa for Wikimedia Commons

collision d'une comète sur la terre: NASA

Git data illustration from Pro Git (MIT license)

Any questions?

Follow @dmarti on Twitter, or contact via email.