gravatar image

Hi! I'm Daniel.

I like writing, tweeting, sometimes give talks, and occasionally write code.

May 31, 2016

Make and Go for Fun and Profit

I've been somewhat interested in Go for quite a while now. It's gotten to the point where it has replaced Ruby for me in those places where I write command line utilities which are too involved for them to make sense to be a shell script. I don't have too many opinions about the language itself, but I like the static type system and that it's a compiled language. And to be honest, the build system and how to utilize it have been the most interesting bits for me so far. One thing I especially like is the fact that go provides a bunch of tooling to do different things but how you tie them together is up to you. So this gives rise to some fun use cases for a nice Makefile.

The Basics

Every project I start gets this Makefile with some basic setups and variable definitions that I always want.

export GO15VENDOREXPERIMENT = 1

# variable definitions
NAME := coolthings
DESC := a nice toolkit of helpful things
PREFIX ?= usr/local
VERSION := $(shell git describe --tags --always --dirty)
GOVERSION := $(shell go version)
BUILDTIME := $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
BUILDDATE := $(shell date -u +"%B %d, %Y")
BUILDER := $(shell echo "`git config user.name` <`git config user.email`>")
PKG_RELEASE ?= 1
PROJECT_URL := "https://github.com/mrtazz/$(NAME)"
LDFLAGS := -X 'main.version=$(VERSION)' \
           -X 'main.buildTime=$(BUILDTIME)' \
           -X 'main.builder=$(BUILDER)' \
           -X 'main.goversion=$(GOVERSION)'

For the most part this just defines a whole bunch of meta data that gets compiled into the binaries via linker flags. This is a pattern I have seen in a lot of Go projects and I really like that this is somewhat of a standard thing to do. Especially with the static nature of Go binaries, the more helpful information you can compile into the binary the better it is when you have to figure out where a binary comes from.

I also always have a handful of tasks defined that are helpful for running tests and such, especially to have a uniform and documented way how they are run locally and on CI.

# development tasks
test:
    go test $$(go list ./... | grep -v /vendor/ | grep -v /cmd/)

PACKAGES := $(shell find ./* -type d | grep -v vendor)

coverage:
    @echo "mode: set" > cover.out
    @for package in $(PACKAGES); do \
        if ls $${package}/*.go &> /dev/null; then  \
        go test -coverprofile=$${package}/profile.out $${package}; fi; \
        if test -f $${package}/profile.out; then \
        cat $${package}/profile.out | grep -v "mode: set" >> cover.out; fi; \
    done
    @-go tool cover -html=cover.out -o cover.html

benchmark:
    @echo "Running tests..."
    @go test -bench=. $$(go list ./... | grep -v /vendor/ | grep -v /cmd/)

These make heavy use of go list to determine existing packages to run tests for. The rules also exclude the vendor folder as I don't want to run those tests and the cmd folder which I will describe more in the next section.

Structure for multiple binaries

Go has this defacto standard of how to structure code if your build produces multiple executables. Since your main entry point in the app is always the main package and there can only be one per directory (which is also true for any other package btw) you need to separate different executables by directory. The pattern here is basically to have a cmd folder that contains subfolders for each executable which in turn just contain a main.go file. This is a pretty nice pattern, once you get used to it and is a convention that lets you easily create make rules for building those executables via the make wildcarding support.

CMD_SOURCES := $(shell find cmd -name main.go)
TARGETS := $(patsubst cmd/%/main.go,%,$(CMD_SOURCES))

%: cmd/%/main.go
    go build -ldflags "$(LDFLAGS)" -o $@ $<

This piece just finds all main.go files under the cmd folder and creates targets from them located at the top level of the repo. Then there is a rule to build those targets via a rule that ties them back to the source file via wildcarding again and runs go build with the linker flags from before.

Of course it's good habit to provide man pages for your tools. So we can rig up a similar set of rules for building man pages for each executable:

MAN_SOURCES := $(shell find man -name "*.md")
MAN_TARGETS := $(patsubst man/man1/%.md,%,$(MAN_SOURCES))

%.1: man/man1/%.1.md
    sed "s/REPLACE_DATE/$(BUILDDATE)/" $< | pandoc -s -t man -o $@

all: $(TARGETS) $(MAN_TARGETS)
.DEFAULT_GOAL:=all

This lets us write man pages in markdown under the man/man1/folder named as ${cmd}.1.md and again uses wildcards in make to generate them top level via an implicit rule. I also added an all target there which is the default and builds all binaries and man pages.

Over time I've come to the conclusion that it's really a good practice to have your main.go files be as slim as possible. Ideally all they should be concerned with is flag parsing, calling a method from your library packages, and formatting and printing the output to the terminal. Any actual logic should live in library modules somewhere else in your repo. This maintains a good code layout to extend, makes sure code is reusable, and provides good conventions for testing.

Installation

So now that we have rules to build the binaries, we also want to be able to install them to the PREFIX we have defined at the top. Go comes with an install command already (go install) which will put binaries in your $GOPATH/bin but there is no need to have to rely on that. Plus on a multi user system you want to provide tools for everyone anyways. Also let's be real, go install is not a replacement for a real package manager. Just because go builds are fast and produce a static binary doesn't mean it's not a good idea to be able to build packages. Plus you want your man pages to be installed with your software as well of course. So let's write some generic install commands:

INSTALLED_TARGETS = $(addprefix $(PREFIX)/bin/, $(TARGETS))
INSTALLED_MAN_TARGETS = $(addprefix $(PREFIX)/share/man/man1/, $(MAN_TARGETS))

# install tasks
$(PREFIX)/bin/%: %
    install -d $$(dirname $@)
    install -m 755 $< $@

$(PREFIX)/share/man/man1/%: %
    install -d $$(dirname $@)
    install -m 644 $< $@

install: $(INSTALLED_TARGETS) $(INSTALLED_MAN_TARGETS)

local-install:
    $(MAKE) install PREFIX=usr/local

We're adding the PREFIX to all targets and man targets here to generate the paths to install. Then we write another implicit wildcarding rule that has the original targets as dependencies and performs install commands to put them into the prefix. This is a quick and easy way to have a generic make install target and also lets us easily add a local install target that we can use as a dependency for building packages later on.

Dependencies, Oh My!

If you've spent time with Go and make before, you will maybe have noticed a flaw in the building step of the Makefile so far. To revisit, we are building binaries from the source in the cmd folder with this implicit rule.

%: cmd/%/main.go
    go build -ldflags "$(LDFLAGS)" -o $@ $<

However this only tells make about the first level of direct dependencies for the binary to the cmd source. Chances are you are using library and vendored code in those. This means while go build technically knows about all dependencies, make doesn't. And it will refuse to rebuild the binaries if something other than the cmd source changes. This is annoying but fortunately also fixable. A simple fix would be to just not have dependencies in make for the executables and mark them as .PHONY so that they are always regarded out of date. This pushes all dependency resolution back to the go tool chain which is nice, but kinda defeats half of the purpose of a Makefile as it will just run all the commands all the time. To be clear, in practice this is a fine solution and the downsides are mostly academic with the speed of a usual go build.

However it's fun to figure out how to make things work and while we're here already, lets utilize make to its full extent and make it aware of all dependencies. The details for the make side of things I got from this awesome blogpost which gives a great overview over automatic dependency management in makefiles. So now all we need is a way to get a list of all dependencies for a go source file. And of course, go files to the rescue again! As it not only lets us print packages for passing to the test runner, but also can print out all dependencies of a file. And with its -f parameter it also supports basic templating for printing out the results. Utilizing that we only need to do a small amount of post processing to print it in make dependency format and we are good to go.

# source, dependency and build definitions
DEPDIR = .d
$(shell install -d $(DEPDIR))
MAKEDEPEND = echo "$@: $$(go list -f '{{ join .Deps "\n" }}' $< | awk '/github/ { gsub(/^github.com\/[a-z]*\/[a-z]*\//, ""); printf $$0"/*.go " }')" > $(DEPDIR)/$@.d

$(DEPDIR)/%.d: ;
.PRECIOUS: $(DEPDIR)/%.d

-include $(patsubst %,$(DEPDIR)/%.d,$(TARGETS))

%: cmd/%/main.go $(DEPDIR)/%.d
    $(MAKEDEPEND)
    go build -ldflags "$(LDFLAGS)" -o $@ $<

The makedepend command here grabs all dependencies that come from github (which was a good enough approximation for me to filter out the std lib), cuts off the project prefix and appends /*.go to each dependency. With the go rules of having a package per folder, this also is pretty accurate most of the time and only occasionally serves false positives to result in a rebuild. We then adapt the implicit build rule to require the dependency file as well but also rebuild it on each build. And BOOM our Makefile knows almost perfectly a out all source dependencies.

Packaging and Documentation

I always aim for providing packages and good documentation for my Go projects. But I've already written about those things more generally here, so if you're interested in the details of it, give that blog post a read. The important part is that the Makefile also holds the logic for building docs and packages, so they can be easily triggered from CI.

Cleanup

Since it's also always good to make it easy to clean up artifacts and generated intermediate and output files, all makefiles also get some clean up tasks.

# clean up tasks
clean-docs:
    rm -rf ./docs

clean-deps:
    rm -rf $(DEPDIR)

clean: clean-docs clean-deps
    rm -rf ./usr
    rm $(TARGETS)
    rm $(MAN_TARGETS)

.PHONY: all test rpm deb install local-install packages govendor coverage docs jekyll deploy-docs clean-docs clean-deps clean

Equipped with those Make tricks I've been having tons of fun building Go code. Some of that is surely more involved than it has to be and especially the dependency resolution stuff is very bonus round. But it's been super interesting to rig it up and I learned a lot of things about Make. And in the end that's what it's all about for me. (Besides having projects with a super nice to use structure :)