I’ve been spending some time on CI improvements at work recently, mostly around cutting down how long things take.

As I looked to see if anything about our overall process could be improved, a couple things bothered me:

Why, if we were using a pretty standard GitHub Actions setup, were there indications that modules were being downloaded as part of every run? Shouldn’t that all be cached?

Why did it seem like there was always a delay before tests actually started running? Shouldn’t the first few packages’ fast tests complete quickly?

At work, we’ve just about completed a migration from the lib/pq Go PostgreSQL driver to pgx.

We use the pgx database/sql compatibility layer so our code doesn’t need to care which driver is used, for the most part.

As we neared the end of the migration, one unsolved issue remained: using COPY inside transactions.

We use COPY in a few places to bulk load data. With lib/pq, that looks something like this:

// db is a *database/sql.DB
tx, err := db.Begin()
if err != nil { // ...
defer tx.Rollback()

stmt, err := tx.Prepare(pq.CopyIn("users", "name", "age"))
if err != nil { // ...

for _, user := range users {
	_, err = stmt.Exec(user.Name, int64(user.Age))
	if err != nil { // ...
}

_, err = stmt.Exec()
if err != nil { // ...

err = stmt.Close()
if err != nil { // ...

err = tx.Commit()
if err != nil { // ...

pq.CopyIn("users", "name", "age") produces a statement that looks like:

Testing code that involves time or concurrency can be a struggle. It often leads to hard-to-debug flakes in CI or long-running tests.

Go 1.24 is scheduled to be released in February and the release freeze has begun.

It’s set to include an experimental testing/synctest package designed to make testing code that involves time or concurrency precise and fast.

I’m pretty excited about it!

Time testing trouble

Suppose you have a test that looks like this:

I’ve been having a lot of call trouble at home lately using Eastlink and an iPhone 13 mini.

Even though I have all the signal bars:

Received and placed calls would fail right away or audio would cut in and out for both sides.

I’ve been working with Eastlink support but no breakthroughs yet.

Where possible, I like to build systems around convergence.

These systems generally look something like:

1. Tick on some regular interval (10 minutes, 1 hour, etc)
2. Update the view of the world (discover new data, observe a value, etc)
3. Act on the updated view, possibly comparing to the previous view (stop some servers, process newly discovered data, etc)

For example, an auto scaling group would continually check instance health, load, etc, and converge on the desired or optimal number of servers by starting new ones or stopping existing ones.

It seems like Twitter may not be around much longer. Even if does stay around, it’s not likely to be a place I want to be anymore.

Like many folks, I decided to dip into the Fediverse. Using Mastodon seemed like a good first step. I set up an account on mastodon.social but quickly realized it was overwhelmed. There were other servers to join but I wanted to run my own.

My goals:

Summary

Starting in Go 1.20¹, DNS lookups when running on macOS will be done via the system instead of via Go’s built-in resolver. That’s even when cgo is not available, such as when cross-compiling from Linux.

That should reduce surprises for people using tools cross-compiled for macOS from Linux, such as CLIs like terraform and kubectl. It should also make it easier for developers of those tools as building on macOS may no longer be necessary.

Halifax has a systemic road safety problem. One way it manifests is people driving too fast, especially on residential streets.

Recently, Halifax Regional Council mostly approved their 2021 capital budget. It includes $1 million for traffic calming and the final budget may include $1 million more. Read more about that here or here.

The city publishes a ranked list of approved requests. Not all requests are approved due to rules that contribute to the systemic problem. Even so, as of the last update on November 27, 2020, there are 282 approved requests.

As part of Heroku Private Spaces we attach extra Elastic Network Interfaces to instances, and we do that in Go with aws-sdk-go.

The process isn’t the most idempotent and we recently discovered we were occasionally leaking ENIs. This could happen if we created an ENI and then encountered an error later in the process, such as when attaching it to an instance. I set out this morning to find a better way.

When an ENI is created, it can be assigned a description. Previously we used descriptions only for the ENI’s function (eg, “nat”) and not anything specific to the instance the ENI was intended for. After seeing that listing ENIs allows filtering on the description, I decided to make that the foundation of the new approach.

One of my favorite localized Go refactorings is reducing nesting by using return as early as possible. Take this example, based on a recently-refactored function at Heroku:

func example() error {
  if err := start(); err != nil {
    existing, err := fetchExisting()
    if err != nil {
      return err
    }

    if existing.IsWorking() {
      return nil
    } else {
      return errors.New("found existing but it's not working")
    }
  }

  return nil
}

This can be hard to follow due to nesting inside if ...; err != nil and use of else. I prefer to return as early as possible and avoid else. Applying that, it now looks something like this: