Big shout-outs to @reillywood, who managed cut the CI times for @nu_shell in half!

If you do @rustlang and see your CI build times creeping up, you should give his fixes a look. Smarter test ordering and build caching makes a huge difference.

— JT (@jntrnr) April 30, 2022

I recently spent a few days tuning Nushell’s GitHub Actions CI pipelines and it paid off: CI used to take about 30 minutes, and now it’s closer to 10. This is not pleasant or glamorous work, but it has a big payoff; every Nu change going forward will spend a lot less time waiting for essential feedback. Here’s how you can do the same.

Use rust-cache

Seriously, it’s really good! GitHub build runners are slow. But GitHub gives every repo 10GB of cache space, and rust-cache takes advantage of that. It caches temporary files for your build dependencies across CI runs, so if you have a lot of dependencies you’ll likely see a big performance boost.

One gotcha to be aware of: GitHub Actions has slightly unintuitive behavior across PRs. PR X is unable to see cache data from PR Y, but they can both see cache data from the base branch (usually main or master). This makes sense from an isolation perspective, but it’s not especially well-documented; I ended up adding an extra CI trigger on main just to fill caches properly.

Split your build and test jobs

Previously we were running cargo build then cargo test in a single job. This was suboptimal for a few reasons:

1. cargo test often needed to recompile crates that had just been built for cargo build. #[cfg(test)] is the most likely culprit here; it makes sense that build output might be different in “test mode”. This has implications for caching too!
2. It’s faster to run build and test in parallel; GitHub gives us 20 build runners for free, and we might as well use them.

Run Clippy after cargo build

Previously we were running Clippy before cargo build. Just switching their order shaved about 5 minutes off every test run! It seems like Clippy can reuse build artifacts from cargo build, but not vice versa.

Use cargo nextest

cargo nextest is “a next-generation test runner for Rust projects.” It’s dead simple to install in CI, and it’s often faster than cargo test. We didn’t see a huge benefit from this (maybe 30-40s faster?), but that’s because our CI time is dominated by compilation; YMMV depending on your code base and test suite.

Conclusion

If you’d like to see the actual changes, they’re all here. Like anything GitHub Actions, this took a lot of tries to get right; those 5 PRs are just the tip of the iceberg, there were a lot more experimental changes in my private fork. I’m hopeful that someday we’ll be able to stop programming in YAML files, but we’re not there yet!

Bret Victor’s talk Inventing on Principle (video, transcript) changed the way I think about computing in 2019. Inventing on Principle is partly about Bret’s guiding principle:

“Creators need an immediate connection to what they create. And what I mean by that is when you’re making something, if you make a change, or you make a decision, you need to see the effect of that immediately.”

The Edit-Compile-Run Cycle

Although Bret doesn’t use the term, programmers are deeply familiar with his principle. We’ve all worked with toolchains that introduce significant delay before you can “see” the results of a change, and we know they’re painful. Everyone wants a short edit-compile-run cycle.

But until IoP, I’d assumed that slow cycles wouldn’t materially change the output – you’d eventually get to the same place. This was wrong. I also didn’t appreciate the very small time scales involved; a 5 second delay used to seem trivial to me, but it’s still meaningfully different from a response time measured in milliseconds.

Through some very impressive custom tools, Bret shows how immediate feedback enables exploration, which then gives birth to ideas which would otherwise never see the light of day. This was an epiphany for me. Since IoP I’ve constantly been looking for better ways to code, and re-evaluating my existing processes for shorter feedback cycles. The results:

Rust

My typical Rust development workflow goes something like this:

1. Write a small function that does roughly what I want
2. Write a small unit test inline to exercise the function (even if it’s a private function)
3. Iterate using cargo test until the function is correct
4. Later, “productionize” the tests if necessary

Rust’s native support for inline unit tests helps a lot here, and the excellent type system catches a lot of issues before I even run cargo test. On the other hand, Rust’s compiler is notoriously slow and that extends to IDE tooling that depends on the Rust Language Server. I’m looking forward to Cranelift for faster debug builds.

Rust has a really powerful macro system. You can use it to do great things safely… or you can have fun with it and quickly prototype coding productivity features. I chose the latter.

todo-macro is a procedural Rust macro that stops compiling after a user-specified deadline. It’s like TODO comments, but with teeth. It’s probably best explained with an example:

Using todo-macro

It’s January 1, 2020. I’m working on some Rust code that compiles, but it’s not quite ready to ship.

I want to take a break, but I know myself – I’ll probably forget about the deficiency. I could add a TODO comment, but that depends on me actively searching for TODO comments next time I open the project.

To save me from myself, I add a quick todo macro with a deadline of January 2 (in ISO 8601 format):

// Implement the timeout handling
todo!("2020-01-02")