Big shout-outs to @reillywood, who managed cut the CI times for @nu_shell in half!

If you do @rustlang and see your CI build times creeping up, you should give his fixes a look. Smarter test ordering and build caching makes a huge difference.

— JT (@jntrnr) April 30, 2022

I recently spent a few days tuning Nushell’s GitHub Actions CI pipelines and it paid off: CI used to take about 30 minutes, and now it’s closer to 10. This is not pleasant or glamorous work, but it has a big payoff; every Nu change going forward will spend a lot less time waiting for essential feedback. Here’s how you can do the same.

Use rust-cache

Seriously, it’s really good! GitHub build runners are slow. But GitHub gives every repo 10GB of cache space, and rust-cache takes advantage of that. It caches temporary files for your build dependencies across CI runs, so if you have a lot of dependencies you’ll likely see a big performance boost.

One gotcha to be aware of: GitHub Actions has slightly unintuitive behavior across PRs. PR X is unable to see cache data from PR Y, but they can both see cache data from the base branch (usually main or master). This makes sense from an isolation perspective, but it’s not especially well-documented; I ended up adding an extra CI trigger on main just to fill caches properly.

Split your build and test jobs

Previously we were running cargo build then cargo test in a single job. This was suboptimal for a few reasons:

1. cargo test often needed to recompile crates that had just been built for cargo build. #[cfg(test)] is the most likely culprit here; it makes sense that build output might be different in “test mode”. This has implications for caching too!
2. It’s faster to run build and test in parallel; GitHub gives us 20 build runners for free, and we might as well use them.

Run Clippy after cargo build

Previously we were running Clippy before cargo build. Just switching their order shaved about 5 minutes off every test run! It seems like Clippy can reuse build artifacts from cargo build, but not vice versa.

Use cargo nextest

cargo nextest is “a next-generation test runner for Rust projects.” It’s dead simple to install in CI, and it’s often faster than cargo test. We didn’t see a huge benefit from this (maybe 30-40s faster?), but that’s because our CI time is dominated by compilation; YMMV depending on your code base and test suite.

Conclusion

If you’d like to see the actual changes, they’re all here. Like anything GitHub Actions, this took a lot of tries to get right; those 5 PRs are just the tip of the iceberg, there were a lot more experimental changes in my private fork. I’m hopeful that someday we’ll be able to stop programming in YAML files, but we’re not there yet!

Nushell

Nushell v0.60 is out, and it’s fantastic. This is the first Nu release where I made meaningful contributions (mostly to the website+documentation) and it feels like a good use of my sabbatical time. It’s been interesting figuring out how to sell+explain Nu succinctly; writing good public-facing documentation is hard!

If you haven’t tried Nu, this is a great time to do so; Nu’s not stable yet, but I think you’ll be very pleasantly surprised by the level of polish. I’ve finally made it my default shell on both Windows+Linux.

Most of the work I’m doing for Nushell has a selfish motivation: I want to live in a world where POSIX shells are a thing of the past, and Nushell seems like the most promising way to get there.

Learning

I’ve started working through Crafting Interpreters by Bob Nystrom. My first exposure to Nystrom’s work was Game Programming Patterns, one of the best programming books I’ve ever read. The title’s a little unfortunate because it covers design patterns that are useful in any field of programming; I genuinely think GPP is much more useful to today’s programmer than the book that inspired it.

Crafting Interpreters walks you through building a scripting language from the ground up. The book walks you through an interpreter implementation in Java then C; I’m doing the Java version in C# (personal preference and experience).

Other

I’ve started rekindling some old friendships with people I haven’t seen in person in 2 years, and that’s been really great.

Spring is finally arriving here in Vancouver, so I’ve been finding lots of excuses to be outdoors. My patio’s never been cleaner and I’m looking forward to a lot of spring gardening. I’d like to get some more trellises set up this year; I have a fairly small urban patio so it’s important to make good use of vertical space. “Green to the eye, not green on the ground.”

.NET has been taking huge leaps and bounds in the last few years, and not everyone is aware of it! We’re now able to build small fast single-file .NET applications; this is arguably Go’s biggest strength, and now you can do it in C# and F# too.

Meanwhile, the .NET community has been making excellent libraries to make console applications slick, polished, and easy to write.

The conditions seem right for a .NET renaissance of sorts; all the pieces are in place for us to build great CLI-first software. Here are some useful+easy things you can do in .NET today:

Publish Small Zero-Dependency Executables

In .NET 6 it’s easy to build your application as a single file with no external dependencies (even the .NET runtime!). Applications that include their own runtime are called self-contained. Self-contained apps can be trimmed to remove unused code; trimming reduces a Hello World application from about 60MB to 12MB.

Trimming is well-supported by nearly the entire standard library. Ecosystem support varies; if a library makes heavy use of reflection, C++/CLI, or COM interop then it might not work with trimming yet.

The easiest way to build a trimmed self-contained single-file executable is dotnet publish with the args --self-contained=true -p:PublishSingleFile=true -p:PublishTrimmed=true.

dotnet-releaser makes publishing your app a breeze; it can build for multiple platforms+architectures, then publish the resulting artifacts in a GitHub release.

Console UI

Spectre.Console is a great replacement for Console.WriteLine() and friends. Coloring text with Spectre is as simple as:

AnsiConsole.MarkupLine("[blue]Hello[/] [yellow]World![/]");

It can also do much more; if you need to print tables or prompt users for input, Spectre.Console has you covered. But if you need to build a full-blown terminal UI, check out gui.cs.

Running External Processes

The System.Diagnostics.Process APIs built into .NET are clumsy, to say the least. Thankfully we have much better options these days! Here’s how to run git status with the excellent SimpleExec:

var (stdout, stderr) = await RunAsync("git", "status");

For a more powerful solution, check out CliWrap; anything you can do in a Bash script, you can do in CliWrap.

I recently spent a bit of time messing around with the Postgres wire protocol. I’d like to build a library that lets any application accept connections as if it were a Postgres database; many database clients and databases speak the Postgres wire protocol, and it seems like a practical way to expose tabular data over the network.

Implementing the protocol isn’t too bad; the docs aren’t great but the protocol itself is relatively straightforward. Unfortunately, client behaviour seems harder to accommodate. Clients can issue arbitrarily complex SQL queries to the database to inspect its schema, and sometimes they expect accurate answers. For example, Npgsql issues this beast and then disconnects if it gets an invalid response.

So I’m stuck with a question: how far should I go to support legitimate client behaviour outside of the protocol? The easiest approach for now is to test with a few popular clients and hardcode responses for any queries that they need implemented, but that will leave a long tail of unsupported clients. On the other end of the complexity scale, I could try running schema queries on an in-memory SQLite instance with database objects that mimic Postgres (certainly a massive yak shave, but tempting nonetheless).

I think my next step will be to dig through the source of a few Postgres-compatible databases to see how they solve this.

I recently assembled a tool:

It’s a helper for scripting in C#. It watches a directory with C# files for changes, and whenever a file changes it gets compiled into its own executable. You can install it as an always-on service, and it comes with an opinionated set of utilities to make scripting easier.

Why?

C# the language has recently become much more succinct (top-level statements, global usings); seems like it should be good for writing fast, maintainable scripts! But the tooling around the language still isn’t quite there yet; you need a .csproj project file for every program, and dotnet run often takes a second or 2 to start up, it’s not (yet) optimized for scripting use. This proposal should help in many ways (upvote it!) but who knows if or when it will be done.

Also, this has been stuck in my head for a while:

My OS has no notion of build system. It has all the source code on it and I can edit anything and run the command again with the change immediately applied. Interpreter or compiler, I don’t know, that’s an implementation detail. Then I wake up.

Give it a spin

Instructions, source and pre-built executables for Linux+macOS+Windows are available on GitHub.