Bret Victor’s talk Inventing on Principle (video, transcript) changed the way I think about computing in 2019. Inventing on Principle is partly about Bret’s guiding principle:

“Creators need an immediate connection to what they create. And what I mean by that is when you’re making something, if you make a change, or you make a decision, you need to see the effect of that immediately.”

The Edit-Compile-Run Cycle

Although Bret doesn’t use the term, programmers are deeply familiar with his principle. We’ve all worked with toolchains that introduce significant delay before you can “see” the results of a change, and we know they’re painful. Everyone wants a short edit-compile-run cycle.

But until IoP, I’d assumed that slow cycles wouldn’t materially change the output – you’d eventually get to the same place. This was wrong. I also didn’t appreciate the very small time scales involved; a 5 second delay used to seem trivial to me, but it’s still meaningfully different from a response time measured in milliseconds.

Through some very impressive custom tools, Bret shows how immediate feedback enables exploration, which then gives birth to ideas which would otherwise never see the light of day. This was an epiphany for me. Since IoP I’ve constantly been looking for better ways to code, and re-evaluating my existing processes for shorter feedback cycles. The results:

Rust

My typical Rust development workflow goes something like this:

1. Write a small function that does roughly what I want
2. Write a small unit test inline to exercise the function (even if it’s a private function)
3. Iterate using cargo test until the function is correct
4. Later, “productionize” the tests if necessary

Rust’s native support for inline unit tests helps a lot here, and the excellent type system catches a lot of issues before I even run cargo test. On the other hand, Rust’s compiler is notoriously slow and that extends to IDE tooling that depends on the Rust Language Server. I’m looking forward to Cranelift for faster debug builds.

I recently started experimenting with building tooling for ETL systems. After many years of wrestling with ETL in industry, I had a few questions on my mind:

1. Can we make common data issues quick to resolve?
2. Can we make automated data transformations as easy to work with as spreadsheets?

I think the answer to both questions is yes, but don’t take my word for it – you can try the prototype at etlsheets.netlify.app (double-click on an issue to get started), and view the source code.

Motivation

Importing data at scale is painful. Your data providers will screw up the formatting, systems will experience connectivity issues, and your transformation logic will fail on cases you didn’t expect. What if our tools focused on helping with those failures, instead of assuming the happy path?

Speaking of transformations, how should we write them? Some systems take a code-first approach, which is great for coders and impenetrable for everyone else. Others take a GUI-driven approach, which usually becomes the stuff of nightmares. I think we can do better, by drawing inspiration from a tool that’s found in every office:

Transformations

Here’s what building a new transformation might look like, and here’s what it might look like when that transformation fails to run successfully.