Reilly Wood

2025 Retrospective

Sun, 28 Dec 2025 11:13:42 -0800

It’s been quite a year. Here’s a grab bag of what stuck in my mind from 2025 and what’s coming next.

Personal Stuff

I started 2025 with a New Year’s resolution to deadlift 300lb. I hit 300 in the middle of the year, upped my resolution to 350lb, but only made it to 310lb. I’d like to hit 350 in 2026, but if I plateau and stay healthy I’m fine with that too.

My big resolution for the new year is to host a gathering at least once a month. Doesn’t have to be anything fancy, just an excuse to see people. My plan to make this happen: even if I’m busy, it’s easy to make a big batch of pasta and have guests bring side dishes and wine.

2025 was the first year I really noticed my dog getting old. He’s 11 and doing well for his age, but I’m more aware that I probably only have a few more years left with him. Retired racing greyhounds make great pets:

Author and dog on a road trip to the Okanagan Valley

We spent a lot of time in late 2025 looking at real estate. After 7 years we’re a little bored of our current place and it’d be nice to have a bedroom for guests. It’s been an emotional roller-coaster; in November we had an offer accepted for a gorgeous condo downtown, only to back out after the inspection found some issues. But hey, it could be worse - when we bought our current place the market was so hot that people didn’t even get inspections done 😬.

Work Stuff

I got promoted to Staff Engineer in December. This required a ton of effort plus some luck, and I’m very proud of it; feels like I finally made it, y’know? The promotion felt especially good because during the tech job rout of late 2022 I accepted a Staff offer from another mid-sized tech company, and then after 3 weeks of delay they retracted it.

Overall it was a very good year for work. I shipped a product, worked on a high-profile keynote demo with OpenAI, and changed teams to launch a new product that’s attracting a lot of interest. I also flew down to SF twice to give talks for work; here’s one I’m particularly proud of.

Software

It is an incredibly crazy time to be working in software.

It feels like 2025 was the year where agents blew up. 1 year ago, I was occasionally using Aider to make commit-sized changes to software projects, and I felt like I was ahead of the curve. Today I tend to use Claude Code (sometimes Codex CLI) to make more ambitious changes, and they are far more capable of iterating on a change until they get it right.

My day to day now involves less “hands-on” coding and more high-level management of coding agents. It’s become incredibly cheap to try things out, and Opus 4.5 is remarkably capable.

I’m spending a lot of time with these new tools and I still feel quite a bit of FOMO. It helps to know that I’m not the only one.

What's MCP good for, anyway?

Fri, 26 Dec 2025 19:27:57 -0800

Among people who do a lot of agent-assisted software development, there is some skepticism about whether MCP is useful:

A quick experiment makes this clear: try completing a GitHub task with the GitHub MCP, then repeat it with the gh CLI tool. You’ll almost certainly find the latter uses context far more efficiently and you get to your intended results quicker.

This is a fair criticism; in some scenarios an agent does better if we let it go wild with bash (which is effectively giving it the ability to write+run code) and a CLI tool. I also agree with Armin’s assertion we need better ways to compose MCP tool results. But I still think MCP is useful as-is, and I’d like to sketch out why I believe that.

MCP is simple

To connect an AI agent to an MCP server, I don’t need to download anything; I just provide a URL. Authentication is taken care of as part of the connection (more on this later). Tools are annotated with info that indicates whether they are safe to run. My agent doesn’t need to be able to execute code, and it doesn’t even need a filesystem.

It’s true that this agent might be less flexible or powerful than one with the ability to run arbitrary code. But that’s a tradeoff, and people are exploring ways to combine MCP with code execution - this is something to keep an eye on in 2026!

Programmers are weird

I’m a programmer who spends a lot of time with coding agents like Claude Code and Codex CLI. You could think of me as a power user driving an agent semi-interactively, and most people discussing MCP are in the same boat.

CLI tools are often a viable alternative to MCP for us, but a big part of that is that we can evaluate whether any given call to bash looks safe. That is not a skill that most people have.

OK, sure, we can’t expect most users to validate bash commands. But can’t we just sandbox their agents?

Maybe someday! But sandboxing is hard, and I don’t think anyone’s fully solved the UX around it yet. Are you sandboxing all of your agents today? The answer is “no, it’s too much of a hassle,” right?

Toward Autonomous Agents

Let’s step away from the well-trodden path of Claude Code. Say you’re building an agent that operates autonomously based on untrusted data. To make this more concrete, let’s say it’s an incident investigator agent; when a monitor goes off, it tries to find the root cause using data from your favourite observability provider. How do you give that agent access to your observability data?

If your observability provider has a CLI available, the agent could use that. But using a CLI means:

Your agent will need access to a filesystem (provisioned with a copy of the CLI)
Your agent will need a sandbox to stop malicious code execution and resource exhaustion
You’re opening yourself up to credential exfiltration attacks. The CLI needs credentials to talk to the observability provider; if the agent can execute arbitrary code, it can almost certainly read those credentials.

All of these problems go away if you connect your agent to an MCP server instead. MCP can get a production-ready agent off the ground almost immediately.

Putting it all together

MCP is a dead-simple way to give agents access to tools safely, and it works today. For some agents, that simplicity is extremely valuable; for others it is not. As you move away from expert oversight and toward fully automated agents, the case for MCP grows stronger.

Tool Calls Are Expensive And Finite

Thu, 18 Sep 2025 18:49:43 -0700

Giving LLMs access to tools (which turns them into ✨agents✨) is an incredibly powerful way to give LLMs capabilities that go beyond generating text. But it’s important to think clearly about the costs and limitations of tool calling, and in particular, people should understand that calling a tool is many orders of magnitude more costly than calling a plain old function from code. There is and probably always will be a limit on how many tool calls an agent can effectively make, and people should design their agentic systems accordingly.

Wait, why?

For this to make sense, you have to consider what a tool call is “under the hood.” LLMs are typically used as very fancy text generation machines. And the way they do tool calls is by generating text, although that’s typically abstracted away from us.

Let’s say you have an agent with one tool, add, for adding 2 numbers together. A user asks the agent a question that’s easy to answer with the add tool:

What’s 15 + 27?

To actually call the add tool, the model generates a message like this (simplified):

{
  "tool_call_id": "call_abc123",
  "tool_name": "add",
  "tool_arguments": "{\"a\": 15, \"b\": 27}"
}

At this point the model stops generating tokens. The thing that’s driving the model (the agentic loop?) parses that message, passes those arguments to some function like add(15, 27), and then puts the output of that into chat history as a new message:

{
  "tool_call_id": "call_abc123", 
  "tool_call_result": "42"
}

Inference resumes, and the LLM now has everything it needs to tell the user that the answer is 42. This works! It’s the foundation of some really incredible software systems! But it wasn’t free:

The model had to generate a bunch of tokens.
We used up precious context window for the 2 messages.

But why does that matter?

If you’re adding 2 numbers once, it probably doesn’t matter. If you’re summing up 1,000 numbers… you’re going to be waiting a very long time for those 999 tool calls to finish, and you might blow through your entire context window.

This might seem like an academic point, but calling a function many times in a loop is one of the most common ways to solve a problem with code. To give a contrived example, say we have 1,000 user IDs and we want to list the users whose name starts with ‘R’:

A programmer with a get_user_info(id) function can write+run a simple for loop. Easy peasy.
An agent with a get_user_info(id) tool can try to make 1,000 tool calls, but it will probably run out of context window long before it finishes
1. Remember, the entire result of every tool call ends up in the context window

Designing agentic tools that are flexible enough for every use case (or even most use cases) is hard, and I don’t think enough people are talking about that.

So what do we do instead?

As always, it depends. Maybe your agent is solving problems where it will never need to make large numbers of tool calls. Maybe you’re clever and you can design your tools to be very flexible+powerful so an agent can do a lot in a small number of tool calls. Maybe you can sidestep this problem by letting your agent write+run code (keeping in mind all of the necessary security precautions).

APIs don't make good MCP tools

Tue, 05 Aug 2025 20:15:49 -0700

The Model Context Protocol (MCP) is a pretty big deal these days. It’s become the de facto standard for giving LLMs access to tools that someone else wrote, which, of course, turns them into agents. But writing tools for a new MCP server is hard, and so people often propose auto-converting existing APIs into MCP tools; typically using OpenAPI metadata (1, 2).

In my experience, this can work but it doesn’t work well. Here are a few reasons why:

Agents don’t do well with large numbers of tools

Infamously, VS Code has a hard limit of 128 tools - but many models struggle with accurate tool calling well before that number. Also, each tool and its description takes up valuable context window space.

Most web APIs weren’t designed with these constraints in mind! It’s fine to have umpteen APIs for a single product area when those APIs are called from code, but if each of those APIs is mapped to an MCP tool the results might not be great.

MCP tools designed from the ground up are typically much more flexible than individual web APIs, with each tool being able to do the work of several individual APIs.

APIs can blow through context windows quickly

Imagine an API that returns 100 records at a time, and each record is very wide (say, 50 fields). Sending those results to an agent as-is will use up a lot of tokens; even if a query can be satisfied with only a few fields, every field ends up in the context window.

APIs are typically paginated by the number of records, but records can vary a lot in size. One record might contain a large text field that takes up 100,000 tokens, while another might contain 10. Putting these API results directly into an agent’s context window is a gamble; sometimes it works, sometimes it will blow up.

The format of the data can also be an issue. Most web APIs these days return JSON, but JSON is a very token-inefficient format. Take this:

[
  {
    "firstName": "Alice",
    "lastName": "Johnson",
    "age": 28
  },
  {
    "firstName": "Bob",
    "lastName": "Smith",
    "age": 35
  }
]

Compare to the same data in CSV format:

firstName,lastName,age
Alice,Johnson,28
Bob,Smith,35

The CSV data is much more succinct - it uses up half as many tokens per record. Typically CSV, TSV, or YAML (for nested data) are better choices than JSON.

None of these issues are insurmountable. You could imagine automatically adding tool arguments that let agents project fields, automatically truncating or summarizing large results, and automatically converting JSON results to CSV (or YAML for nested data). But most servers I’ve seen do none of those things.

APIs don’t make the most of agents’ unique capabilities

APIs return structured data for programmatic consumption. That’s often what agents want from tool calls… but agents can also handle other, more free-form instructions.

For example an ask_question tool could perform a RAG query over some documentation, then return information in plain text that is used to inform the next tool call - skipping structured data entirely.

Or, a call to a search_cities tool could return a structured list of cities and a suggestion of what to call next:

city_name,population,country,region
Tokyo,37194000,Japan,Asia
Delhi,32941000,India,Asia
Shanghai,28517000,China,Asia

Suggestion: To get more specific information (weather, attractions, demographics), try calling get_city_details with the city_name parameter.

That sort of layering and tool chaining can be very effective in MCP servers, and it’s something you’ll miss out on completely if auto-converting APIs to tools.

If an agent needs to call an API, it could just do that

Agents like Claude Code are remarkably capable of writing+executing code these days, including scripts that call web APIs. Some people take this so far as to argue that MCP isn’t needed at all!

I disagree with that conclusion, but I do think we should skate to where the puck is going. Sandboxing of agents is improving rapidly, and if it’s easy+safe for an agent to call APIs directly then we might as well do that and cut out the middleman.

Conclusion

Agents are fundamentally different from the typical consumers of APIs. It’s possible to automatically create MCP tools from existing APIs, but doing that is unlikely to work well. Agents do best when given tools that are designed for their unique capabilities and limitations.

Land Values and Affordability

Thu, 26 Jun 2025 20:02:19 -0700

I want to get something off my chest: attempts to keep the price of urban land down are not necessarily good. Many people in local politics place a high priority on keeping land prices down. For example, the new Vancouver councillor who opposed a church building apartments on their own land:

Land values displace people. This will increase land values.

…and then used that same reason to vote against apartments at a major train station:

I’m worried that filtering will take too long, that land value increases will lead to displacement…

Vancouver’s planning staff share these concerns and try to keep land values down when changing zoning. For example, the recent multiplex policy was designed to avoid raising land values:

Proposed density bonus contribution requirements & rates (are) set to… limit any potential land value escalation

That makes sense; if land value is lower then homes are more affordable, right?

WRONG (if you keep land values down by stopping development)

Land Prices Are Not Housing Prices

The main way people save on housing costs in cities is by using less land. For example, imagine the following uses on a 4000 sqft lot:

Building	Land/Household
Single family home (small)	4000 sqft
Duplex	2000 sqft
5-unit apartment/condo building	800 sqft

It is generally much cheaper to buy 800 square feet of land than it is to buy 4000. But where this gets interesting is that those denser uses may cause higher land prices. Let’s walk through how:

Say that 4000 sqft lot is zoned to only allow a single-family home. Richie McRicherson is willing to pay $1M so he can build a house on that land. The land sells for 1 MILLION DOLLARS.
Now, suppose the land is zoned to allow a duplex. 2 households who each have $600k pool their money together and outbid Richie. The land sells for 1.2 MILLION DOLLARS.
Finally, suppose the land is zoned to allow a 5-unit condo building. 5 households who each have $400k pool their money and outbid both Richie and the duplex buyers. The land sells for 2 MILLION DOLLARS.

Building	Land Price	Land Price/Sqft	Land Price/Household
Single family home	$1,000,000	$250	$1,000,000
Duplex	$1,200,000	$300	$600,000
5-unit condo building	$2,000,000	$500	$400,000

It is really important to note that even though allowing more homes drove land prices up, households are paying less for land.

OK that’s the theory; what about in practice?

It can be hard to observe this in real life, because dense city centres tend to be pretty expensive. That’s a complicated topic that’s beyond the scope of this blog post, but there are places where it’s easy to see this specific phenomenon in Vancouver with a map of land values. For example:

North West Point Grey

Left: cheap land and expensive homes. Right: expensive land and relatively cheap homes

This is one of the most expensive neighbourhoods in Vancouver, by design. Apartments are forbidden everywhere, only houses are allowed. And city planning rules require each house to use up much more land west of Blanca Street:

Area	Minimum Lot Size	Land Price/Sq Ft	Land Price/Lot
West of Blanca	12,000-18,000 sqft	Usually around $300	$7M-$30M
East of Blanca	3000-5400 sqft	Usually around $800	$3M-$8M

This is exactly what we were talking about. When the city lets people use less land per home, land prices go up and home prices go down. To be clear, $3M still isn’t cheap; we should go a lot further.

Shaughnessy

It’s a similar story in Shaughnessy, historically Vancouver’s most exclusive neighbourhood:

Top: Fairview/South Granville apartments+condos. Bottom: Shaughnessy mansions

South of 16th we zone for mansions on very large lots (making the land relatively cheap), and north of 16th we allow apartments and condo buildings (making the land relatively expensive). If you know Vancouver at all, you know that those apartments are a lot cheaper than the $10M+ Shaughnessy mansions!

Takeaway

It’s important to distinguish between the cost of land per square foot and the cost of land per home. Limiting density does work to drive the former down, but at a terrible cost: it stops people of modest means from pooling their resources to outbid someone much richer.

My encounter with evil SEO

Mon, 16 Jun 2025 19:02:33 -0700

I had an odd experience with this website, and I’m finally writing it up. The short version:

In August 2024 I wrote a blog post that documented how a local “independent journalist” had written for white nationalist websites.
In October 2024 he filed a DMCA complaint with my host (Netlify).
1. Netlify support rubber-stamped the complaint without giving me a reasonable way to appeal.
2. I moved to CloudFlare and cut the blog post back to a few essential facts+links, to make it easier for the next overworked support person to interpret.
In February 2025 CloudFlare approved another DMCA complaint from someone who’d copied my entire post to a content mill and backdated it!

This post will mostly focus on the 2nd DMCA complaint, as it’s the most interesting one.

My post was copied to… MormonFind.com?

On February 14, while on vacation, I received the following email:

Cloudflare received the below copyright infringement complaint regarding your account. If the content identified in the complaint is not removed within 48 hours, Cloudflare will take steps to disable access to the content, consistent with section 512(c) of the Digital Millennium Copyright Act. Please note that these steps will include disabling access to the reported URL on which the content is located, which will affect any other content located on the same URL.

Complaint Information:

Reporter’s Name: Aaron Bennet

Reporter’s Email Address: <redacted>

Reporter’s Title: Copyright Infringement

Reporter’s Company Name: Bennet Media Association

Reporter’s Address: <redacted>

Reported URL(s): hxxps://www[.]reillywood[.]com/blog/riley-donovan/

Original Work Description: https://mormonfind.com/2024/04/10/riley-donovan-contributes-to-white-supremacist-websites/

To respond to this issue, please reply to abusereply@cloudflare.com.

Sure enough, at that link (archive) was the text from my original post:

Uh, maybe not the most trustworthy news website.

I replied immediately:

It looks like the complainant has copied my blog post and posted it on that mormonfind.com website so they can file a DMCA complaint. That website is an obvious content mill and this complaint was not submitted in good faith.

I even took the post down (worried that they were going to nuke my entire site):

I’ve taken the content down for now but it’s really concerning that Cloudflare would accept a DMCA complaint like this. It should be clear that a content mill described as “Mormon Find the Best News For You!” is not a real news website.

Alas, I never heard back and I don’t know if a human ever read my emails. CloudFlare blocked the post but not my whole website, and I enjoyed the rest of my vacation.

Further Investigation

This was all very frustrating, but also kind of fascinating. Someone had put a lot of effort into getting this one post taken down! Mormon Find wasn’t an especially convincing website, but it was good enough for its purposes.

I poked around a bit and found the sitemap (archived link) for Mormon Find, which turned out to be a basic WordPress site. And conveniently each post had a Last Modified date:

From the sitemap I gathered that:

The site had been “seeded” with an initial round of 30 generic news posts in 2022
In February 2025 the owner posted another ~40 articles including:
1. My blog post, backdated to 2024
2. What looked like a critical review of a dog supplement, backdated to 2023

Pretty clever right? Set up a generic looking website, leave it on ice for a bit, and then use it to do evil SEO.

Lessons Learned

It’s pretty easy to abuse the DMCA to get content taken down online! The process is very biased towards the complainant.
I should probably pay for web hosting. Web hosts are (understandably) not willing to spend much effort defending their free users.

Agents all the way down

Tue, 10 Jun 2025 18:28:48 -0700

Say you’re working on an agent (a model using tools in a loop). Furthermore, let’s say your agent uses the Model Context Protocol to populate its set of tools dynamically. This results in an interesting UX question: how should you show text tool results to the user of your agent?

You could just show the raw text, but that’s a little unsatisfying when tool results are often JSON, XML, or some other structured data. You could parse the structured data, but that’s tricky too; the set of tools your agent has access to may change, and the tool results you get today could be structured differently tomorrow.

I like another option: pass the tool results to another agent.

The Visualization Agent

Let’s add another agent to our system; we’ll call it the visualization agent. After the main agent executes a tool, it will pass the results to the visualization agent and say “hey, can you visualize this for the user?”

The visualization agent has access to specialized tools like “show table”, “show chart”, “show formatted code”, etc. It handles the work of translating tool results in arbitrary formats into the structures that are useful for opinionated visualization.

And if it can’t figure out a good way to visualize something, well, we can always fall back to text.

Why do it this way?

The big thing is that we can display arbitrary data to the user in a nice way, without assuming much about the tools our agent will have access to. We could also give the main agent visualization tools (tempting! so simple!), but:

That can be very wasteful of the context window
1. Imagine receiving 10,000 tokens from a tool, then the agent decides to pass those 10,000 tokens by calling a visualization tool - the 10,000 tokens just doubled to 20,000 in our chat history
The more tools an agent has access to, the more likely it is to get confused
A specialized visualization agent can use a faster+cheaper model than our main agent

It’s not all sunshine and roses; calling the visualization agent can be slow, and it adds some complexity. But I like this approach compared to the others I’ve seen, and we’re not far away from fast local models being widely available. If you’ve got another approach, I’d love to hear from you!

Function Calling is Structured Output

Sun, 02 Mar 2025 16:01:49 -0800

This is a brief post about something that confused me a great deal when I started working with LLMs.

Context

Many LLM providers (Anthropic, OpenAI, Google) support “function calling”, AKA “tool use”. In a nutshell:

When calling the provider’s chat completion APIs, you tell the model “if needed, I can run these specific functions for you.”
The model responds saying “hey go run function X with arguments Y and Z.”
You go and run the function with those arguments. Maybe you append the result to the chat so the model has access to it.

Weather lookup is a common example. You tell the model “I have a function get_temperature(city: String) that looks up the current temperature in a city”, and then when a question like “What’s the weather like in Tokyo?” comes up the model responds to your code with “please call get_temperature("Tokyo")”.

Structured Output

All well and good, but where this gets interesting is that function calling is also a good way to get structured data out of LLMs. You can provide a function definition that you have no intention of “calling”, purely to get data in the format you want.

For example, using the Rust genai library:

// Text to analyze
let text = "The quick brown fox jumps over the lazy dog.";

// Define a tool/function for rating grammar
let grammar_tool = Tool::new("rate_grammar")
    .with_description("Rate the grammatical correctness of English text")
    .with_schema(json!({
        "type": "object",
        "properties": {
          "rating": {
            "type": "integer",
            "minimum": 1,
            "maximum": 10,
            "description": "Grammar rating from 1 to 10, where 10 is perfect grammar"
          },
          "explanation": {
            "type": "string",
            "description": "Brief explanation for the rating"
          }
        },
        "required": ["rating", "explanation"]
    }));

// Create a chat request with the text and the grammar tool
let chat_req = ChatRequest::new(vec![
    ChatMessage::system("You are a professional English grammar expert. Analyze the grammar of the given text and provide a rating."),
    ChatMessage::user(format!("Please rate the grammar of this text: '{}'", text))
]).append_tool(grammar_tool);

// ...and execute it
let chat_res = client.exec_chat("gpt-4o-mini", chat_req, None).await?;

The result will include some JSON like:

{
    "rating": 10,
    "explanation": "This sentence is grammatically perfect..."
}

…and we’re done. We just used function calling to get structured data, with no intention of calling any functions. This is much nicer and more reliable than string parsing on the raw chat output.

This approach is probably obvious to many people, but it was unintuitive to me at first; I think “function calling” is a misleading name for this functionality that can be used for so much more.

Alternative Approaches

This isn’t the only way to get structured data out of an LLM; OpenAI supports Structured Outputs, and Gemini lets you specify a response schema. But for Anthropic, it seems like function calling is still recommended:

Tools do not necessarily need to be client-side functions — you can use tools anytime you want the model to return JSON output that follows a provided schema.

My latest attempt at collaborative text editing

Sun, 12 Jan 2025 14:02:41 -0800

I tried to use Automerge again, and failed.

For those of you who aren’t familiar, Automerge is a neat library that helps with building collaborative and local-first applications. It’s pretty cool! I work on a collaborative notes application that does not handle concurrent edits very well, and Automerge is one of the main contenders for improving that situation.

I gave Automerge a try in 2023 and wasn’t able to get it working, to my chagrin. This weekend there was an event in Vancouver for local-first software with one of the main Automerge authors, so I decided to attend and give it another try. I made a fair bit of progress, but ultimately gave up after spending ~5 hours on the problem. A few thoughts+observations:

I am going off the beaten path (web)

Automerge’s “golden path” is web apps. The core of Automerge is written in Rust, but it’s primarily used via WASM in the browser.

This approach is unpleasant for me; I like Rust, I have a good understanding of how code runs+executes on a “real computer”, and I do not want to write an application where 99% of the business logic runs in the browser. Instead, I tried to write an application where my Rust backend was the primary Automerge node and browser/JS Automerge nodes would talk to it.

This did not go well; the documentation and ergonomics of the Rust library are lacking, and most tutorials assume that you are using the JS wrapper around the Rust library. And then when I tried to use the JS version in my simple web UI, the docs assumed a level of web development sophistication that I don’t have.

To be clear, this is mostly a me problem: primarily targeting the browser is absolutely the way to go in 2025!

I am going off the beaten path (local-first)

Automerge tries to solve a lot of problems related to local-first software. But I wanted to “start small” and solve the problem of concurrent text editing for an application that isn’t local-first. In retrospect this was a mistake; the documentation was written for a very different audience than me, and I wasn’t especially aligned with what other people at the event were building.

Chrome is winning

Something that was discussed at the event: if you are building entirely in-browser local-first applications you may want to target Chrome, because Firefox is way behind on several new+useful APIs. This is sad, but not surprising.

What next?

I think it’s possible to build an Automerge-based collaborative text editor the way I want, but it’s a lot harder than I expected. I’m going to shelve this and revisit it next time I have time+energy to hack on it.

Strength Training

Sun, 29 Dec 2024 10:15:45 -0800

I started strength training for the first time in January 2024, in my late 30s. I wish I’d started earlier, but it’s been a hugely positive change in my life. Some observations:

Standing up straight is a lot easier; back muscles help with posture!
Stability is cool. I’m noticeably better at tightening my core in daily life, balancing, etc.
I can put much more power into specific movements (ex: pedal hard on a bike) before feeling exerted or losing control
I can lift my gangly 85lb dog easily without worrying that I’ll mess up my back
RSI pain in my arms + wrists is nearly gone

I’ve been going to the same gym 3x/week on average. They have a pretty good model where:

They connect you with a specific personal trainer
Before starting classes you do X training sessions, to make sure you have decent form and won’t injure yourself
After starting classes you do a training session every other month or so
The relationship with the trainer is a form of accountability; they notice if you miss classes for a couple weeks

The classes are Crossfit-ish; about 2/3 strength, 1/3 cardio. They’re surprisingly fun now that I know what I’m doing.

Why now?

Sometime in 2023 I saw people I respect in a professional context (Jack Rusher, Andreas Kling) talking about how it’s increasingly hard to build muscle as you grow older, and thought “hmm, concerning.”

Later I started thinking about New Year’s resolutions etc. (as one does), and reached out to a local gym on a whim.

Why didn’t I start earlier?

I’d never enjoyed lifting weights before, and I didn’t know how to do it safely. To be honest, I associated weight lifting with jock culture and looked down on it a bit. I’ve gone through on-and-off phases of going to the gym, but it’s always been individual cardio work. I didn’t enjoy team sports as a kid, and I’ve always been someone who prefers to figure things out on their own. That approach has served me well in some areas, but it wasn’t working for fitness.

Parting Advice

If you haven’t been able to commit to a fitness routine, try trading some money for training and a community of like-minded people. Expert advice is useful and social accountability works!

Goodbye, Twitter

Thu, 07 Nov 2024 20:49:18 -0800

After 13 years of daily Twitter usage, I bit the bullet and switched to Bluesky (urbanism stuff, mostly) and Mastodon (computer stuff) full time.

I made friends on Twitter, I learned a lot, and I was part of a few communities that would not exist without Twitter. I’m proud of my small contributions to Vancouver urbanism over the years, and they mostly happened on Twitter or adjacent to it.

Leaving was sad but unavoidable; the site’s really gone downhill since Musk purchased it. The quality of discourse has plummeted now that the replies to any popular tweet are dominated by bluechecks. Twitter has a critical mass of shitty resentful people who wouldn’t be out of place on 4chan, and the site shoves their opinions in your face.

Anyway, it’s a few months in and I’m glad I made the change. I don’t love Mastodon, but Bluesky feels a lot like an earlier, more pleasant version of Twitter.

How I Use LLMs (Sep 2024)

Sun, 15 Sep 2024 08:13:36 -0700

It feels a bit early to be writing an update to something I wrote 1.5 months ago, but we live in interesting times. Shortly after writing that post, I started trying out Aider with Claude 3.5 Sonnet. Aider’s an open source Python CLI app that you run inside a Git repo with an OpenAI/Anthropic/whatever API key¹.

My Aider workflow

I direct Aider toward a file or multiple files of interest (with /add src/main.rs or similar)
I describe a commit-sized piece of work to do in 1 or 2 sentences
Aider sends some file contents and my prompt to the LLM and translates the response into a Git commit
I skim the commit and leave it as is, tell Aider to tweak it some more, tweak it myself, or /undo it entirely

This works shockingly well; most of the time, Aider+Claude can get it right on the first or second try. This workflow has a few properties that I really like:

It’s IDE-agnostic (no need to switch to something like Cursor)
It’s very low-friction, which encourages trying things out
1. No need to copy code from a browser, write commit messages, etc.
2. Undoing work is trivial (just delete the Git commit or run /undo)
It’s pay-as-you-go (I pay Anthropic by the token, no monthly subscription)

Prompts

Here are some examples of the prompts I do in Aider:

Library updates should be streamed to all connected web clients over a WebSocket. Add an /updates websocket in the Rust code that broadcasts updated LibraryItems to clients (triggered by a successful call to update_handler). The JS in index.html should subscribe to the WebSocket and call table.updateData() to update the Tabulator table
Add a new endpoint (POST or PUT) for adding new items to the library. It will create a new LibraryItemCreatedEvent, save it to the DB, apply it to the in-memory library, then broadcast the new item over the websocket
add a nice-looking button that bookmarks the current song. don’t worry about hooking it up to anything just yet
Add a new “test-api” command to justfile. It should curl the API exposed by add_item_handler and check that the response status code is CREATED
Write a throwaway C# program for benchmarking the the same SQLite insert as create_item() in lib.rs

I’m still developing an intuition for how to write these, but with all of these examples I got results that were correct or able to be fixed up easily. Sometimes I am very precise about what I want, and sometimes I am not; it all depends on the task at hand and how confident I am that the LLM will do what I’m looking for.

What does all this mean?

I dunno! The world is drowning in long-winded AI thinkpieces, so I’ll spare you another one.

All I know for a fact is that if I have a commit-sized piece of work in mind, there’s a very good chance that Claude+Aider can do it for me in less than a minute — today. I’m still exploring the implications of that, but Jamie Brandon’s Speed Matters post feels very relevant. I can try out more ideas and generally be more ambitious with my software projects, which is very exciting.

You can also point Aider at a locally-hosted LLM, which is cool, but in my experience the quality is nowhere near as good as Claude. ↩︎

Riley Donovan contributes to white supremacist websites

Wed, 28 Aug 2024 17:44:51 -0700

This post has been temporarily taken down while I respond to a fraudulent DMCA complaint.

The short version:

I wrote a post about how Riley Donovan (a guy from Salt Spring Island who runs the “Dominion Review” and submits op-eds to many Canadian newspapers) used to contribute to the “Council of European Canadians”, a white nationalist website. He does not like this.
He’s copied the full text of my post to a content mill called “Mormon Find” (backup), backdated it, and submitted a DMCA complaint to get my post taken down. Clever!

How I use LLMs

Thu, 01 Aug 2024 19:46:02 -0700

I find LLMs to be pretty useful these days. I don’t consider myself to be on the frontier of LLM experimentation, but when I talk to (technical) people it sounds like my workflow is pretty uncommon, so I should probably write about it.

LLM (the command-line tool)

Simon Willison’s llm command-line tool is the primary way I use LLMs. I sometimes struggle to describe the appeal of llm to people because it’s boring. llm lets you do the following with any popular LLM (hosted or local):

Ask the LLM one-off questions (optionally taking stdin as context)
Start a chat session (optionally starting from the last ad-hoc question)

And that’s about it! It’s one of those lovely tools that does a few things well. I usually start sessions with exploratory questions/requests, sometimes piping in data:

cat xycursor.rs | llm "the end() function in this file is confusing, explain it"

And then if I need to follow up on a question, llm chat --continue drops me into an interactive chat that starts after the last question+response:

> llm chat --continue
Chatting with claude-3-5-sonnet-20240620
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
> write a comment explaining that function, using ASCII diagrams

Important things about this workflow:

It’s trivial to “connect” the LLM to other data+files
1. For example, every week I used to manually rewrite the output of this script to be more readable before publishing it; now I pipe it to llm and tell it to do an initial rewrite first
llm makes it trivial to go from exploratory work to more focused iteration

I have llm set up to use this custom prompt, no matter what underlying LLM it’s using. I find that it helps make responses much more succinct.

GitHub Copilot

It’s good, I use it every day. It’s a lot more widely known than llm so I won’t spill too much ink over it.

Observations

I use LLMs and web search in a similar way: do a quick exploratory investigation into something, taking the initial results with a grain of salt. The skills+knowledge you need to evaluate Google results are very similar to the ones you need to evaluate LLM results!

I mostly use LLMs for computer stuff, and it’s often really easy to verify whether a programming/computing answer is any good; just try it out! LLMs are probably not quite as useful for fields where that’s not the case.

I’m happy with llm but it is, ultimately, a wrapper around a basic chat interface and we can probably do better. Claude Artifacts is very appealing in that it can offer a faster iteration cycle for web development (but is unfortunately coupled to an expensive subscription service), and Aider is interesting as a better way to give an LLM access to context from an entire code base. I’m hoping we’ll see more tools like these that extend what we can do with LLMs.

Recently

Tue, 02 Jul 2024 17:57:28 -0700

Been a while since the last update, I’ve been busy with the new job. Things have been going well!

I went to New York in March for my first week at Datadog, which was a good way to start the job; I met other new joiners, met people on my team, saw a bit of the city. And the view from the office is alright:

Datadog operates at a much larger scale than any company I’ve ever worked at, and that has some upsides and downsides (but mostly upsides). I feel lucky to work with a lot of smart, enthusiastic people.

Last week I was in New York again, for Datadog’s annual DASH conference. I was helping run a booth for my team and it was good to talk to users (and potential users) in person. I got to see a bit more of the city outside of work; the Intrepid Museum was a highlight (an aircraft carrier! a Space Shuttle! a submarine!).

Outside of work, I’ve been spending a little bit of time on Nushell (but not as much as I would like). I’ve been driving some changes to the explore interactive pager (which reminds me, I need to update that documentation). I’m trying to get it to a point where I’m happy with it for version 1.0; I’m not quite there yet but I’ve made a lot of changes under the hood.

Nonfiction I read recently

What Goes Around Comes Around… And Around… (⭐⭐⭐⭐) Michael Stonebraker’s back for another opinionated overview of databases, this time with Andy Pavlo. The whole thing is good but I particularly like their take on vector databases:

They are single-purpose DBMSs with indexes to accelerate nearest-neighbor search. RM DBMSs should soon provide native support for these data structures and search methods using their extendable type system that will render such specialized databases unnecessary

On a related note, I like Simon Willison’s point that maybe you don’t need vectors for RAG:

The more time I spend with this RAG pattern (ed: one using full-text search) the more I like it. It’s considerably easier to reason about than RAG using vector search based on embeddings, and can provide high quality results with a relatively simple implementation.

Fiction I read recently

Lonesome Dove (⭐⭐⭐⭐⭐) This was described to me as “the Western novel to read if you don’t normally read Westerns” and yeah, it was great.

Glorious Exploits (⭐⭐⭐⭐) Funny + touching story about 2 unemployed potters in 412 BC who decide to put on a play with imprisoned Athenian soldiers as the cast.

Microserfs (⭐⭐⭐⭐) Strange that I hadn’t read this before, but Douglas Coupland has a not-entirely-positive reputation in his hometown. He’s kinda known as the guy who got too many undeserved public art commissions around here. Anyway! It was a really fun read and it felt like it could have been written yesterday (surprising for a book about the tech industry written nearly 30 years ago).

A bunch of Horatio Hornblower and Richard Bolitho books (⭐⭐⭐) I was looking for something along the same lines as the excellent Aubrey-Maturin series. These weren’t quite it. Hornblower isn’t very fun as a protagonist and Bolitho is a boring one, I couldn’t make it very far into either series.

Work Update

Fri, 01 Mar 2024 16:15:52 -0800

So, big news: I’m off to a new job at Datadog, working on their CoScreen screen sharing tool 😎. I’m gonna be working remotely from Vancouver, which will be interesting because I’ll be dogfooding CoScreen as I work on it.

It’ll be good to work on native software again. It feels like the vast majority of jobs today operate at a level of abstraction where the OS doesn’t matter much; when you’re writing REST services it’s rare that you actually interact with the OS itself, right? CoScreen is very different in that it’s native software that needs to do lots of interaction with the OS to provide a great UX. Which is exciting and fun and cool, at least until I get bogged down in the details of GDI or whatever 😅.

My home-cooked software

Fri, 05 Jan 2024 21:42:28 -0800

I reread Robin Sloan’s post about a “home-cooked” app, and it continues to resonate; there’s something special about writing software for very small audiences. In that vein, here’s something that my partner and I have been using since last year:

We needed a solution for shared notes that works well across Android, iOS, and the (desktop) web… so I made one. We use it a lot, mostly for things like our grocery list and cooking notes. It’s simple, fast, and it works for us.

Technology-wise, it’s just a website that works well on desktop and mobile. The back-end is written in Rust, it uses HTMX for some dynamic updates, and the note contents get saved to a SQLite database (backed up to S3 with Litestream). The whole thing compiles down to a single-file executable that gets run on a cheap VPS.

Many corners were cut during the development of this project! Proper collaborative text editing is hard, and I took a quick-and-dirty approach. Eventually I’ll get it working better with Automerge or whatever, but for now I have something good enough for a user base of 2.

And now it’s really easy to do stuff with our shared notes:

That’s a cheap 7" e-ink display with a Raspberry Pi on the back, showing a customized view of our grocery list plus enough information to answer the question “do I need an umbrella today?”. It’s like a little household dashboard that we can take a quick look at as we head out the door. Eventually I need to make a frame for it and mount it on the wall, but it’s pretty handy as-is.

Why do all this?

Shared notes are a solved problem, I’m sure I could have found an existing service for this. But:

I like having a project to tinker on for someone I care for, in the same way that I like cooking for them
This is permanent in a way that cloud services are not. It will never change unless we want it to, and it will will work for decades without much effort. We might be using this in a retirement home one day
It’s trivial to tweak and extend software I wrote myself. I didn’t need to figure out an API to get the the e-ink display, I just built a new HTML view on top of existing data
Software designed for an audience of 2 is able to be much simpler than software that anticipates the needs of a large+diverse user base

Bonus: Burninator.exe

Now that I’ve shown you something useful, here’s some much sillier home-cooked software:

A friend asked:

What is a program that I can run in the background to raise the temperature of a laptop? If my laptop gets too cold the screen starts to flicker…

I wasn’t aware of any, so I wrote one that queries WMI for the current CPU temperature and then does busy work until the temperature is high enough. Is it dumb? Yes. Does it solve this one person’s very specific problem? Also yes!

Recently

Mon, 21 Aug 2023 20:26:41 -0700

It’s been a good summer so far. Some things I’ve been up to:

systemctl-tui

I got annoyed that I couldn’t find a decent GUI for systemd services, so I built one:

It was partially an excuse to experiment with ratatui for terminal UIs in Rust, and I think I like it quite a bit. ratatui gives you tools you need to do immediate-mode rendering efficiently, but the broad strokes of your application’s architecture are up to you.

Gardening + Patio

I spent more time than usual on our little outdoor space. Lots of flowers, a new planter, pressure washing, etc.

So many petunias

Patio furniture is a work in progress

I’ve organized a fair number of get-togethers/dinners on the patio, it’s been really nice meeting new people and catching up with people who I haven’t seen since before the pandemic.

New ‘Puter

I finally took the plunge and bought a Framework Laptop:

A normal computer

Intel’s multicore performance has gotten a lot better in recent generations, so I figured I could replace my desktop for Rust work. So far, so good!

I like the hardware a lot. There were some teething pains getting Linux working properly, but that kinda comes with the territory.

New Project: Escape Artist

Sun, 23 Apr 2023 11:03:55 -0700

I recently participated in the Handmade Network Visibility Jam and built a tool that I’d been wanting for my own use:

Escape Artist in action

In a nutshell, Escape Artist shows ANSI escape sequences that are normally invisible. It’s kinda like View Source for the terminal; my hope is that it will be useful for anyone working on shells and terminal applications.

Tech Stack

Escape Artist is primarily written in Rust, using Axum to serve up a web UI using Tailwind+Preact. Events get streamed to the front-end over a websocket. I took great pains to keep the front-end simple; nearly all business logic lives in the Rust back-end, there is no front-end build step, and all dependencies are bundled with the application.

Escape Artist compiles down to a single-file 1.8MB executable, and nearly half of that is Tailwind and an embedded font.

Lessons Learned

Parsing escape sequences out of a stream of bytes is remarkably tricky. Thankfully the vte crate does most of the heavy lifting for us.

This minimal approach to web UI was generally pretty nice! I occasionally ran into a bit of trouble wrangling JS libraries that assume everyone is using NPM, React, and a bundler like webpack, but overall it wasn’t too bad.

I ended up needing to roll my own tooltip solution using Floating UI, and… wow. Their introductory tutorial is one of the best I’ve ever seen for a library in any language.

Vancouver Building Age Map

Sat, 25 Mar 2023 17:15:35 -0700

Here’s a pandemic project that I think turned out pretty well:

An 18-piece map of Vancouver above my couch

In 1954, Vancouver’s planning department put together a remarkable hand-coloured map showing the age of every building in the city. Thankfully for us, the Vancouver Archives made scans of the map available in impressively high resolution:

Imagine how long it took to colour every building!

I took the scans for the northern (and most interesting) half of the city, had them cleaned up and colour-corrected, then printed them on wooden blocks. The whole thing is about 2 metres wide, and let me tell you it was a pain to hang.

I’m quite happy with how the project turned out, but if I were to redo it I’d whiten the yellowed background; I think that would make the colours really pop.

OK but what’s the point?

My sole motivation when starting this project was “dang, that’s a neat-looking map!” However, looking at it daily really drives home how little Vancouver’s built form has changed since 1954. Most of Vancouver was single-family houses then; most of Vancouver is single-family houses now.

If you’d like a copy of the cleaned up scans, just give me a shout.

The panels for the downtown peninsula

Recently on reillywood.com

Sun, 05 Mar 2023 22:39:25 -0800

I find working on my personal website to be soothing. It’s one of those never-ending projects in a good way, and I work on it in small bursts of energy whenever I have the time and inspiration. I’ve added a handful of new features in recent months and, uh, I’m sorry.

Some of these features are an attempt to revisit the glory that was the Geocities-era web; you might have noticed the glittery cursor trail (based on the excellent cursor-effects), or the animated snowy background that only shows up in December.

I’ve added custom emoji that I can easily use throughout the website (with Hugo shortcodes), here’s a small sample:

I use a lot of custom emoji on Slack and Discord; adding them to my website was the logical next step. I’ve also been toying with the idea of letting readers emoji-react to my blog posts but haven’t quite come up with a design I’m happy with. Speaking of emoji…

Whoa, what’s this?

I also added these, uh, Socratic dialogue bubbles between me and arbitrary emoji heads. These are heavily inspired by “Cool Bear” but with more characters because why not?

Got it. I’m sure everyone will love this and nobody will find it annoying.

These features are pretty dumb and they’d never make it past design review in the real world. In a way, that’s the point. Let’s make websites fun again.

Why Nushell?

Sat, 04 Feb 2023 12:25:26 -0800

I work on a command-line shell called Nushell (Nu for short) with some internet friends, and I think it’s pretty cool. To convince you that it’s cool (or at least worth a try), here’s a whirlwind tour.

Basic Querying

Let’s start by taking a look around the filesystem. We’ll use ls to take a look at the files in /usr/bin:

We got an actual table back! It has columns! 1551 rows is a bit much though; why don’t we see what the 10 biggest files are? We can use sort-by to sort by size, and then grab the first 10 with first:

Maybe we’re only interested in files that have been updated recently. Let’s add a where to the pipeline to filter data:

And to cap it all off, let’s project the results; we’ll use select to grab only the columns we care about:

OK, so what’s all the fuss about? Those seeem like pretty basic operations.

They are basic! Sorting, filtering, and projecting are all table stakes (🥁) for working with tabular data. But think about how how much harder they are in a POSIX shell like Bash:

Sorting: maybe the command you’re using has sorting functionality built in, like -S in coreutils ls. That’s the best case, and it involves remembering a bunch of different sorting flags for different commands. In the worst case, you’ll have to do string parsing to get a substring out of each line and then sort based on that.
Filtering: good luck! Filtering strings in simple ways with grep isn’t too bad. But if you’re working with more complex data types (like dates), you’ll have trouble. For example there isn’t a good way to filter ls output by date, and so most people give up on ls and use find instead.
Projecting: I hope you like string parsing. You’ll have to use awk or something similar to extract substrings from each line, and hope that your input data never changes in a way that causes things to break.

Working With Data

Nushell prides itself on making it easy to work with data from external sources — not just our built-in commands like ls.

Say you have a TOML file. Reading it into a Nushell value is as simple as open foo.toml, and once that’s done you can sort/filter/project it with exactly the same commands as we used with ls above.

This isn’t limited to TOML; you can do the same with XML, JSON, CSV, Excel, and even SQLite. If the usual Nu commands work for your SQLite needs, great. If not, it’s easy to switch to SQL with query db:

And if you want to bring in data from a web API, that’s easy too. Just run fetch <url>, and JSON content will be automatically converted to a Nu table:

Cross-platform. Really.

Nu takes cross-platform support seriously. We develop and test against Linux, macOS, and Windows. This is a big selling point for me; other shells tend to support *nix well (bash, zsh) or Windows well (PowerShell) but not both.

What’s the catch?

It’s still (relatively) early days for Nu. We didn’t get everything right on the first try, there are still a few rough edges when it comes to command design. Accordingly, breaking changes can and do occur.

If you need a shell that will remain unchanged for decades to come, Nu is probably not right for you (yet). But if you’re willing to join us for a ride, you won’t be disappointed.

Why does Vancouver have 895 zoning districts?

Sun, 29 Jan 2023 11:28:09 +0100

If you try to do any kind of analysis of Vancouver’s urban planning, you quickly run into a problem:

An amusing map by Mike Feaver

Vancouver has a lot of zoning districts. 895 of them, to be exact¹. That works out to roughly 1 zoning district for every 740 residents; I hope you weren’t planning on reading every one.

“Comprehensive Development”

About 9/10 of those zoning districts are bespoke ones; unique zoning bylaws that apply to one specific property, never to be reused. The pejorative term for this practice is “spot zoning,” but the official name is “comprehensive development” (CD). You can see the full list of 813 CD zones here.

Why would 1 property ever get its own zone? Don’t zoning districts usually apply to a whole neighbourhood?

The theory is that sometimes people want to build unusual things that the city never thought to plan for. Like maybe you want to build an amusement park, but the city doesn’t have a zoning district that allows for amusement parks — and the easiest solution is to make you a new, unique zoning district.

Does that mean we have 813 super special buildings in Vancouver?

Well…

If you look back at the earliest CD zones in Vancouver (starting in the 1950s), they often fit the theory; they apply to unusual sites like the (huge) Oakridge Centre mall and the PNE fairgrounds. Unfortunately things got out of hand, and over the years CD zones became a go-to tool for everything:

A duplex in Mount Pleasant with its own CD zone

Low-rise seniors’ housing with its own CD zone

Apartments 2 blocks from Commercial+Broadway station, with their own CD zone

Yaletown towers. Nearly every building in this photo has its own CD zone

If you pick a random multifamily building from the 1980’s or later, the odds are good that it has its own CD zone.

OK, but does it matter?

Giving nearly every new building its own zoning bylaw creates some problems:

This doesn’t scale. Writing a new zoning bylaw takes a lot of time for a trained professional, and every new bylaw needs to be approved by council at a public hearing.
The more zones we have, the harder it is to make changes to our urban planning regime. Changing 1 zoning bylaw is really hard; changing 100 bylaws is nearly impossible.
No one person² can read and understand 800+ zoning bylaws; we’ve long since reached a point where our planning code is beyond the grasp of professionals (to say nothing of us interested amateurs).

As an example of #2, consider one train station on the Canada Line:

A decade after the Canada Line was built, the new things around the busy King Edward/Cambie station are as follows:
- 9 midrise market condos
- Only one has retail outlets
- Those outlets are a fitness place, an insurance company, and a Tim Hortons
- That's it
Vancouver!
— Justin McElroy (@j_mcelroy) March 12, 2021

Each condo building was given a unique CD zone, and for nearly all of those the planners just… didn’t allow retail uses.

If the buildings had all been assigned a standard multifamily zone, the zoning could be changed for multiple buildings at once. As is, every building will need its unique zoning bylaw updated individually — but Vancouver rarely does proactive zoning changes, and frankly doesn’t have the administrative capacity to rewrite 8 distinct bylaws at once. The best we can hope for is that one or two strata councils pressure the city to fix their building, and even that’s unlikely.

Why do we do this?

⚠️ Warning: Speculation Ahead ⚠️

It’s about control. CD zones can be (and usually are) narrowly written to allow one specific building form and use; no more, no less. This approach gives the planning department a great deal of control over what gets built, as compared to reusable zoning districts which need to be a little more flexible. And as we know, the planning department prizes their ability to shape and control urban design.

Typically this approach gets justified in the name of “good urban design”, but from what I hear it’s more about risk aversion. Planning is a highly politicized profession, and politicians+planners don’t want to get blamed for any unintended consequences of a zoning bylaw. One way to mitigate that risk is to be extremely prescriptive about what can be built, and CD zones accomplish that.

Admittedly, this is speculation and hearsay — but it’s informed by a decade of paying close attention to Vancouver planning and politics. If I got this wrong, please let me know.

Determined using the official Zoning districts and labels data set on January 30, 2023. ↩︎
To mitigate this, the UBC Sociology Zoning Project stepped up with a small army of research assistants to read and summarize zoning codes for us. They’ve done incredible work, but their work shouldn’t be necessary. ↩︎

"Requires the decision of the Director of Planning"

Wed, 11 Jan 2023 16:40:25 -0800

For a long time, my mental model of urban planning was basically “there are written rules about what you can build, and to build something you just follow the rules.” Unfortunately, this is not an accurate way to think about Vancouver.

I get alerts for new rezoning+development applications (using a small tool I wrote), and this little snippet shows up over and over:

…the application is “conditional” so it may be permitted. However, it requires the decision of the Director of Planning.

You can see this on virtually any type of housing proposed in Vancouver, here’s a selection:

I’ve been tracking rezoning+development applications for about a year and a half, and my data set has 599 items with the words “Director of Planning”.

OK, but… why? Doesn’t the Director of Planning have better things to do than approve individual houses and small apartment buildings?

I’ll defer that question to a senior planner at the City of Vancouver:

Oh you mean real Outright! There’s not much. The discretionary system, invented 30’yrara ago, kept outright aspecrs pretty low in order to empower staff to negotiate good urban design for conditional density and height.
— Paul C P Cheng (@PaulCPCheng) June 8, 2021

To translate: Vancouver intentionally allows very little development without giving planners a ~~discretionary~~ subjective veto. Nearly anything other than a suburban house (but also many suburban houses!) is “conditional”, and for conditional uses the Director of Planning can impose any requirements they like. A typical zoning bylaw often looks like this:

Conditional uses go something like this for developers:

Submit a detailed architectural proposal
Wait a long time (often over a year)
Receive a document with many pages of conditions requiring specific changes to the initial proposal - or a rejection
1. The changes can be to architecture, landscaping, parking, or whatever the planners fancy this month - conditions are entirely up to the discretion of staff

For example, here’s a small apartment building near Commercial-Broadway that did not require a rezoning but it did get 17 pages of conditions.

The Director of Planning can delegate this work to other planners with architecture credentials. I’m told that “about 7” people other than the Director have the authority to approve a project.

OK, so what?

You could tell a story in which this system makes sense. Some plots of land are unique, and perhaps important development sites really do benefit from senior planners’ time and site-specific requirements.

On the other hand… like Paul said, there’s not much that this system doesn’t apply to. The law, in its majestic equality, forbids outright development of townhouses and large condo towers alike.

In practice, our conditional approval system means that a lot of important planning rules live in the heads of planning staff. This isn’t so bad if you’re a large developer; they’re sophisticated enough to know what staff are likely to approve, in no small part due to their ability to hire the well-connected.

It is, however, remarkably difficult for small builders (and interested amateurs like me) to answer the question “what can be built on this land?”. Good luck out there!

Windows UI in 2022

Thu, 24 Nov 2022 08:42:46 -0800

Like many other people, I unexpectedly found myself stuck at home without much to do in early 2020. To stay sane, I set about answering a question I’d had for a while: what’s the deal with native GUI frameworks on Windows these days?

It’s rarely obvious which of Microsoft’s several supported technologies is the best choice for a new project, and Windows doesn’t have the same culture of idiomatic+consistent native GUIs as macOS. After months of obsessive ~~escapism~~ focus I emerged with a decent understanding of the problem space; let’s start with an overview of the technologies in use today:

Win32

This is kind of a catch-all term for the old-school way to build a Windows application using arcane, less-than-ergonomic C APIs. This is unpleasant and your application will look ugly, but one big advantage is that the Win32 APIs will live forever and they underpin all the other UI frameworks.

It’s well worth your time to learn a bit of this (read Charles Petzold’s book ), but it’s rare to see people building pure-Win32 GUIs these days.

Windows Forms (WinForms)

A designer-first GUI framework for .NET applications. It gets a bad rap because the average WinForms UI is pretty ugly, but it remains surprisingly practical. WinForms is a thin layer over the old-school Win32 APIs, which is both a blessing (simplicity) and a curse (mediocre layout+styling options). WinForms is well-supported by the .NET team.

Windows Presentation Foundation (WPF)

A sophisticated GUI framework using a declarative HTML-like language (XAML). It’s pretty good but complex and nearly abandoned; it hasn’t received new features in a decade. Ownership has changed to the Windows team (previously WPF was owned by the .NET team) and they’re doing the absolute bare minimum to support it.

Many people still use WPF to build new applications today, and I can’t blame them. But it doesn’t exactly have a bright future.

Universal Windows Platform (UWP) / WinUI 2

To make a long story short, this is what Microsoft started pushing heavily with Windows 8. It uses XAML just like WPF, but the default styling looks a lot nicer. UWP applications are quite nice to use.

Unfortunately Microsoft saddled UWP developers with incredibly painful sandboxing and packaging requirements, and as a result it never saw much adoption. Microsoft finally pulled the plug about a year ago, and UWP is in maintenance mode; do not start new projects in UWP.

WinUI 3

A small team is trying to salvage what they can from UWP’s UI stack (WinUI). Slow, buggy, and weighed down by a lot of historical baggage. Very few teams at Microsoft are dogfooding WinUI 3, and so I hesitate to recommend it.

WebView2

The Windows team has been going full steam ahead embedding Edge/Chromium in Windows, and WebView2 is the result. WebView2 makes it easy to embed web UI in any native application (here’s a demo), and Microsoft teams are adopting it en masse.

It’s a little sad, but if you’re looking for an actively developed UI technology with a lot of energy+investment behind it… this is it. All the usual complaints about web UI apply, but if you can get past those I have no qualms recommending WebView2.

Wow, what a mess. Why did things end up this way?

~~It’s all Steven Sinofsky’s fault.~~ Well, it’s complicated.

Win32 has many shortcomings for historical reasons, and Microsoft’s obsession with backwards compatibility means that it can’t be improved.

WinForms made it a lot easier to build a UI, but it’s a thin layer over Win32 and it can only go so far.

WPF was a huge leap forward with a big team behind it. But then it got abandoned because:

The Windows team bet big on UWP. In some ways they did a great job; UWP made it easy to build attractive, fast UIs that work well on multiple devices (computer, tablet, phone). Unfortunately, UWP’s good parts were coupled with an unacceptable bargain: to use UWP technologies, developers had to cede a lot of control over distribution of their application. Most developers decided this was a raw deal, and rejected it.

Microsoft realized their mistakes with UWP a few years ago, and have been trying to make various UWP-related technologies (including the UI framework WinUI) available with fewer restrictions. Unfortunately I suspect these efforts are too little, too late; I believe that if WinUI 3 had a bright future, we’d be seeing Microsoft adopt it internally. By and large, they are not.

S3 buckets and custom domains

Tue, 02 Aug 2022 10:06:15 -0700

Say you want to host some files in an S3 bucket, under your own custom subdomain with nice short HTTPS URLs. For example, you own foo.com and you want files to be accessible at URLs like https://files.foo.com/bar.txt.

This is a lot more complex than it should be! It involves configuring 3 separate AWS services and I’m already forgetting the boring details, so let’s write them down for future reference.

Cloud is the future… wait, 3 separate AWS services?

YEP.

…

Creating the S3 bucket

Naming is important here - the S3 bucket must have the same name as the subdomain it will be accessed at. Open up S3 in the AWS console, and:

Create a new bucket named files.foo.com.
Disable “Block all public access”.
Under the bucket’s Permissions tab, add a bucket policy to make all objects public by default (replace files.foo.com with the name of your bucket):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::files.foo.com/*"
        }
    ]
}

Certificate Creation+Config

Next up, we need to create a certificate in AWS Certificate Manager.

Hot Tip

Certificates must be created in the us-east-1 region to work properly with CloudFront. Learn from my mistake, make sure you’re in the right region when performing this step.

Request a public certificate in Certificate Manager (us-east-1)
1. Fully Qualified Domain Name: the subdomain you want (ex: files.foo.com)
2. Use DNS validation
Open up the certificate in CM. It will have “CNAME name” and “CNAME value” fields under Domains, those are used for verification of ownership
Go to your domain registrar and create a new CNAME that redirects “CNAME name” to “CNAME value”
- note: for some reason AWS puts a '.' at the end of each field, strip those off when creating the CNAME
Wait a bit (5 minutes?) until AWS verifies that you own the domain

CloudFront Setup

Finally, you need to set up a CloudFront distribution that uses the certificate to serve content from your S3 bucket.

Open up CloudFront (in us-east-1) and create a new distribution
1. Origin domain: pick your S3 bucket
2. Pick “Redirect HTTP to HTTPS” (optional)
3. Add files.foo.com to the “Alternate domain name (CNAME)” list
4. Pick your certificate in “Custom SSL certificate”
Open the distribution behaviors and set cache policy = CachingDisabled (optional)
1. Without this, CloudFront will cache files and serve stale ones for a while. Probably not what you want
Get the “Distribution domain name” from the CloudFront distribution page
Go to your domain name registrar and create a new CNAME redirecting foo.bar.com to the CloudFront distribution domain name.
Wait a bit for DNS and the CloudFront distribution to catch up, and… you’re (hopefully) done! 🤞

Recent Nushell/Rust Work

Sun, 29 May 2022 19:43:53 -0700

I’ve joined the Nushell core team. This doesn’t really change what I’m doing day-to-day, but it makes my work on Nu feel a little more official 🙂.

SQLite Support

This is the biggest feature I’ve implemented so far:

I’m pretty proud of how this turned out; it’s very convenient to be able to browse SQLite databases in your shell and interact with them the same way you would any other data source. Nu is often-but-not-always smart enough to avoid unnecessary work when loading things from the database; there’s still some work to do here and it will probably involve rearchitecting how Nushell queries data.

File watcher

I also implemented a watch command that runs arbitrary Nu code in response to file changes. Nothing groundbreaking, but I find myself needing this kind of low-key automation all the time: run tests when code changes, restart a development server, log changes in a directory, etc. I think the ability to respond to file changes should be a more widely available primitive, and now it is.

Rust for Windows

Against all odds, I somehow got sucked back into Windows development. I spent a solid week helping one of Nushell’s dependencies do a big upgrade of their Windows functionality. This required a deep dive into the current state of calling Windows APIs from Rust, and… it’s a mixed bag.

I used the windows crate which is maintained by Microsoft. It’s an automatically generated set of Rust bindings for Windows APIs, which is both good (very comprehensive, always kept up to date) and bad (some rough edges that might be solved in a handmade solution like winapi). The crate is actively being worked on and it frequently has breaking changes; this means documentation is a little scarce and often out of date. Overall I was impressed and I think the crate has a bright future . But until it settles down a bit, expect some growing pains.

Rust in 2022

Sat, 07 May 2022 10:29:50 -0700

I’ve been using Rust full time for the last month and a bit while contributing to Nushell (more on that later). A lot has changed since I first tried Rust in 2019 and this is my first time working on a big Rust project. Here are some thoughts on the language while they’re still fresh in my head.

Compile times and feedback loops

Rust’s compile times are notoriously slow. Rust development was slow enough on my laptop that I finally gave up on mobile computing and bought a desktop with a top-of-the line CPU (12900K). Along the way I switched from Windows to Linux (more on that later) and started using the mold linker, and now… things are OK!

I’m able to do incremental builds of Nushell (a huge project) in a second or 2, and a full debug build takes 25s. For smaller projects, incremental builds are nearly instant. There’s certainly room to improve here, and the development experience is not great on average hardware, but… this works for me.

Another thing to consider is that the typical Rust feedback loop is tighter than you might expect from the slow compile times. The Rust compiler catches a lot of bugs before a full build needs to happen, and that reduces the need to do a full build and try things out.

Complexity + monotony

Rust is not a simple language. In total I’ve spent nearly 3 months working mostly in Rust, and the language still has a lot of corners that I don’t have a solid grasp on. To improve on this I’m going to need to branch out from Nushell and write a lot of little tools for myself.

Despite the complexity, I’ve found that writing Rust is sometimes a bit… braindead? The type system is very expressive and the compiler catches a ton of errors, so I spend 25% of my time thinking real hard and 75% painting by numbers to make the compiler happy. I can’t quite decide how I feel about this style of development, it can be a little tedious but it also makes for a better end product.

I (sometimes) want a higher-level Rust

Rust has a lot of great things going for it; the tooling, community, package ecosystem, compiler, and syntax are all fantastic. But the focus on systems programming does mean that Rust isn’t quite as ergonomic as it could be for many use cases.

Sometimes I just want a garbage collector! Sometimes I’d be perfectly happy for Rust to implicitly allocate memory if it makes my code work (for example: converting from a &str to a String)! I don’t know if that will ever be possible in standard Rust, but… maybe there’s room for a Rust variant intended for higher-level use cases.

On the other hand, the ability to go as low as you want is great. It’s nice to work in a language with a very “high ceiling”; no matter where your Rust project goes, you won’t have to switch to C or C++.

How to Make Rust CI 2-3x Faster

Sat, 30 Apr 2022 15:14:43 -0700

Big shout-outs to @reillywood, who managed cut the CI times for @nu_shell in half!

If you do @rustlang and see your CI build times creeping up, you should give his fixes a look. Smarter test ordering and build caching makes a huge difference.
— JT (@jntrnr) April 30, 2022

I recently spent a few days tuning Nushell’s GitHub Actions CI pipelines and it paid off: CI used to take about 30 minutes, and now it’s closer to 10. This is not pleasant or glamorous work, but it has a big payoff; every Nu change going forward will spend a lot less time waiting for essential feedback. Here’s how you can do the same.

Use rust-cache

Seriously, it’s really good! GitHub build runners are slow. But GitHub gives every repo 10GB of cache space, and rust-cache takes advantage of that. It caches temporary files for your build dependencies across CI runs, so if you have a lot of dependencies you’ll likely see a big performance boost.

One gotcha to be aware of: GitHub Actions has slightly unintuitive behavior across PRs. PR X is unable to see cache data from PR Y, but they can both see cache data from the base branch (usually main or master). This makes sense from an isolation perspective, but it’s not especially well-documented; I ended up adding an extra CI trigger on main just to fill caches properly.

Split your build and test jobs

Previously we were running cargo build then cargo test in a single job. This was suboptimal for a few reasons:

cargo test often needed to recompile crates that had just been built for cargo build. #[cfg(test)] is the most likely culprit here; it makes sense that build output might be different in “test mode”. This has implications for caching too!
It’s faster to run build and test in parallel; GitHub gives us 20 build runners for free, and we might as well use them.

Run Clippy after cargo build

Previously we were running Clippy before cargo build. Just switching their order shaved about 5 minutes off every test run! It seems like Clippy can reuse build artifacts from cargo build, but not vice versa.

(Dec 2024: I’ve been told that this doesn’t work anymore. Possible that something’s changed in Cargo/Rust)

Use cargo nextest

cargo nextest is “a next-generation test runner for Rust projects.” It’s dead simple to install in CI, and it’s often faster than cargo test. We didn’t see a huge benefit from this (maybe 30-40s faster?), but that’s because our CI time is dominated by compilation; YMMV depending on your code base and test suite.

Conclusion

If you’d like to see the actual changes, they’re all here. Like anything GitHub Actions, this took a lot of tries to get right; those 5 PRs are just the tip of the iceberg, there were a lot more experimental changes in my private fork. I’m hopeful that someday we’ll be able to stop programming in YAML files, but we’re not there yet!

Recently

Thu, 24 Mar 2022 22:01:30 -0700

Nushell

Nushell v0.60 is out, and it’s fantastic. This is the first Nu release where I made meaningful contributions (mostly to the website+documentation) and it feels like a good use of my sabbatical time. It’s been interesting figuring out how to sell+explain Nu succinctly; writing good public-facing documentation is hard!

If you haven’t tried Nu, this is a great time to do so; Nu’s not stable yet, but I think you’ll be very pleasantly surprised by the level of polish. I’ve finally made it my default shell on both Windows+Linux.

Most of the work I’m doing for Nushell has a selfish motivation: I want to live in a world where POSIX shells are a thing of the past, and Nushell seems like the most promising way to get there.

Learning

I’ve started working through Crafting Interpreters by Bob Nystrom. My first exposure to Nystrom’s work was Game Programming Patterns, one of the best programming books I’ve ever read. The title’s a little unfortunate because it covers design patterns that are useful in any field of programming; I genuinely think GPP is much more useful to today’s programmer than the book that inspired it.

Crafting Interpreters walks you through building a scripting language from the ground up. The book walks you through an interpreter implementation in Java then C; I’m doing the Java version in C# (personal preference and experience).

Other

I’ve started rekindling some old friendships with people I haven’t seen in person in 2 years, and that’s been really great.

Spring is finally arriving here in Vancouver, so I’ve been finding lots of excuses to be outdoors. My patio’s never been cleaner and I’m looking forward to a lot of spring gardening. I’d like to get some more trellises set up this year; I have a fairly small urban patio so it’s important to make good use of vertical space. “Green to the eye, not green on the ground.”

Recently

Thu, 10 Mar 2022 09:10:25 -0800

“Work”

I’ve been spending a lot of time writing Nushell documentation. The Nushell core team has been banging out new features rapidly and the documentation hasn’t always kept up; improving the documentation feels like a high-impact way to help out.

Nushell’s got a big release coming up later this month and it has the potential to attract many new users. I want the documentation to be in a better place by then, I’m going to keep spending time on this.

I’ve tried to make a few commits to Nushell itself, but I keep bouncing off Rust. I’m reasonably familiar with Rust, but this is my first time working on a big Rust project. The compile times are painful and they inhibit rapid iteration; I haven’t found a development approach that I like yet.

Reading

Anthony Bourdain’s Typhoid Mary was pretty disappointing (3 stars?); you can tell that Bourdain was short on source material so most of the book is him imagining what Mary might have done.

I started rereading the Wheel of Time series in December and I’m on book 9 now. My interest started to wane around book 8; I think I’ll press onward but I’m really looking forward to the later books (written by another author).

Other

DDR mat with confused dog

I think I’m a Dance Dance Revolution guy now. I ordered a fancy mat from Poland and have been playing Club Fantastic. I’m still pretty bad but it’s a lot of fun.

Recently

Thu, 03 Mar 2022 14:33:22 -0800

I’m officially unemployed again 😎. In the interest of self-accountability, I’m going to try to document what I’m up to on my break; expect more frequent updates to this blog.

HYTRADBOI

I bought a ticket to Have You Tried Rubbing A Database On It?, which could loosely be described as a hipster database conference; lots of people using databases in unusual ways, not much in the way of enterprise RDBMSs. The speaker list is like a Who’s Who for offbeat database work, and I’m really looking forward to it.

Nushell

I’ve been using Nushell as my shell on both Windows and Linux, about half the time. Nushell is a fascinating project; it’s a shell that operates on structured data like PowerShell, but without PowerShell’s (many) pain points.

Nushell has recently seen some massive upgrades (the parsing and evaluation engine was completely rewritten) and it’s a very good time to give it a try. It’s still early days, but I’m hopeful Nushell will be able to displace POSIX shells; it’s liberating to work with much richer data types than plain text:

Nu is a way of saying “what if we didn’t need tools like awk so often?” Since you’re working with structured data, as we add more support for file types, it’s less often you need to reach for “awk”, “jq”, “grep”, and the array of other tools to open and work with common file types. In a way, it’s taking the original spirit of Unix — where you use pipelines to combine a set of tools — and imagining how that original spirit would work today, with what we know about programming languages and tools.

Building data-centric apps with a reactive relational database

This essay touches on a lot of my favourite things: SQLite! The intersection of native apps and web UI! iTunes clones! In a nutshell, it’s a very cool approach to building GUI applications in which all of the application’s state lives in a local database.

It’s more of a provocation than a fully finished system, but I think it shows promise. I’d like to see a bit more investigation of “escape hatches”; how hard would it be mix in a little imperative code when SQL/SQLite aren’t the right fit for a task? Also, this was a bit depressing:

One challenge has been inter-process communication. When the reactive graph is running in the UI thread and the SQLite database is on a web worker or native process, each query results in an asynchronous call that has to serialize and deserialize data.

This would have been a non-issue in traditional GUI code (just query SQLite on a background thread in the same process); one more thing we lose as web UI takes over, I guess 😞.

On The Road Again...

Thu, 24 Feb 2022 19:35:28 -0800

I handed in my notice; my last day at work is next Wednesday. My employer for the last 2 years has been excellent to me, but I’ve been itching to try something new.

I don’t have a new job lined up yet; I’m planning to use this time partially as a sabbatical, and partially to see what’s out there on the job market. A “serendipity break” during which I will actively explore different possible paths.

One of my plans is to explore the intersection of native apps and web UI; I’ve done some promising experiments with that and it would be a good excuse to overhaul ReiTunes. I’d also like to build some new tools for working with the best database. Here we go!

Building .NET CLI Apps in 2022

Fri, 11 Feb 2022 17:51:44 -0600

.NET has been taking huge leaps and bounds in the last few years, and not everyone is aware of it! We’re now able to build small fast single-file .NET applications; this is arguably Go’s biggest strength, and now you can do it in C# and F# too.

Meanwhile, the .NET community has been making excellent libraries to make console applications slick, polished, and easy to write.

The conditions seem right for a .NET renaissance of sorts; all the pieces are in place for us to build great CLI-first software. Here are some useful+easy things you can do in .NET today:

Publish Small Zero-Dependency Executables

In .NET 6 it’s easy to build your application as a single file with no external dependencies (even the .NET runtime!). Applications that include their own runtime are called self-contained. Self-contained apps can be trimmed to remove unused code; trimming reduces a Hello World application from about 60MB to 12MB.

Trimming is well-supported by nearly the entire standard library. Ecosystem support varies; if a library makes heavy use of reflection, C++/CLI, or COM interop then it might not work with trimming yet.

The easiest way to build a trimmed self-contained single-file executable is dotnet publish with the args --self-contained=true -p:PublishSingleFile=true -p:PublishTrimmed=true.

dotnet-releaser makes publishing your app a breeze; it can build for multiple platforms+architectures, then publish the resulting artifacts in a GitHub release.

Console UI

Spectre.Console is a great replacement for Console.WriteLine() and friends. Coloring text with Spectre is as simple as:

AnsiConsole.MarkupLine("[blue]Hello[/] [yellow]World![/]");

It can also do much more; if you need to print tables or prompt users for input, Spectre.Console has you covered. But if you need to build a full-blown terminal UI, check out gui.cs.

Running External Processes

The System.Diagnostics.Process APIs built into .NET are clumsy, to say the least. Thankfully we have much better options these days! Here’s how to run git status with the excellent SimpleExec:

var (stdout, stderr) = await RunAsync("git", "status");

For a more powerful solution, check out CliWrap; anything you can do in a Bash script, you can do in CliWrap.

Pretending to be a Postgres server - Part 1

Sun, 30 Jan 2022 15:34:49 -0800

I recently spent a bit of time messing around with the Postgres wire protocol. I’d like to build a library that lets any application accept connections as if it were a Postgres database; many database clients and databases speak the Postgres wire protocol, and it seems like a practical way to expose tabular data over the network.

Implementing the protocol isn’t too bad; the docs aren’t great but the protocol itself is relatively straightforward. Unfortunately, client behaviour seems harder to accommodate. Clients can issue arbitrarily complex SQL queries to the database to inspect its schema, and sometimes they expect accurate answers. For example, Npgsql issues this beast and then disconnects if it gets an invalid response.

So I’m stuck with a question: how far should I go to support legitimate client behaviour outside of the protocol? The easiest approach for now is to test with a few popular clients and hardcode responses for any queries that they need implemented, but that will leave a long tail of unsupported clients. On the other end of the complexity scale, I could try running schema queries on an in-memory SQLite instance with database objects that mimic Postgres (certainly a massive yak shave, but tempting nonetheless).

I think my next step will be to dig through the source of a few Postgres-compatible databases to see how they solve this.

Scripting in C#

Fri, 28 Jan 2022 17:15:29 -0800

I recently assembled a tool:

It’s a helper for scripting in C#. It watches a directory with C# files for changes, and whenever a file changes it gets compiled into its own executable. You can install it as an always-on service, and it comes with an opinionated set of utilities to make scripting easier.

Why?

C# the language has recently become much more succinct (top-level statements, global usings); seems like it should be good for writing fast, maintainable scripts! But the tooling around the language still isn’t quite there yet; you need a .csproj project file for every program, and dotnet run often takes a second or 2 to start up, it’s not (yet) optimized for scripting use. This proposal should help in many ways (upvote it!) but who knows if or when it will be done.

Also, this has been stuck in my head for a while:

My OS has no notion of build system. It has all the source code on it and I can edit anything and run the command again with the change immediately applied. Interpreter or compiler, I don’t know, that’s an implementation detail. Then I wake up.

Give it a spin

Instructions, source and pre-built executables for Linux+macOS+Windows are available on GitHub.

Blog.exe version 0.1

Thu, 13 Jan 2022 18:47:47 -0800

Here’s something I’ve been messing around with:

Download here or get more info on the MinimalWebView and MaximalWebView repos. I’ll write more about these projects later!

.NET Hot Reload: Where Do We Go From Here?

Wed, 12 Jan 2022 12:18:31 -0700

I had a weird week back in October, and it all started when I posted this GitHub issue.

What happened?

The .NET team built a feature called Hot Reload for .NET 6, which would allow for code to be changed while an application is running. It was planned as a feature for all of .NET from the start:

Our goal is to provide a consistent dev inner loop experience in .NET Core across all platforms, architectures, and workloads.

A preview version of the feature landed in April 2021 and by October it was pretty much done. I used Hot Reload throughout the preview period and loved it. Then at the last minute, Microsoft announced that the feature would be locked to Visual Studio; to say I felt disappointed would be an understatement. I posted on GitHub asking for more information, and I soon found out that many, many others felt the same way.

There was a huge community uproar, I was quoted in multiple news articles, and Microsoft eventually backed down. It was a happy ending, but I was still left with a bitter taste in my mouth.

OK, so what?

To many people, it’s not obvious why this was such a big deal. I think there are 2 main reasons:

Quick feedback isn’t a niche feature; it’s an essential element of any creative activity.
Limiting Hot Reload to Visual Studio would exacerbate .NET’s existing weaknesses.

I’ve rambled about 1 before, so I’ll focus on 2 here.

.NET’s at a bit of a disadvantage when it comes to cross-platform development, in both perception and reality. The tooling situation outside of Visual Studio (or Rider) often leaves much to be desired, and newcomers often get a poor first impression from the OmniSharp tooling in VS Code. Hot Reload in dotnet watch is a big step forward for .NET outside of Visual Studio; it’s a feature I can point to and say “yes, .NET has good tooling no matter where you use it.”

That’s the crux of the matter; removing Hot Reload from dotnet watch would harm .NET in an area it needs to do much better in. The .NET team understands this, but the change was imposed on them by senior Developer Division leadership against strong internal opposition. This suggests that senior leadership does not understand .NET (bad) or prioritizes Visual Studio over the long-term health of .NET (worse).

I like working in .NET, and the .NET team is doing a world-beating job of pushing the ecosystem forward from its stodgy enterprise-first origins. It’s unsettling that senior Developer Division leadership was willing to throw so much away to sell a few Visual Studio licenses.

Recently

Sun, 14 Mar 2021 11:15:38 -0700

It’s been a while since my last update. I wrapped up my sabbatical, did a whole bunch of interviewing, and have since been working on some interesting spreadsheet-related problems at a new job.

Oh, and there’s been a global pandemic on, it’s been weird. Anyway, I kept busy over the last year and a bit:

Switched to Windows

I switched to Windows full time, after a lifetime of Macs for personal use and a decade of Windows for professional use.

This was an experiment in intentionally shaping my own interests; I knew that if I used Windows at home I’d be more interested in the minutia of making good Windows applications.

It worked; I’m just over a year in, and I’ve learned a lot. I still miss a lot of things about macOS but I’m mostly happy on Windows. The Surface devices are really neat pieces of hardware.

ReiTunes

I built my own music player. This was the culmination of something I’ve been thinking about for years; I listen to a lot of long-form music and existing music services don’t serve that niche well. iTunes is progressively less interested in catering to users who bring their own music, and the Windows client has many irritating bugs.

So I decided to build my own music library system with an offline-first native client; how hard could it be? Well, 500+ commits later…

This was a fun opportunity to get my hands dirty with a distributed system; I designed it with easy push/pull metadata syncing across multiple clients. It also made me grapple more with the current state of client applications on Windows, which led to:

Windows native development deep dive - WinRT/UWP/WinUI/Project Reunion

I missed the culture of high-quality native apps from macOS, and I wanted to make ReiTunes feel like a proper native Windows application. After some agonizing, I decided to build it as a UWP application; I’d heard many complaints about the limits of UWP in the past, but the UWP applications I used on Windows often felt better than the alternatives (especially on touch devices).

UWP was largely rejected by people building enterprise applications for Windows (or to be more accurate, UWP rejected them) so it was mostly new to me. My feelings about the area are complex and I’ll probably write about them another time, but some high-level thoughts:

The Windows team is currently catching up from a lost decade with Project Reunion. This comment explains it well.
The development experience of WinUI is similar to working with WPF XAML. Some nice improvements have been made in binding and animation, but many pain points (styling, verbosity) still remain.
The Windows Runtime APIs are remarkably extensive; it’s sad how much work went into them only to go mostly unused.
It will probably be another year before WinUI 3 is useful in production scenarios, and I suspect that’s going to be too little too late. Blazor Desktop will make Electron-like applications much more attractive.

I would not recommend that most people learn UWP in 2021. You can deliver a user experience that is nicer than many alternatives, but there’s a lot to learn and it’s unclear how much will be obsolete soon. But conveniently, I was stuck at home with a lot of free time; it really gave me a better understanding of where Windows has been recently and where it’s going.

Overall, I’m mostly optimistic about the future of Windows. Some very good people are working on Project Reunion, in particular Rich Turner who drove a lot of the WSL and Windows Terminal+ConPTY work. I’m confident that Windows is moving in the right direction, but time will tell if it’s moving quickly enough.

Performance work

I’ve been doing a lot of .NET performance work and it’s been really fun. It’s satisfying to chip away at a slow process and speed it up bit by bit; I’ve made the main product I work on an order of magnitude faster, which starts to open up some cool possibilities for interactive use.

Interactive applications

This past year has given me the opportunity to build a few greenfield projects where interactivity and immediate feedback are critical. It’s been a steep learning curve; I mostly focused on back-end systems in the earlier stages of my career, and if I’m being honest I looked down a bit on front-end work as being easier. Was I ever wrong!

Trying to build performant interfaces has been a great learning experience; I’ve never had to deal with so many multithreading and concurrency problems before.

This has some nice synergy with the performance work.

Wat: a tiny CLI documentation tool

Tue, 18 Feb 2020 10:38:54 -0800

Robin Sloan’s post about “home-cooked” code has been making the rounds, and for good reason. It’s about highly personalized programs; when you build something purely for yourself or a small audience, it’s more like home cooking than industrial production. I really love the analogy, and it’s inspired me to write about one of my own “home-cooked” tools: wat.

The Problem

I spend a lot of my day in a terminal, and command line interfaces have notoriously poor discoverability. It’s easy to remember the handful of commands+flags you use on a regular basis; it’s much harder to remember the ones you only use a couple times a year. So, like many people, I used to spend a lot of time digging through man pages when 1) I know what I want to accomplish 2) I can’t remember the exact CLI syntax to make it happen.

Unfortunately, man pages usually aren’t designed for this. The average man page is more like a lengthy novel than Cliff’s Notes; it’s thorough+exhaustive documentation when I just want a quick reference.

There are a few excellent projects that provide succinct documentation for *nix commands: tldr and cht.sh are my favourites. But they don’t provide good facilities for adding personalized documentation of my own; that’s where wat comes in.

What Wat Does

wat started out as a simple wrapper around tldr and cht.sh. The two provide similar content, but they do so in different ways; tldr is installed locally, cht.sh is usually accessed through a dead-simple web interface: curl cht.sh/command. The first iteration of wat was just a simple fish function that would call tldr if it’s installed, and otherwise fall back on cht.sh.

These days, wat is written in Python and it’s a tiny bit more sophisticated: it also displays custom Markdown notes which are synced across all my machines using Git.

Say I want to get some information about a video file, but I can’t remember the exact ffprobe syntax. I just type wat ffprobe in my terminal:

This returns a handy tldr/cht.sh summary and it renders the contents of ~/notes/ffprobe.md if it exists. In this case, it looks like I got annoyed enough by ffprobe to write down a note for the -hide_banner flag.

I’ve been maintaining command line notes for a long time, in various tools that just happened to be at hand: OneNote, Notes.app, Markdown in a repo… but wat lets me organize that information better, and view it inline with excellent community-provided documentation.

How Wat Works

wat is embarrassingly simple; it’s a single-file Python script gluing together more sophisticated tools.

I sync wat and its notes across all of my computers with a Git repo and the excellent Dotbot tool. This is just piggybacking on my existing approach for dotfiles. The note files are rendered from Markdown using mdcat, and tldr pages are cached/accessed using the excellent tealdeer client. I’m also using the lovely sh module, which greatly reduces the friction of “shelling out” in Python.

Conclusion

wat solves CLI documentation for me. I have an unusually strong preference for Markdown, and I wanted something that would integrate nicely with my existing dotfile syncing. That’s all I really need. It might not work for everyone, and that’s totally okay! I’m ~~cooking~~ coding for an audience of one.

Vancouver Mapping Cheat Sheet

Sun, 02 Feb 2020 20:44:38 -0800

You might have already seen Jens von Bergmann’s interactive map of Vancouver which integrates a ton of different data sets. It’s really useful for looking up zoning, year of construction, tax data, and even the estimated amount of floor space in buildings. I find it essential for understanding the city.

It has also has a great feature that many people don’t know about: the ability to filter data for better visualization through the use of a query parameter in the URL. With Jens’s permission, I’ve compiled a list of ways to use this.

Basics

Say you want to see the impact of Vancouver’s early building boom, circa 1905-1914. Just add a years_1905_1914 filter to the URL like so: https://mountainmath.ca/map/assessment?filter=[years_1905_1914]:

And those are just the buildings still left standing! It’s pretty wild how Mount Pleasant, Grandview-Woodland, and Shaughnessy all sprung into existence in a decade.

If you wanted to see all buildings with an estimated floor space ratio (FSR) between 0 and 1, add a fsr_0.0_1.0 filter: https://mountainmath.ca/map/assessment?filter=[fsr_0.0_1.0]

Filters can easily be combined with a comma. If you wanted buildings that were built between 1900-1940 and are between 0 and 1 FSR, the URL would look like this: https://mountainmath.ca/map/assessment?filter=[fsr_0.0_1.0,years_1900_1940]

Area

This lets you filter on lot size (in square metres). Small lots between 100 and 300 square metres: https://mountainmath.ca/map/assessment?filter=[area_100_300]

Assessed Value

You can filter by building value (building), land value (land), and total value (total, building+land).

Lots with building value between $500k and $1M: https://mountainmath.ca/map/assessment?filter=[building_500000_1000000]

You can also filter on relative land value with the rlv filter. This shows lots with land value between $1000 and $2000 per square metre: https://mountainmath.ca/map/assessment?filter=[rlv_1000_2000]

Floor Space Ratio (FSR)

Floor Space Ratio or Floor Area Ratio is one measure of how dense a building is. This data is not exact - it is estimated by Jens based on Vancouver’s LIDAR survey of buildings. Buildings with an estimated FSR between 0 and 1: https://mountainmath.ca/map/assessment?filter=[fsr_0.0_1.0]

Geographic filters

If you want to see everything west of Ontario Street at -123.105118 degrees latitude, you’d use the westof filter: https://mountainmath.ca/map/assessment?filter=[westof_-123.105118]

Same idea for eastof_, northof_, and southof_.

Land Use

Jens has also imported Metro Vancouver data on land use. This is sometimes different from how the land is zoned - it shows how the land is actually being used, not what’s allowed. Some older properties may have grandfathered noncompliant uses.

You can filter for specific land use codes with the lu filter, or exclude codes with the nlu filter.

Here are all lots with “Residential - High-rise Apartment” uses: https://mountainmath.ca/map/assessment?filter=[lu_S135]

Land use codes:

S110: “Residential - Single Detached & Duplex”
S131: “Residential – Townhouse”
S130: “Residential - Low-rise Apartment”
S135: “Residential - High-rise Apartment”
S410: “Residential - Institutional and Non-Market Housing”
S200: “Commercial”
S230: “Mixed Residential Commercial - Low-rise Apartment”
S235: “Mixed Residential Commercial - High-rise Apartment”
S400: “Institutional”
A500: “Agriculture”
S300: “Industrial”
R100: “Recreation Open Space and Protected Natural Areas”
S420: “Cemetery”
U100: “Undeveloped and Unclassified”
M300: “Industrial – Extractive”
S120: “Residential – Rural”
S100: “Residential - Mobile Home Park”
S700: “Rail Rapid Transit Other Transportation Utility and Communication”
S600: “Port Metro Vancouver”
R200: “Lakes Large Rivers and Other Water”
S500: “Road Right-of-Way”
W400: “Protected Watershed”
S650: “Airport/Airstrip”
F100: “Harvesting and Research”
J000: “Recent Redeveloped / Misclassified”

Residential other than purpose-built rental

Per Jens, the priv_res filter does this:

Land use residential or mixed or affordable, but taking out cases where unit count=1 and not single family/duplex. So trying to filter out purpose-built rental.

https://mountainmath.ca/map/assessment?filter=[priv_res]

Teardown Coefficient

The “Teardown Coefficient” is a measure developed to estimate the chance that a building is likely to be torn down. Use the tdc filter.

This shows all buildings with teardown coefficient between 0.05 and 0.1 (so building value between 5% and 10% of total value): https://mountainmath.ca/map/assessment?filter=[tdc_0.05_0.1]

Years

This lets you filter by the year a building was constructed. Look how much the West End grew between 1960 and 1975: https://mountainmath.ca/map/assessment?filter=[years_1960_1975]

Zone

This lets you view specific zoning districts. If you want to see how many lots are zoned RS-1, Vancouver’s most common “single family”* zoning district: https://mountainmath.ca/map/assessment?filter=[zone_RS-1]

You can specify only the first part of a zoning code if you like. For example, if you want to see all RS lots not just RS-1: https://mountainmath.ca/map/assessment?filter=[zone_RS]

What if you want everything but a specific district? You can use “nzone” instead of “zone”: https://mountainmath.ca/map/assessment?filter=[nzone_RS]

Exact matches can be done using zoneE. This will only show RS-1 and exclude, say, RS-1A: https://mountainmath.ca/map/assessment?filter=[zoneE_RS-1]

If you want to include multiple zones you can separate them with underscores: https://mountainmath.ca/map/assessment?filter=[zone_RT-1_RT-5]

_{*I’m very aware that most RS lots can now have up to 3 suites (1 main, secondary, 1 laneway house). But it’s hard to describe that succinctly, and even the City of Vancouver still calls RS “One-Family Districts” and state that a major goal is to retain their “single-family residential character”.}

Immediate Feedback in Programming

Sun, 29 Dec 2019 19:01:49 -0800

Bret Victor’s talk Inventing on Principle (video, transcript) changed the way I think about computing in 2019. Inventing on Principle is partly about Bret’s guiding principle:

“Creators need an immediate connection to what they create. And what I mean by that is when you’re making something, if you make a change, or you make a decision, you need to see the effect of that immediately.”

The Edit-Compile-Run Cycle

Although Bret doesn’t use the term, programmers are deeply familiar with his principle. We’ve all worked with toolchains that introduce significant delay before you can “see” the results of a change, and we know they’re painful. Everyone wants a short edit-compile-run cycle.

But until IoP, I’d assumed that slow cycles wouldn’t materially change the output – you’d eventually get to the same place. This was wrong. I also didn’t appreciate the very small time scales involved; a 5 second delay used to seem trivial to me, but it’s still meaningfully different from a response time measured in milliseconds.

Through some very impressive custom tools, Bret shows how immediate feedback enables exploration, which then gives birth to ideas which would otherwise never see the light of day. This was an epiphany for me. Since IoP I’ve constantly been looking for better ways to code, and re-evaluating my existing processes for shorter feedback cycles. The results:

Rust

My typical Rust development workflow goes something like this:

Write a small function that does roughly what I want
Write a small unit test inline to exercise the function (even if it’s a private function)
Iterate using cargo test until the function is correct
Later, “productionize” the tests if necessary

Rust’s native support for inline unit tests helps a lot here, and the excellent type system catches a lot of issues before I even run cargo test. On the other hand, Rust’s compiler is notoriously slow and that extends to IDE tooling that depends on the Rust Language Server. I’m looking forward to Cranelift for faster debug builds.

JavaScript/TypeScript/Node

I use Nodemon to watch my files and automatically run a JS/TS file as soon as my code changes on disk. With pure JS, the feedback time is nearly instant. For TypeScript using ts-node, it tends to take a couple seconds – not ideal, but still pretty good. It’s a tradeoff; TypeScript catches bugs as I code and enables better IDE tooling, but JS enables immediate execution+feedback for quick iterations. The best choice depends on the nature of the task.

C#/.NET

This is an area where .NET’s recent cross-platform support stumbles a bit.

For .NET developers on Windows, NCrunch is a fantastic enabler of rapid feedback when coding. It’s a sophisticated automatic test runner; you make a code change, and it runs the relevant tests. It works very well, and has a lot of fans among the Domain Driven Design crowd. My only quibble is more to do with C#: private methods can’t be called by unit tests, which is (arguably) justified for production code but a hindrance for rapid exploration+prototyping. Visual Studio for Windows is also great; the tight integration with the Roslyn compiler allows for detailed code feedback before you even save a file.

However, these strengths don’t always extend to .NET development on other platforms. There is no serious equivalent to NCrunch for Visual Studio Code or VS for Mac; the VS Code extensions in that space are pretty unreliable. In theory, OmniSharp provides first-class language support for C# in VS Code; in practice, it needs to be restarted several times a day and I can never quite trust that I’m getting up-to-date feedback on my code as written.

The dotnet watch CLI tool can be quite useful in the same way as nodemon, but it typically takes a few seconds to compile even trivial applications. Hopefully it will get faster.

Golang/Hugo

I’m not an expert Go programmer, but I get a lot of exposure to it from Hugo (the templating engine I used to build this website).

I appreciate how fast the Go compiler is; speed is a primary goal for the Go team, and it shows. I can make a small change to a Hugo template, and see the results of that change in my browser nearly instantly. On the other hand, Hugo tooling in VS Code is poor, so I rarely get useful information from my IDE before saving a file. I intend to spend some time exploring better Hugo tooling.

(Aside: I’m also not convinced that Go is the right language for templating, because the lack of interfaces greatly complicates working with collections of structured data.)

Grab bag

IoP has changed my perspective on Unix scripting; I better appreciate the rapid iteration and observability of chaining multiple independent steps together with pipes. On the other hand, I get frustrated with the lack of typing; when every input is just raw strings, my development environment is unable to give me useful feedback before I run the script. I feel more warm and fuzzy toward PowerShell these days.

VS Code’s remote development feature is superb; it’s as if your IDE was running on a remote server. This eliminates a whole class of “copy the executable elsewhere to run it” delays (like when writing Linux-specific code on macOS or Windows).

I still really like Tailwind CSS, but it takes a few seconds to compile after configuration changes. This is a pain for rapid iteration on UIs, hopefully it gets faster.

What I've been up to: Nov-Dec 2019

Sun, 15 Dec 2019 11:48:48 -0800

I’ve been busy for the last month, and I completely forgot to update the blog. In no particular order, here’s what’s been occupying my time lately:

Coworking

I joined a coworking space in East Vancouver with a friend, and I’m working from there 3-4 days/week. It’s like an airy spacious café with fast internet and quiet space, I love it. I find that getting out of the house helps me be more disciplined with my working hours; I’m much less likely to disappear down a Wikipedia+YouTube rabbit hole at the office. And then when I inevitably do that at home, I feel less guilty about it because I’ve accomplished so much at the office.

Node.js

I got really into Node and the modern back-end JS ecosystem.

javascript.info has been remarkably helpful (I can finally remember exactly what a closure is!), it might be the first .info website that is genuinely a great source of information.

This talk by Franziska Hinkelmann on the V8 team is a great overview of JS engine internals.

I’ve been pleasantly surprised by Node APIs a few times; the Cluster module makes it trivial to fork workers and take advantage of multiple cores, and Buffer+friends make low-level bit manipulation quite pleasant. Which leads me to:

Cryptopals

The Cryptopals Crypto Challenges by Matasano Security are a delightful introduction to practical cryptography. I decided to do them in Node to get more practice with back-end JS, my solutions are on GitHub. The problems are very well-designed. They’re small enough that you can (usually) do each one in a single sitting, and it feels great every time you decode a ciphertext.

Raspberry Pi

I bought a Raspberry Pi 4 and have been loving it. For some reason I always thought of the Pi as only relevant for education and hardware hackers, but I was wrong – it’s a remarkably capable little Linux machine. I’m currently turning mine into a private Dropbox clone using ownCloud.

Anki

I started using Anki flashcards. I have decks for Node internals, uncommon JS syntax, and infrequently-used keyboard shortcuts and CLI options. Creating cards is a bit of an initial investment, but once that’s done I find that 5 minutes of daily study is enough.

Stupid Rust Tricks

Sun, 17 Nov 2019 17:09:46 -0800

Rust has a really powerful macro system. You can use it to do great things safely… or you can have fun with it and quickly prototype coding productivity features. I chose the latter.

todo-macro is a procedural Rust macro that stops compiling after a user-specified deadline. It’s like TODO comments, but with teeth. It’s probably best explained with an example:

Using todo-macro

It’s January 1, 2020. I’m working on some Rust code that compiles, but it’s not quite ready to ship.

I want to take a break, but I know myself – I’ll probably forget about the deficiency. I could add a TODO comment, but that depends on me actively searching for TODO comments next time I open the project.

To save me from myself, I add a quick todo macro with a deadline of January 2 (in ISO 8601 format):

// Implement the timeout handling
todo!("2020-01-02")

That compiles for now, but as soon as the deadline is passed (i.e. our system clock returns Jan 3), builds start failing:

error: proc macro panicked
 --> src/main.rs:5:5
  |
5 |     todo!("2020-01-02");
  |     ^^^^^^^^^^^^^^^^^^^^
  |
  = help: message: Tsk tsk. You missed your deadline.

Think of todo-macro as a reminder that actively forces your future self to deal with a problem.

Obviously, don’t use this in real projects unless you’re really comfortable with non-deterministic builds (I’m not). But still, wouldn’t it be nice to have a (safe) feature like this in your favourite IDE?

Linux .NET Development in 2019

Tue, 05 Nov 2019 08:51:59 -0800

I’ve recently been building .NET Core back-end services that run on Linux. Linux .NET development is in an interesting place; it’s clearly the future of back-end .NET, but it’s still a little rough around the edges compared to our old friend .NET-on-Windows.

Let’s dive into what you (an experienced Windows .NET developer or a .NET-curious Linux developer) need to know to start building .NET services for Linux. I’ll cover IDEs, service hosting, Linux system calls and more.

Background & Motivation

.NET development for Linux has been possible via Mono since 2004, but it was always a bit… fringe compared to Windows .NET development. That all changed when Microsoft released .NET Core in 2016 as a cross-platform .NET implementation; first-party support from Microsoft is a big deal to most .NET developers.

We’re also in a world where Linux is the lingua franca for back-end development; if you want to do anything involving cloud services, distributed systems, or containerization, Windows is typically an afterthought (if it’s supported at all). I want to skate to where the puck is going (sorry!), and it’s headed toward Linux.

Which IDE should I use?

You can eschew an IDE and do everything with the dotnet CLI tool, but personally I want more tooling. There are 4 big IDEs for modern .NET development:

Visual Studio for Windows
Visual Studio Code (Windows, Linux, Mac)
Visual Studio for Mac
JetBrains Rider (Windows, Linux, Mac)

The best choice for Linux development will depend on your personal OS and the specifics of your application.

Visual Studio for Windows is the one to beat, it’s got excellent .NET tooling. However, it doesn’t have a great story for remote Linux .NET development; to test Linux-specific functionality, you’ll need to deploy to a remote system after compiling. Not the end of the world, but I prefer a tighter edit-compile-debug loop. Interestingly, VS support for remote Linux C++ development has seen a lot of investment recently; it’s a safe bet that this feature will come to .NET eventually.

VS Code supports remote development on a Linux server, and it’s a killer feature. I just point VS Code at my Ubuntu VM, and bam, it’s like I’m on a Linux workstation. Debugging and file browsing just work. The downside is that VS Code’s C# tooling is much less mature than “real” VS. I need to restart the OmniSharp language server multiple times a day (Microsoft employees complain about this too!), and you’ll need to fall back on the CLI frequently.

VS for Mac is the descendant of Xamarin Studio. It has much better .NET tooling than VS Code, but no remote development feature. I’m on a Mac and I still prefer VS Code most of the time, but YMMV.

JetBrains Rider runs natively on Mac, Windows, and Linux. I’m told that it’s a great choice if you run Linux as your main OS, but it doesn’t have a remote development feature.

Systemd Services

If you want a long-running service on Windows, typically you host your code in a Windows Service. The equivalent on Linux (for most distros) is a systemd service. It’s really easy to write a systemd service in .NET Core using the Microsoft.Extensions.Hosting.Systemd package:

Host your application using the .NET Generic Host (HostBuilder)
1. The dotnet new worker template is the easiest way to do this from scratch
Add the Microsoft.Extensions.Hosting.Systemd NuGet package to your project
Call the UseSystemd() extension method on your HostBuilder
Create a systemd unit file

public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .UseSystemd()
            .ConfigureServices((hostContext, services) =>
            {
                services.AddHostedService<Worker>();
            });

Once you do that, the service acts just like any other systemd service; manage it with systemctl, view its logs with journalctl, etc.

Microsoft.Extensions.Hosting.Systemd isn’t magic, and it’s surprisingly easy to implement your own systemd layer if you need to; under the hood that library is just making a Linux syscall, reading from /proc, and writing to a Unix Datagram socket.

Syscalls

Sometimes you want to do platform-specific things, right? Good news, that’s (usually) easy.

You can make syscalls directly using P/Invoke. Here’s how to call getpid to get the current process ID:

// leave EntryPoint off if you want it inferred from the name of the C# function
[DllImport("libc", EntryPoint = "getpid")]
private static extern int getpid();

However, most of the time you’ll want to use the Mono.Posix.NETStandard package instead. It handles the thankless work of wrapping native calls safely. Don’t be turned off by the Mono namespace; Mono.Posix.NETStandard is the current Microsoft-blessed way to do syscalls.

Update: the new NativeLibrary class in .NET Core 3 is a handy alternative to P/Invoke that can be configured at runtime.

Miscellaneous

Sometimes you’ll run into situations where common .NET APIs don’t handle *nix features well. For example, the .NET I/O APIs weren’t designed for symlinks:

FileInfo does not resolve symbolic links, leading to subtle bugs. There’s no API that gives information about the link target.

I expect that this will get better as .NET development for Linux becomes more mainstream.

Open-Source Projects for Learning Modern C and C++

Sun, 20 Oct 2019 09:52:25 -0700

Every now and then, I wonder “Is low-level programming still unpleasant?” Call it a morbid fascination, but I really did enjoy some aspects of working in C and C++; in particular, reasoning about low-level behaviour becomes a lot easier with fewer layers of abstraction on top of the metal. On the other hand: memory management and interoperability are hard, the C/C++ ecosystems have accumulated decades of cruft, and both languages are missing a lot of features that are now par for the course.

It turns out that the short answer is “No, because Rust is great”, but it’s still useful to get a feel for modern C and C++. Here are some projects and tools that I’ve found particularly indispensable for that.

Preface: Why Not Book Learnin'?

There are a lot of books out there that will teach you how to use newer C++ features. Effective Modern C++ is a popular one, but personally, that’s not what I’m looking for. I know that I learn best from hands-on experience, and so what I really wanted was to play with some decent-sized example projects using modern tooling.

CMU’s Research/Educational Databases for C++

I can’t say enough good things about CMU’s Database Group, and specifically professor Andy Pavlo’s work. CMU has 3 (yes, 3!) open source relational databases written in modern C++ on GitHub:

BusTub, an educational system written for the Database Systems course
Terrier, CMU’s current research database
Peloton, CMU’s older research database

BusTub and Terrier use C++17, but Peloton uses C++11. I’d recommend BusTub and Terrier since they’re both under active development. Terrier has excellent documentation for getting up and running on the wiki. They recommend using CLion, a modern C/C++ IDE from JetBrains.

CMU’s even provided some small pieces of work to dip your toes in; the projects for the Database Systems course involve implementing basic functionality in BusTub, and grading scripts for the projects will be available soon.

Quantum Mechanical Keyboard Firmware (QMK)

I’m a bit of a keyboard dork, so of course I jumped at the chance to program my keyboard’s firmware.

QMK is a very popular open-source keyboard firmware project , mostly written in C. QMK’s documentation is outstanding, and will help you get up and running with hardly any fuss. I was able to implement a custom keymap in C in just a few hours, armed with only the C I remembered from school 10 years ago.

New Project: ETL Sheets

Tue, 15 Oct 2019 07:32:49 -0700

I recently started experimenting with building tooling for ETL systems. After many years of wrestling with ETL in industry, I had a few questions on my mind:

Can we make common data issues quick to resolve?
Can we make automated data transformations as easy to work with as spreadsheets?

I think the answer to both questions is yes, but don’t take my word for it – you can try the prototype at etlsheets.netlify.app (double-click on an issue to get started), and view the source code.

Motivation

Importing data at scale is painful. Your data providers will screw up the formatting, systems will experience connectivity issues, and your transformation logic will fail on cases you didn’t expect. What if our tools focused on helping with those failures, instead of assuming the happy path?

Speaking of transformations, how should we write them? Some systems take a code-first approach, which is great for coders and impenetrable for everyone else. Others take a GUI-driven approach, which usually becomes the stuff of nightmares. I think we can do better, by drawing inspiration from a tool that’s found in every office:

Transformations

Here’s what building a new transformation might look like, and here’s what it might look like when that transformation fails to run successfully.

This works very much like a spreadsheet, albeit with visually distinct stages/steps. New columns can be created by entering a simple formula, and existing rows can be filtered+altered with simple row-level formulae. Significantly, formula changes are reflected immediately in the data – this might seem like a trivial feature, but I think it’s a key part of usability. Bret Victor’s made an entire career out of his “creators need an immediate connection to what they’re creating” principle, and for good reason.

Under the hood, formulae are written in JavaScript with some syntactic sugar (to make it easier to refer to columns) and macros (for common uses like filtering). I’m not tied to JS in particular, but using a “real” programming language gives us a lot more power and flexibility. Interoperating with existing libraries+code becomes trivial.

Try opening up the failed extraction. Our counterparty has sent us an identifier with a typo (MSFY instead of MSFT), but we can just fix the data inline.

Extractions

Extractions are a bit less interesting than transformations, but we can go a long way with some relatively simple interfaces. We know that connectivity and formatting issues are common, so when they happen let’s show users exactly where the problem is:

Next Steps

To be honest, I’m not 100% sure where to take this next. The prototype makes for a great demo, but it would need a lot of work to make it production-ready. I currently run JS formulae in-browser because that was quick to develop, but I’d want to evaluate other languages and build out server-side execution. Change management needs some thought too; given that these transformation definitions are essentially code, do we serialize them and put them in source control?

These challenges are definitely surmountable, but they do require a lot of thought+work and I’ve mostly moved on to other projects. If you have any ideas or suggestions, give me a shout!

Managing Postgres with Pulumi and Terraform

Thu, 19 Sep 2019 13:52:18 -0700

I’m a big fan of Pulumi, and I’ve previously used it for cloud infrastructure. That’s their main selling point; Pulumi’s slogan is literally “Program the cloud.” Interestingly, Pulumi can also be used to manage on-premise infrastructure, including PostgreSQL databases. Let’s dive into the details.

Pulumi basics

Pulumi takes a declarative infrastructure definition and handles provisioning said infrastructure, just like Terraform, AWS CloudFormation, and Azure Resource Manager. The main difference is that in Pulumi you’re writing real code (Node, Python, or Go) to build that infrastructure definition instead of a YAML or JSON file, it’s great.

Pulumi/Terraform PostgreSQL Provider

After Pulumi has executed your code to build up an infrastructure definition, it needs to interact with external infrastructure resources (cloud providers, databases, etc.) to turn that definition into a reality. Some of Pulumi’s functionality for working with external resources is derived from Terraform providers. For example, Pulumi has a PostgreSQL provider which is derived from the Terraform PostgreSQL provider.

At a high level, the PostgreSQL provider lets you define databases, roles, schemas, permissions, and PostgreSQL extensions. It stops short of letting you manage DDL objects and data within the database (which is probably for the best given the complexities of schema+data migrations).

Defining a Postgres database with TypeScript

Pulumi can be used to instantiate entire Postgres clusters, but for this tutorial let’s assume you already have an existing cluster managed outside of Pulumi. It can be running anywhere: in the cloud, in a container, on your local machine, whatever.

First, install Pulumi then instantiate a new TypeScript project with pulumi new typescript. This scaffolds a bare-bones Node+TypeScript Pulumi project with the following files:

index.ts                package-lock.json       tsconfig.json
Pulumi.yaml             node_modules            package.json

Install the Pulumi PostgreSQL provider with npm i @pulumi/postgresql (or just edit your package.json directly).

Next, we need to tell Pulumi which Postgres cluster to connect to and how. The PostgreSQL provider’s configuration points are documented here. My Postgres cluster is running on a local VM with the hostname fedora-vm, so I set the postgresql:host variable like so:

pulumi config set postgresql:host fedora-vm

This creates a YAML configuration file for the current Pulumi stack, or modifies one if it already exists. For my dev stack, this creates a file named Pulumi.dev.yaml with the following contents:

config:
  postgresql:host: fedora-vm

You’ll want to do the same for postgresql:username and postgresql:password, to tell Pulumi which credentials to use. Sensitive configuration values like passwords can be encrypted using the --secret flag.

Finally, we’re ready to work with the fun stuff: configuring a database in real TypeScript code. Let’s say we want to instantiate a Pulumi-managed database, create a role with login permissions, and grant the role SELECT permission on tables in the public schema in the new database. This can all be done in the index.ts file like so:

import * as pulumi from "@pulumi/pulumi";
import * as postgresql from "@pulumi/postgresql";

const config = new pulumi.Config();

const managedDatabase = new postgresql.Database("managedDatabase", {
  name: "pulumi"
});

const publicReaderRole = new postgresql.Role("publicReaderRole", {
  login: true,
  name: "public_reader",
  password: config.require("publicReaderPassword")
});

const publicReaderSelectTablesGrant = new postgresql.Grant(
  "publicReaderSelectTablesGrant",
  {
    database: managedDatabase.name,
    objectType: "table",
    privileges: ["SELECT"],
    role: publicReaderRole.name,
    schema: "public"
  }
);

It’s fairly self-explanatory, but some points of interest:

We can retrieve arbitrary configuration values, even encrypted ones, using config.require().
1. Use config.get() for optional configuration that might not be present in every stack.
It’s a bit weird that we write names like managedDatabase twice, right?
1. The names in the constructors are Pulumi resource names, which Pulumi uses to track resources across multiple deployments.
2. The TypeScript variable names could be anything, Pulumi doesn’t use them outside of when our code is run.
3. It’s a bit awkward that we need to repeat ourselves – in most cases I don’t have any reason to name Pulumi resources differently from the variables that represent them. Perhaps Pulumi should use reflection to find default resource names, or perhaps I should come up with a better naming convention.

Deployment

Once we have something like the above, deployment is trivial with the pulumi up command. It runs our TypeScript program then provides a nice visual preview of the resources that will be created:

Selecting yes will create the database, role, and grant in just a few seconds. With just a little bit of work, we’ve defined our core database infrastructure as real code.

Future Imperfect 2.0

Sat, 14 Sep 2019 00:00:00 +0000

I spent a few weeks in August rewriting this website, and as promised here are the deets. The source code is available here under the MIT license.

Background

When I initially put this website together in January 2018, I used Julio Pescador’s Hugo port of the Future Imperfect theme. I’d heard good things about Hugo, I wanted to write blog posts in Markdown, and it was a good-looking theme that required minimal additional setup. It served me well for over a year, but a few things kept bothering me:

The styling was very difficult to modify – it was nearly 3000 lines of CSS in a single file, with many duplicated colours and styles.
The theme relied on a lot of JavaScript libraries: jQuery, highlight.js, Fancybox, Skel…
1. My simple static website was serving up hundreds of kilobytes of largely unnecessary scripts. As someone who grew up using a dialup modem, this offended me.

My goal was to rewrite the Hugo theme for extensibility and performance, and I figured it would take maybe a week. Of course, it took about 3 times that.

Major Changes

I rewrote the CSS using Tailwind CSS, because Tailwind is great.
1. Tailwind’s generated CSS is automatically pruned using PurgeCSS as part of a PostCSS workflow to keep file sizes down.
2. The layout looks roughly the same as before, but I’ve made a lot of small tweaks.
Dark mode using the new prefers-color-scheme media query and the Solarized colour scheme.
I rewrote the side-menu JavaScript because it used jQuery extensively.
The full FontAwesome icon set has been replaced with a custom font from IcoMoon that only contains the icons used in the website.
1. Yes, I went a bit overboard with performance.
I replaced the Fancybox lightbox with a customized Flickity implementation.
1. This allows for nicer fullscreen interactions, image carousels, and best of all – no jQuery.
I replaced the highlight.js client-side code syntax highlighting with Hugo’s compile-time highlighting.
The theme now uses Hugo’s built-in bundling and minification for CSS and JS files.
Image hosting+transformation+optimization is now handled by Cloudinary
1. I strongly considered using Netlify Large Media, but the requirement to use Netlify for continuous deployment scared me away – I want to avoid vendor lock-in.
Generated HTML is automatically formatted using Prettier.
CSS is automatically linted using Stylelint.
There are new shortcodes for images, email obfuscation, card elements…
Site search is now done with DuckDuckGo instead of Google.
The elaborate social sharing functionality has been removed, because my target audience is technical people who would rather just copy a URL.

Inspiration

Tom MacWright’s explanation of how he optimized his personal website was a major inspiration, and Carlos Bueno’s Mature Optimization Handbook helped curb the worst of my tendency toward premature+unnecessary optimization. My former coworker Colin Bate helped with ideas – check out his Develop.cheap newsletter for great recommendations.

UI components: Angular vs React

Mon, 09 Sep 2019 00:00:00 +0000

I’ve been building web UIs in Angular and React for the last few years, and I’ve started to greatly prefer React. Extracting and using UI components is just easier and, for lack of a better word, more JavaScripty.

Extracting components in React

Say I notice that I’m creating multiple <span> elements with the same class and icon:

function Main() {
  return (
    <div>
      <span class="alert-tag"><i class="fa alert"></i>foo</span>
      <span class="alert-tag"><i class="fa alert"></i>bar</span>
      <span class="alert-tag"><i class="fa alert"></i>baz</span>
    </div>
  );
}

It’s trivial to refactor this repeated element into its own <Alert> component function within the same file:

function Main() {
  return (
    <div>
      <Alert>foo</Alert>
      <Alert>bar</Alert>
      <Alert>baz</Alert>
    </div>
  );
}

function Alert({ children }) {
  return (<span class="alert-tag"><i class="fa alert"></i>{children}</span>);
}

Super easy, uses JavaScript’s native language constructs, and I was able to do it all within the same file (but I can easily move Alert() elsewhere for wider re-use if needed). It’s just like extracting a function in a “regular” programming language, it’s something you do without even thinking about it.

Extracting Components in Angular

First, we have to create the new component. The most common way to do this in Angular is with Angular CLI and its various ng commands. By default, the command to create a new component (ng generate component Alert) literally touches 5 files:

CREATE src/app/alert/alert.component.css (0 bytes)
CREATE src/app/alert/alert.component.html (23 bytes)
CREATE src/app/alert/alert.component.spec.ts (614 bytes)
CREATE src/app/alert/alert.component.ts (261 bytes)
UPDATE src/app/main.module.ts (720 bytes)

Yikes – that’s definitely overkill for our tiny helper component! Thankfully we can get that down to 2 with the following parameters instead: ng generate component Alert --inlineStyle=true --inlineTemplate=true --skipTests=true

Once we’ve done that, the component file still has an awful lot of boilerplate:

import { Component, OnInit } from '@angular/core';
 
@Component({
  selector: 'alert',
  template: `<span class="alert-tag"><i class="fa alert"></i><ng-content></ng-content></span>`,
  styles: []
})
export class AlertComponent implements OnInit {
  constructor() { }
  ngOnInit() {
  }
}

That can be trimmed down a bit:

import { Component } from '@angular/core';
@Component({
  selector: 'alert',
  template: `<span class="alert-tag"><i class="fa alert"></i><ng-content></ng-content></span>`
})
export class AlertComponent {}

… but we also need to reference the new component in our module file, even if it’s a tiny helper within another component:

import { MainComponent, AlertComponent } from './main.component';
@NgModule({
  declarations: [
    MainComponent,
    AlertComponent
  ],
...

These steps aren’t impossible. They’re negligible for large components, to be honest. Where they become a problem is small components, especially those which are only used within a limited context. The fixed cost of extracting a component is simply much higher in Angular than in React, which limits our ability to write small components in a Clean Code style.

Using Extracted Components

To make matters worse, Angular components often need to be used with Angular-specific syntax instead of standard JavaScript. If you want to generate an <Alert> for every variable in a collection you’d use ngFor instead of a standard JavaScript function like map(). You might also use ngIf, ngSwitch, or Angular’s not-exactly-JavaScript template expressions. It’s not good that Angular is reinventing fundamental language functionality.

Closing thoughts

Working with components in Angular just involves so much ceremony compared to React. I’m always aware that I’m working within the constraints of a framework, whereas React feels like I’m working in regular JavaScript (after the brief introduction to JSX, anyway). I know which I prefer.

Relational Database History

Mon, 02 Sep 2019 16:43:23 -0700

I recently read 2 papers that I’d highly recommend if you ever wonder about the underpinnings of modern relational databases:

What Goes Around Comes Around (Stonebraker and Hellerstein, 2005)

This is an opinionated history of database architectures from roughly 1970 to 2005, and it’s got some serious credentials behind it – Michael Stonebraker was the driving force behind Postgres and many other things. It’s an easy read, and it’s a good explanation of how relational DBs and SQL came to (mostly) rule the world. It was written amidst a lot of hype over XML databases, and it (rightly, in retrospect) critiques XML DBs and suggests that they won’t see much commercial adoption.

One interesting bit: Stonebraker and Hellerstein do not like SQL, and they claim that it became dominant largely because of IBM’s dominance in the marketplace during the 1980’s. Which leads us to the next item…

A Critique of the SQL Database Language (Chris Date, 1983)

This was cited approvingly in the 1st paper as a “scathing critique of the semantics of SQL”, which caught my eye - virtually every database supports SQL, how bad can it be?

I won’t summarize the entire thing, it’s a laundry list of complaints, but this resonated with me:

Notice that it is just table-names that appear in the FROM clause. Completeness suggests that it should be table-expressions (as Gray puts it, “anything in computer science that is not recursive is no good”).

…a simple table-reference (i.e. a table-name) ought to be just a special case of a general table-expression

To give a (trivial) example, say we want to union 2 tables and extract a column. We’d write that like this in SQL:

SELECT EMPLOYEE_ID FROM EMPLOYEES
UNION
SELECT EMPLOYEE_ID FROM CONTRACTORS

But if table names were just special cases of table expressions, we could write it like this instead:

SELECT EMPLOYEE_ID FROM (EMPLOYEES UNION CONTRACTORS)

SQL gets the job done, and it could be a lot worse - but it is a little sad that many of the issues academics have known about for 35+ years still exist today. It’s a reminder of how entrenched something that’s only “good enough” can become.

Dark Mode on the Web

Tue, 20 Aug 2019 16:08:09 -0700

I recently rewrote most of this website (more on that soon!) and it’s now much easier to work on. Last week, I shipped a feature that I really like: dark mode using the new prefers-color-scheme media query. If you have dark mode enabled in your OS and a reasonably new browser, reillywood.com now shows you a dark UI using the Solarized colour scheme:

Dark mode

Light mode

This is really easy to enable, and I hope more sites start using it soon. Under the hood, it’s just a straightforward media query in CSS:

@media (prefers-color-scheme: dark) {
    // Dark style rules go here
}

I borrowed a trick from Mark Otto where I dim images slightly in dark mode:

@media (prefers-color-scheme: dark) {
  img {
    opacity: 0.75;
    transition: opacity 0.5s ease-in-out;
  }
  img:hover {
    opacity: 1;
  }
}

Dark mode in Tailwind

Integrating it with Tailwind CSS was pleasantly simple. I added a custom dark media query in my Tailwind config like so:

extend:{
  screens:{
    'dark': {'raw':'(prefers-color-scheme: dark)'}
  }
}

And once that’s in place, I use it with a dark: prefix on my Tailwind classes:

// This sets a white background normally, and a black background in dark mode
<div class="bg-white dark:bg-black">
...

Browser support

For detailed information on browser support for prefers-color-scheme, caniuse.com is the place to go. It’s supported in the latest versions of Safari, Firefox, and Chrome, but it is on the bleeding-edge – Firefox and Chrome only shipped support for it last month, and dark mode itself is only supported in recent versions of macOS+Windows.

Still, it’s a nice progressive enhancement – any browser that doesn’t support it will just render the “regular” website styling. You can start using prefers-color-scheme now, even if most of your audience is stuck on older browsers.

I'm taking a sabbatical

Wed, 14 Aug 2019 00:00:00 +0000

I’m taking a self-funded sabbatical after 8 years of working full-time. It feels like the right thing to do at this stage of my career.

I’ve had an unusually stable career for a software developer. When I graduated from university in 2011 I interviewed at a few places, including a mid-sized investment firm. I didn’t know anything about finance and the… rustic state of their website was a little concerning, but the people seemed great. So I took the job, and told myself I’d stick around for 2 or 3 years.

8 years later I was still at the same company. Orbis offers a lot of different opportunities; I wrangled big financial data systems, ran a small team, got my hands dirty building a modern web stack, and worked in London and Cape Town. It was a blast, and I’d highly recommend Orbis as an employer.

Still, after 8 years, it’s time to try something new. I miss the open-ended learning that’s so common in school, and I want to explore my technical interests with no regard for immediate relevance to my day job. Gianfranco Chicco’s description of a “serendipity break” really resonated with me:

In the note I sent out to my friends and network I mentioned that I’d be undertaking a Serendipity Break, which wasn’t a nice way to say that I wasn’t going to work for a few months but that I wanted to actively explore different possible paths.

In the book The Craftsman, sociologist Richard Sennett describes how “skill builds by moving irregularly, and sometimes by taking detours”, which is akin to keeping the Serendipity Engine in perpetual motion to encourage the strengthening of current skills and allowing the development of new ones.

Leaving a great job at a great company was ~~a little~~ scary, but I think it’s necessary for my long-term growth. Reading about Joel Spolsky’s sabbaticals helped a lot; it’s reassuring to see successful developers following similar paths.

My last day at Orbis was July 27th, and since then I’ve been trying all kinds of things. I’ve been diving into database internals, rewriting this website, and even learning Lisp/Scheme. I’m not exactly sure where my interests will take me next, but I’m looking forward to finding out.

Quick T-SQL Regression Testing

Mon, 10 Jun 2019 18:40:45 -0700

Here’s something I’ve found useful in SQL Server, but it should apply in any DBMS with checksum functions. Next time you’re refactoring some database objects and you want to do some regression testing, give the CHECKSUM (MSDN) and CHECKSUM_AGG (MSDN) functions a try.

CHECKSUM and CHECKSUM_AGG

They behave pretty much as you would expect; CHECKSUM returns 1 checksum given 1 row, and CHECKSUM_AGG is an aggregate function that returns 1 checksum given multiple checksum rows. Between the two of them, you can get a checksum for any arbitrary collection of data:

-- Returns 2 rows with 2 columns
SELECT Field1, Field2
FROM (VALUES ('foo','bar'),('baz','foo')) TempTable(Field1,Field2)

-- Returns 2 rows with 1 checksum column
SELECT CHECKSUM(Field1,Field2)
FROM (VALUES ('foo','bar'),('baz','foo')) TempTable(Field1,Field2)
 
-- Returns 1 row with 1 checksum column
SELECT CHECKSUM_AGG(CHECKSUM(Field1,Field2))
FROM (VALUES ('foo','bar'),('baz','foo')) TempTable(Field1,Field2)

Regression testing

These help with a common problem: when you’re refactoring a database object, how can you be sure that you haven’t changed the existing functionality? Sometimes it’s enough to spot check a few inputs, but often the logic is complex enough that this does not provide a high level of confidence. With CHECKSUM and CHECKSUM_AGG, you can quickly check large numbers of test cases.

For example, say we have a TVF called MyDateTVF that takes in a date parameter and returns columns Column1 and Column2. I want to refactor this function and then test that the functionality is unchanged for every date in 2018. If we have a table called DateList which contains every date (more useful than you might expect in a DB with a lot of temporal aspects), I can simply run the following before and after my change:

SELECT CHECKSUM_AGG(CHECKSUM(MDT.Column1, MDT.Column2))
FROM DateList DL
CROSS APPLY (
    SELECT Column1, Column2 from dbo.MyDateTVF(DL.Date)
) MDT
WHERE DL.Date BETWEEN '2018-01-01' AND '2018-12-31'

If that query returns the same checksum before and after my change, we can rule out any regressions for those inputs. Nice and easy!

Tailwind CSS

Mon, 18 Feb 2019 07:50:59 -0800

I recently overhauled the UI for my letter builder web app, switching from Bootstrap to a neat framework named Tailwind CSS. It’s been great so far.

I dabble in web development, but it’s not “my thing”. Most of my time is spent on back-end systems and the occasional native UI. When I’m building a web UI, I usually spend a lot of time on MDN or W3Schools looking up syntax details.

Tailwind

Enter Tailwind. It’s a utility-first CSS framework, which in practice means they give you a whole bunch of “utility” classes which are effectively aliases for common styles. Here’s an example from when I was styling a button:

<button class="bg-blue-dark text-sm text-white rounded-sm py-2 px-4 my-2"/>

With these classes, I’m saying that I want a dark blue background, small white text, 2 units of padding on the top and bottom (py-2), 4 units of padding on the left and right (px-4), 2 units of margin on the top and bottom (my-2), and rounded corners. Whew!

Right away, we can notice a few advantages relative to the normal way of doing things:

This is fast to iterate on. I can add another utility class in-line without digging through my CSS file and reasoning about selectors.
The class names are concise yet informative. py-2 is clearly operating on the Y axis (top and bottom), which is much easier to remember than a padding style with multiple unlabeled values.
I have fewer choices to make. I didn’t need to worry about whether to use #0033CC or #0000FF for a dark blue, I just asked for dark blue. Same thing with padding+margins, I just chose from a small set of integers instead of wondering whether to do 0.5rem or 0.6rem.

Isn’t putting all your styles inline just asking for trouble?

Yes, this can get out of hand if you put too much stying inline – but Tailwind utility classes can be extracted into components as soon as you need some additional abstraction. The authors even recommend that:

Tailwind encourages a “utility-first” workflow, where new designs are initially implemented using only utility classes to avoid premature abstraction.

While we strongly believe you can get a lot further with just utilities than you might initially expect, we don’t believe that a dogmatic utility-only approach is the best way to write CSS.

Curation VS organic growth

I’m being a little hard on the “usual” way of writing styles in CSS. The web grew more-or-less organically, so it’s not fair to expect an overall design from the mess of styles available to modern browsers. Thankfully, that’s where projects like Tailwind come in.

It’s remarkable how much nicer it is to build web UIs when working with a thoughtfully curated and documented subset of styles. Sometimes a big step forward doesn’t need to come from a clever technical breakthrough – simply organizing existing information+symbols in a better way can reap massive benefits.

Pulumi for Cloud Infrastructure Management

Tue, 07 Aug 2018 16:03:34 -0700

I’ve recently been playing around with some cloud development tooling by a startup named Pulumi, and I thought I’d write up my first impressions.

Pulumi can be summed up as “Infrastructure as Code – but really, we mean it this time.” Instead of mucking around with YAML files and proprietary syntax, you define the infrastructure you need in actual code (JavaScript or Python, with more languages coming later). Is your production environment slightly different from your test environment? No problem, that’s just an if statement (or a more complex abstraction) away.

It’s still very early days (as of this writing, Pulumi is on version 0.14.3), but the general approach shows a ton of promise.

Pulumi basics

I first heard about Pulumi from this ‘Tooling in the age of serverless computing’ talk by Donna Malayeri (of Pulumi, formerly of the Azure Functions team at Microsoft). It’s a great overview of serverless tooling options, and I learned a lot from the slides. Best of all, there are code samples for each tooling provider available on GitHub. Let’s take a look at the Pulumi ones.

You can work at a few different levels of abstraction with Pulumi. You can write a serverless function with complete control over every aspect of AWS Lambda and API Gateway. If that’s too nitty-gritty, you can use @pulumi/aws-serverless to simplify configuration of API Gateway+Swagger. If you like, you can even move your Lambda function’s code into the Pulumi script.

Then, at the highest level of abstraction, you can use @pulumi/cloud to define cloud components that aren’t tied to any particular cloud. This isn’t fully built out yet – it’s only been implemented for AWS at this time – but it’s by far the most exciting to me. Seriously, take a look at the example where Donna defines a NoSQL DB table and a serverless function that uses it in just 24 lines of code.

My motivation for using Pulumi

Most of my hobby coding in the last year and a bit has been done using AWS Lambda and Azure Functions. I’m using Serverless Framework for the most complex one, and it’s been OK. It provides a layer of abstraction around Lambda and other AWS resources, which is nice. What’s not so nice is that your infrastructure definition is moved to an ever-growing YAML file with unpleasant syntax once you go beyond very basic scenarios.

This is how to define an S3 bucket conditionally in Serverless Framework, i.e. if and only if an environment variable named S3_LOGGING_BUCKET is not empty:

  Conditions:
    CreateS3Bucket: 
      Fn::Not:
        - Fn::Equals:
          - ${file(env.${opt:stage}.yml):S3_LOGGING_BUCKET}
          - ""
  Resources:
    S3LoggingBucket:
      Type: "AWS::S3::Bucket"
      Condition: "CreateS3Bucket"
      Properties: ...

This is… unpleasant. It’s verbose, it’s not especially well documented, and I probably spent an hour of my life figuring out how to accomplish this conceptually trivial thing. When I first read about Pulumi, a lightbulb went off. “You mean I could have written that logic with an if statement instead? Sign me up!”

if(environment.S3_LOGGING_BUCKET != "") {
    // provision S3 bucket
}

My experience with Pulumi

Over a few weekends, I attempted to migrate my Serverless-managed Node function to Pulumi. I’ve run into a few barriers and so I haven’t been able to successfully finish that yet, but it is still very early days for Pulumi. Pulumi is progressing at an impressive rate and I’m fairly confident that I’ll be able to accomplish this in the not-so-far future.

I was hoping to use the highest level of abstraction, and so I started using Pulumi’s abstract API class (formerly HttpEndpoint). It all went well until I started trying to use other Node packages in my function. Turns out this is a scenario where 1) you need to do something a little nonstandard to import modules, 2) the recommended approach wasn’t documented. Pulumi team members were super responsive and helpful in explaining how to fix this.

I’m now able to import+use Node packages in my function, but then I encountered an issue with POST APIs. The short version is that standalone POST calls work, but modern browsers ‘preflight’ POST calls with an OPTIONS call and OPTIONS calls fail even when I try to handle them explicitly. I’m guessing this is a Pulumi bug/missing feature that will be implemented eventually.

Next steps

I’m considering ditching the higher-level @pulumi/cloud API abstraction and just using Pulumi to upload a Lambda deployment package. This will require some further investigation – I need to figure out the best way to put together a transpiled-from-TypeScript deployment package that only contains my production npm dependencies. I don’t think that will be especially difficult, but it needs to wait until I have another spare weekend afternoon. Perhaps the POST issue will get sorted out before that. :)

The State of Serverless

Tue, 22 May 2018 22:00:43 -0700

In my spare time I have been mucking around with 2 big Functions-as-a-Service (FaaS) offerings, AWS Lambda and Azure Functions. I’ve been meaning to write up a “The state of serverless development” post, but today Mike Roberts updated his overview of the market for serverless computing and it’s far more thorough than I could ever be.

The whole thing is well worth a read if you’re interested in the area, but these parts (emphasis mine) really resonated with me:

Serverless is not the correct approach for every problem, so be wary of anyone who says it will replace all of your existing architectures. Be careful if you take the plunge into Serverless systems now, especially in the FaaS realm. While there are riches — of scaling and saved deployment effort — to be plundered, there also be dragons — of debugging and monitoring — lurking right around the next corner.

Serverless services, and our understanding of how to use them, are today (May 2018) in the “slightly awkward teenage years” of maturity. There will be many advances in the field over the coming years, and it will be fascinating to see how Serverless fits into our architectural toolkit.

This is exactly right in my experience.

Lambda and Azure Functions let you write+deploy code quickly without an infrastructure team and an execution platform. However, the developer experience is often a big step backwards – Lambda doesn’t offer any remote debugging support, and just running+debugging functions locally is a big pain. Azure is further ahead in debugging, but things are still more complicated and less reliable than when debugging traditional apps. Integration testing is difficult on both platforms.

The benefits of serverless platforms outweigh the costs for my small hobby projects, but they almost certainly would not for larger-scale development in in most organizations.

The good news is that this area is maturing very rapidly; the tooling for .NET Azure Functions is much better today than it was 1 year ago. Serverless might not be right for enterprise development today, but just a few years from now a lot of these rough edges will be a distant memory.

At First, Everyone Hated Vancouver's West End

Thu, 26 Apr 2018 21:09:28 -0700

Vancouver’s West End, 1966. Source: Vancouver Archives

A few years ago, this short piece by former Vancouver councillor Gordon Price made a big impression on me.

The Canadian Institute of Planners had decided to celebrate Vancouver’s West End neighbourhood by giving it their 2015 “Great Neighbourhood” award. Price noted that this is ironic given that during its most recent development boom, the modern West End was widely regarded as a terrible planning decision:

…this neighbourhood of old converted single-family homes was largely bulldozed to create the Great Neighbourhood of today. The West End, during the boom era of highrise construction in the 1960s, was considered a concrete jungle – what most Vancouverites didn’t want anywhere near them: everyone’s best bad example of urban redevelopment.

Impossible to do that today. Imagine taking a square mile anywhere south of 16th Avenue and rezoning it for the kind of development that characterizes the West End.

This really surprised me as someone who moved to Vancouver in the 2000s. Nearly everyone loves the West End now! It’s got a lot of relatively affordable rental apartments, it’s close to all kinds of natural amenities, and it’s a short walk from downtown.

Fast-forward to late 2017, and I’m digging through old newspapers at the VPL for Abundant Housing Vancouver. When I get to the 1960s, I find out that Price is right: many people really hated the modern West End when it was being built. Here are just some of the articles I found.

Quality in the West End (June 24, 1965)

With the greedy onrush of speculators crowding boxy apartment block next to dumpy apartment block – each new one shutting out the coveted view from another – the city has reason to be ashamed of its lack of supervision in a site so fortunate.

Quick freeze for West End skyline (Nov. 2, 1972)

Most of those council members who are not seeking re-election have been in office throughout the building-boom years, in which the West End was allowed to reach a density unparalleled in this country, perhaps anywhere this side of Calcutta. At last the first steps have been taken to control the planning nightmare which resulted.

Fairview Slopes rezoning

Fairview Slopes underwent a major rezoning in the early 1970s. During the planning process, negative comparisons to the West End were very common. Common concerns: too dense, not suitable for families, and too expensive.

Another West End Not Wanted (Nov 1970)

Concern that redevelopment on the south shore of False Creek could create another West End was expressed by several members of city council at a meeting Tuesday. Councillors called the West End “too densely populated” without enough “accommodation for families.”

Fairview for Families (Nov 25, 1970)

City council doesn’t want redevelopment of the south shore of False Creek to result in another West End.

Untitled column (Vancouver Sun, Feb 13 1971)

Alderman Hugh Bird of the NPA: “We don’t want another West End, where it’s just for bachelors or boys and girls getting together to have fun.” (1, 2, 3, 4)

Students face ‘push-out’ (Vancouver Sun, May 27 1971)

AMS president Steve Garrod said the area would get the bad social environment of the West End if the rezoning goes into effect. “It will be a place of fairly expensive high-rises where students won’t be able to live,” he said.

Kitsilano

You know the small handful of West End-like concrete apartment buildings in Kitsilano? Shortly after they went up, Kitsilano residents mobilized to stop others from being built.

Row sparks new look at zoning (Vancouver Sun, Feb 16 1972)

[Alderman Ed Sweeney] said it was important to move immediately before Kitsilano becomes “another West End.”

Finish Lines (Vancouver Sun, Feb 16 1972)

…a nine-storey apartment in the 2200 block Cornwall, which they feared would be the start of a West-End-type wall of highrises across the waterfront…

High-rise plan protested (Vancouver Sun, Feb 17 1972)

Vancouver Tenants Council secretary Bruce Yorke said assurances by mayor Tom Campbell “cannot be considered a sufficient guarantee that the Kitsilano area won’t in the future become like the West End.”

Plan to limit building height in Kitsilano to be drafted (1972)

Alderman Ed Sweeney said council must take immediate steps to prevent Kitsilano developing the way the West End has.

Project halted in Kitsilano (Vancouver Sun, Mar 13 1972)

Residents in the area have been angered about the proposed 10-storey building, saying it would lead to a jungle of concrete and glass, like the West End, in Kitsilano.

Highrise limit backed at Kitsilano meeting (Vancouver Sun, Apr 13 1972)

Brian Mason, another spokesman for [the Kitsilano Area Resources Council], said people in Kitsilano do not want their area to suffer the same fate as the West End. (1, 2)

High-rise plans opposed (Vancouver Sun, Mar 15 1973)

This is the foot in the door. Before you know it, Kits will be another West End.

‘Threat to Kitsilano’ High-rise plan protested (Vancouver Sun, Aug 17 1973)

On a 13-storey Legion seniors’ housing building: “The situation in the West End is deplorable. We don’t want this area in Kitsilano to have a repetition of that.”

Conclusion

The West End really was dramatically transformed during the 1960s and 1970s, as can be seen in a casual glance at this map. With the benefit of hindsight, most contemporary objections to this seem wrong or at least short-sighted. To this day the West End provides a significant amount of Vancouver’s purpose-built rental units, and half of the West End’s population lives in buildings constructed 1960-1980.

Perhaps we should take a longer view when evaluating objections to densification and new apartment buildings. Wouldn’t Vancouver be nice with a few more West Ends?

The West End and Stanley Park in April 2018. Photo by author.

Sources

All articles were found on microfiche at the Central Vancouver Public Library in the Pacific Press newspaper clippings, subject “zoning”. Some clippings did not specify the newspaper, but the Vancouver Sun or Province are safe bets.

Land Use Visualization using Angular and the Canvas element

Sat, 10 Mar 2018 17:50:34 -0800

I recently built a web tool to solve a simple question that comes up often in urban planning: after taking setback requirements into account, how much of lot can be built on? The answer is often surprising: for example, Vancouver’s most common residential zone only allows houses to cover about 28% of the land.

It was a fun weekend project, and a few weeks later I decided to upgrade it on a long plane ride. It’s now a neighbourhood-level simulator with many more parameters:

These are mobile and desktop-friendly, and the visualization is entirely done in the browser. Here’s how it all works.

Technologies Used

The visualization is drawn natively in the browser using the HTML5 Canvas element. This was my first time using Canvas and it (unsurprisingly) feels very similar to 3D graphics programming; simple operations are generally done by moving the drawing context instead of the objects you’re drawing. Once you wrap your head around that it’s not bad at all, especially with judicious use of the save() and restore() methods on CanvasRenderingContext2D to push and pop rendering contexts for later use.

As I often do, I used Angular CLI to create an Angular project. It makes setting up an Angular project dead simple:

ng new neighbourhood-visualizer

Angular isn’t really needed for a small single-page project like this, and I admit that I wince every time I see the size of its compiled output. Still, it’s a really nice development experience. You get a lot for free (TypeScript integration, build+test tooling, templating, reactive forms…) just by creating an Angular project.

In this case, I wanted to use Reactive Forms to bind controls to the underlying data model. Specifically, I was worried about the <canvas> visualization being too expensive, and I wanted an easy way to throttle the rendering if needed (say someone moves the sliders very quickly and the rendering can’t keep up).

The reactive programming model makes this super easy because the data changes are exposed as observables and we can trivially transform the stream of changes from the observable. For example, this code calls render() every time the form values change:

this.inputForm.valueChanges.subscribe(val => {
      render();
    }

If we want to throttle it so render() is called only every 200 milliseconds at most, we just add a call to debounceTime():

this.inputForm.valueChanges.debounceTime(200).subscribe(val => {
      render();
    }

As it turns out, <canvas> performance is more than fast enough and I don’t need to throttle rendering at all. Premature optimization something something. :)

What’s Next?

Extrapolating from recent progress, I should have a decent SimCity clone in a few years’ time. In the meantime, I have a few smaller goals on my plate:

I’d like to add some presets – it would be nice if people could easily compare different neighbourhoods in popular cities.
I’d like to add more esoteric options like laneway houses
- This might require a rethink of the UI – perhaps separate ‘beginner’ and ’expert’ modes.

The source code for the visualizer is available on GitHub, so if you’d like to customize it for your own needs please do! I’m also keen to see this used for real world pedagogy. If you’d like to use this for teaching just give me a shout and I’d be happy to discuss potential customization.

AHV Letter Builder

Mon, 19 Feb 2018 18:53:59 -0800

I’m a member of a nonprofit called Abundant Housing Vancouver, and as you can probably tell, I happen to do some programming too. In 2017 I was able to spend a lot of time combining these interests which was pretty great!

Over a few blog posts I’ll briefly outline-solid the projects I worked on – they’re all open source and who knows, they might even be useful for other housing advocacy groups someday. First up: the Abundant Housing Vancouver Letter Builder.

Background

In Vancouver, many housing projects are decided on at contentious public hearings. At AHV we often try to rally support for good projects at the public hearing stage, but it’s hard because feedback at hearings tends to be biased against new housing for structural reasons. Without going into too much theory, we need to make it easy for people to support new projects, or nobody will. Thankfully Vancouver accepts hearing comments via email – it’s much easier to send an email than it is to spend an entire evening waiting at City Hall to speak!

Still, it takes a long time to write an email if you’re starting from scratch: first you need to familiarize yourself with the project and the public hearing process, then take quite a bit of time to actually write it. We needed a way to quickly 1) describe a project to supporters, 2) help them compose a personalized email to the City. Inspired by a clever letter generator project by Melody Ma and Ivy Tsai, we decided to write our own tool for this.

Design Goals

The tool should be really quick to use
The letters generated by the tool should not look like form letters! They should vary from one another and be easily customized by users
It should be super easy to configure the tool for new projects

Finished Product

Here’s what we ended up with. You can give it a spin, the testenv=true parameter ensures that it won’t really send emails. It’s quite easy to use:

Fill in personal information (name, address)
Pick the following from a list of options: your relation to the project, what you like about the project, and what you think could be improved
Click Generate Letter and a letter is generated based on your input, using randomized text.
Edit the letter if desired then click Send Letter

After that, the letter is sent to a Slack channel where we can approve it:

If the letter isn’t spam, we click Approve and it gets sent to the City of Vancouver. Done!

Technical Details

I wrote the letter generator front-end in TypeScript, using Angular 4 with Bootstrap theming. I used angular-cli for setup and scaffolding.

All configuration for the front-end (project info, text snippets for letter generation) lives in a Google spreadsheet and I read it in using the excellent Tabletop.js. Configuring the letter generator for a new project is as easy as adding a new row to the spreadsheet. Project descriptions are written in Markdown and then converted to HTML using the handy ng2-markdown-to-html library.

The front-end code is available on GitHub and if it sounds useful I encourage you to fork it and give it a spin.

The front end passes messages off to an Amazon Lambda function which was written by the remarkable Roman Lisagor. This function handles notification+approval (via a Slack webhook) and then sends the email using AWS SES.

Lessons Learned

This was largely an educational project for me – it had been years since I paid much attention to web development and I was keen to try building something in Angular. As such, caveat emptor if you end up using the code.

Programming in TypeScript is pretty nice, especially compared to the JavaScript of yore. Compile-time safety has saved me many times, and declaring variables with let eliminates the usual JavaScript scoping confusion. Newer versions of JavaScript also support let but… personally I’ll stick with TS for the static typing alone.

I generally liked Angular, but good lord does it ever generate huge script files. A simple project like this compiles to 1.3MB of scripts (in the production build, mostly vendor scripts). I cringe to think about how much bandwidth is being used to for relatively little program logic.

angular-cli is very helpful for getting all the necessary scaffolding set up, and I wouldn’t hesitate to use it again.

Tabletop.js was a lifesaver. For small+casual projects like this, sticking our config in a Google spreadsheet has a lot of advantages:

Google handles permissioning and authentication pretty well! It’s trivial to control access to the ‘database’ and everyone already has a Google account
The configuration is easily read and edited by non-technical people
You don’t have to spend time administering and hosting a database somewhere

I would shy away from this approach for most professional projects, but if you just want to hack something small together you get a lot for free with Google Sheets + Tabletop.js.

Japanese Apartment Hunting

Thu, 25 Jan 2018 21:10:33 -0800

Over the past 10 years, I’ve spent a lot of time looking for apartments in both Vancouver and in Tokyo. It’s been an interesting chance to compare two very different cities.

Vancouver has a remarkably low vacancy rate, so right off the bat renters are competing for a very small pool of available homes. Apartment hunters are also usually stuck with Craigslist. Craigslist is far and away the best place for browsing apartments in Vancouver, but not because it’s a well-designed site. It’s a perfect example of network effects: it offers a poor user experience, but landlords can’t afford to ignore the large pool of renters using Craigslist and vice versa.

Things are very different in Tokyo. It seems difficult to get comparable rental vacancy data (see the numbers and caveats here), but the sheer number of rental listings available online makes me suspect that the vacancy rate is much higher than in Metro Vancouver. The tools are better too: apartments tend to be listed by brokers on a few big websites, and the experience for renters on those websites is much better than on Craigslist.

Since most North Americans don’t know what they’re missing, I’ll give a quick overview of what it’s like to look for an apartment in Japan.

Apartment listing sites in Japan

There are a handful of big listing sites in Japan:

Suumo and At Home handle all kinds of real estate listings, but Apamanshop specializes in the rental market. Let’s take a look at rental options in the Kanto region (which includes Tokyo) on Suumo:

Here we have a map of the different Kanto prefectures plus Tokyo. For each of those areas you can pick whether you want to search by train line or by sub-area. Up top, it says that there are 339,703 new listings this week in the Kanto region (!!!). If we want to search the entire Kanto region, there are yet more options on the bottom and right:

Search by commute time (enter your places of work or study and how long you’re willing to spend on the train, and this will show you listings within commuting distance)
Search by rental price (this lets you specify which type of home you’re looking for, then shows you the average monthly rent for that type near various neighbourhoods and train stations)
Search by regular map
Search by train map

Just to get a sense of what’s out there, let’s try browsing Tokyo by train line:

In this screen we’re just picking train lines to narrow down our search. The train lines are divided up by operator (JR, Tokyo Metro, and various private railways), and the numbers in parentheses show how many housing units are available for rent near each line. There is certainly some overlap between lines, but it’s still amazing to see so many train lines with thousands and thousands of homes listed nearby.

A specific apartment search

Now let’s look for something more specific. I’m a Metafilter regular, and last year I stumbled across this heartbreaking question. A young Seattle resident was looking for “a room in the $500/month range, within a 90 minute commute from downtown,” and multiple commenters jumped in to say that this isn’t possible. This would be very difficult in Vancouver too – let’s see if it’s achievable in Tokyo.

$500 USD is about 55,000 JPY. Tokyo doesn’t have a single “downtown” persay, but it does have a few major transit hubs. Shinjuku Station is the busiest train station in the world, due to the huge number of rail lines connecting to it. To approximate a “90 minute commute from downtown”, let’s look for an apartment within a 1-hour commute to Shinjuku (since from Shinjuku you should be able to get just about anywhere in ultra-central Tokyo in 30 minutes).

To search for something like that, I’ll use the following conditions:

Within 40 minutes (by train) of Shinjuku Station
Within a 20 minute walk of a local train station
Between 35,000 and 55,000 JPY/month (the minimum is intended to exclude really low-end places), including management fees
In Tokyo (we could get more results by including the surrounding prefectures, but Suumo only lets you choose one area at a time)

With those search conditions we get 76,884 results:

Most of these apartments are small (as you’d expect on a budget like this), but you might be surprised by how clean+modern many of them are:

At this point it should be clear that low-income urban residents have a much easier time finding housing in Tokyo than in many other world cities. I think this is largely due to Japan’s unique land use policies, and I’ll elaborate on that in future posts.

More Search Options

I used some pretty vanilla search parameters above, but you can specify a lot more search conditions on Japanese websites. You can browse the search page translated to English if you want to see all of the available search parameters. These are just a few of the options that I find useful:

Distance from the nearest train station (can look for homes within a 5 minute walk of a train station, for example)
Floor area
Age of building
Construction material (wood frame, steel frame, etc.)
Distance from amenities (supermarkets, convenience stores, parks)
Does it have a balcony/air conditioner/bike storage?

The especially nice thing about this is that nearly every listing has this information populated correctly. Anyone who’s ever suffered through Craigslist’s sea of low-effort listings will be thankful for that.

Databases as file formats

Sun, 21 Jan 2018 07:30:31 -0800

I’m building an interactive online map of all properties in Vancouver, and along the way there have been a few pleasant surprises. Most recently: the .MBTiles tileset format is surprisingly cool.

Background

Mapbox is one of the biggest players in the open source mapping space (especially now that Mapzen and Carto have thrown in the towel – Mapzen is closing and Carto is now using Mapbox tech). One of the many nice things about Mapbox is that they developed an efficient open standard for vector map tiles, appropriately named Mapbox Vector Tiles (read this if you’re not sure why vector tiles are great).

Map tiles are often pre-computed for each zoom level, and once you’ve done that you need to store them somewhere. Enter the .MBTiles tileset format.

Poking around under the hood

My first encounter with this file format occurred when I used Eric Fischer’s excellent tippecanoe tool to simplify my data set at lower zoom levels. Tippecanoe generates .mbtiles files, which are easy to serve to clients either by uploading to Mapbox, using a third party tile server, or even by rolling your own server with something like the mbtiles Node.js package.

All great… but after setting up a server my Mapbox GL JS client refused to render the tiles. I tried a few things without much luck, and then as a last resort I decided to poke around in the .mbtiles file. I was expecting to need a hex editor or similar, but then I saw this beauty in the spec:

MBTiles is a specification for storing tiled map data in SQLite databases

The files themselves are just relational databases in a known schema – how cool is that? Emboldened, I grabbed a SQLite client and opened up my .mbtiles file:

My .mbtiles file in a SQLite client

Right away, I spotted my issue in the metadata table: my client was attempting to bind a style to a layer using the wrong layer ID.

Conclusion

Relational databases are a great way to store data for easy retrieval+manipulation – SQL is far and away the most popular query language, and most developers are already familiar with it. SQLite is probably the most frequently used relational database in the world and it’s supported by tons of clients as a result.

Mapbox has done something extremely simple but very clever here. By using SQLite databases as data files, they’ve established a tileset format that is:

Easy to query, even in complex ways
Already supported by hundreds of tools (there are a lot of SQLite clients out there)
Instantly familar to most developers
Conceptually very simple
Flexible (from the spec: “the schemas outlined are meant to be followed as interfaces. SQLite views that produce compatible results are equally valid”)

It seems like this is already paying off for them, as there are quite a few third-party implementations of the .mbtiles format.

More generally, I really love the idea of using databases as a file format. It wouldn’t be suitable for everything (maybe don’t write your next-gen word processor document format as a position int, word varchar(max) table…) but it seems great for general collections of data like this. Versioning of the underlying database format is theoretically a concern, but that’s significantly mitigated by SQLite’s exceptional focus on maintaining compatibility.

First!

Sat, 20 Jan 2018 00:00:00 +0000

Hi there - I’m Reilly Wood, a software developer from Vancouver, Canada. This site is still under development, but I’m planning to fill it up with thoughts and impressions about software, cities, finance, and whatever strikes my fancy. Stay tuned (perhaps with the handy RSS feed).

Vancouver’s rocky start with secondary suites

Sun, 03 Dec 2017 13:35:03 -0800

Or: that time that Vancouver decided single-family zoning was more important than defeating Hitler

Secondary suites (often just “basement suites”, or ADUs) are an everyday part of Vancouver neighbourhoods now. Even before they were fully legalized in 2004, they provided a large amount of Vancouver’s low-cost housing stock.

The story I usually hear about suites goes like this: people started building a lot of unauthorized suites in Vancouver Specials in the 1970s and 1980s, and this set the stage for a long drawn-out political battle that eventually ended with suites being legalized.

It’s a good story, and it’s true! But Vancouver’s history with secondary suites goes back much further. People were trying to live in secondary suites – and Vancouver was trying to stop them – for a long time before the Vancouver Special.

World War II

Vancouver’s first zoning code was introduced in the late 1920s, and it outlawed apartments, townhouses, and even secondary suites in most of the city. But even then there were Vancouverites who could not afford a single-family home, and the restrictive code soon caused growing pains. The city’s first brush with legalizing suites began just over a decade later, prompted by Canada’s involvement in World War II.

Canada’s wartime government had an especially strong incentive to let more people live in productive urban areas. One of the approaches taken was to restrict cities’ ability to use exclusionary single-family zoning.

In 1942, the federal Wartime Prices and Trade Board issued Order No. 200, also known as the “Respecting Housing Accommodation in Congested Areas” order. This order effectively said “if a homeowner wants to rent a suite or room out to someone, cities can’t prohibit that”. According to historian Jill Wade, Order 200 was a success:

As a result, many larger houses in older, one-family dwelling zones and in Shaughnessy Heights became multiple-family units. -Houses for All: The Struggle for Social Housing In Vancouver, 1919-50

The response from Vancouver council was swift. Less than a year after the introduction of Order 200, council ordered a bylaw amendment expressly designed to constrain the order as much as possible. The city was still bound by the terms of the order for existing homes, but they could use a legal loophole to ensure that it did not apply to new homes. The city’s chief lawyer Donald McTaggart was incredulous:

The corporation counsel told the committee that the amendment it suggests will be quite legal, but he expressed the opinion that the idea of Order 200 is “being lost sight of.” … “The government,” he reminded aldermen, “said ‘forget zoning bylaws’ for the sake of getting on with the war.” –Bylaw Ordered on Plea of Town Planning Board, The Vancouver Sun, 1943

UBC students and the post-war era

11 years after the war, Vancouver started eliminating the wartime suites – even the ones that had been created in existing houses:

With the City’s adoption of the Zoning and Development By-law in 1956, secondary suites and other multiple-unit buildings in the areas zoned RS-1 (single-family) became illegal. In the late 1950s, Council decided to remove all illegal suites over a 10-year period, and by 1966, over 2,000 suites had been removed under the program. –The Role of Secondary Suites, City of Vancouver 2009

The strongest opposition to the city came from UBC students. In 1960, UBC student council begged the city to relent and allow “up to four students per single family dwelling until the housing situation on campus eases”. They were opposed by local homeowners:

Members of the Northwest Point Grey Homeowners Association fear their area will become a slum unless city zoning bylaws are enforced. … The homeowners spokesman said: “The best information we can get shows conversions to boarding houses is a prominent factor in the cause of slums.” -Boarding Houses Irk Homeowners, The Vancouver Sun, 1960

The Point Grey homeowners won, but UBC students would keep trying in vain for another 8 years. The UBC Alma Mater Society came back in 1966 with a smaller “ask”: this time they asked for 3 boarders in a house (down from 4), and only in the West Side neighbourhoods closest to UBC. Again, they were rejected by Vancouver council and planners.

Planners agreed that there is a shortage of suite accommodation at UBC but said provision of such accommodation for out-of-town students should not be at the expense of single-family residential areas in Vancouver. –Council Rejects UBC Suite Plea, The Vancouver Sun, 1966

The Alma Mater Society tried twice more in 1968 without success:

Alderman Earle Adams said the first responsibility of council is to taxpayers. He said students from Vancouver can live at home, so the shortage really affects only those who come from outside the city. “Why should the city be responsible for housing out-of-town students?” he asked. –UBC Housing Crisis: City Won’t Okay Illegal Suites, The Vancouver Sun, 1968

Even in the 1960s, there was significant demand for more affordable housing in Vancouver’s exclusive single-family neighbourhoods. Council didn’t eliminate that demand when they forced the closure of wartime suites, but they were able to ignore it until the Vancouver Special forced another city-wide conversation.

Conclusion

Vancouver is often cited as a leader in terms of legalizing secondary suites, and with good reason – we’ve gone far beyond what most North American cities have done to date. However, when put in historical context, it becomes clear that Vancouver’s policy changes were still too little, too late.

The federal government forced the city’s hand during World War II and showed that there was demand for denser living in single-family neighbourhoods, even in 1942. That demand didn’t go away after the war – students were begging council to stop barring secondary suites near UBC all throughout the 1960s.

It’s great that Vancouver eventually decided to allow secondary suites in all neighbourhoods. It would have been even better if the policy hadn’t come 60 years after it was first shown to be necessary.

Works Cited

The Wartime Prices and Trade Board Order No. 200, Respecting Housing Accommodation in Congested Areas

From Canadian war orders and regulations, page 177.

Houses for All: The Struggle for Social Housing in Vancouver, 1919-50

Jill Wade. Page 117

The Role of Secondary Suites

City of Vancouver 2009

Vancouver Sun articles

(Available on microfiche at the Vancouver Public Library central branch, reproduced here for convenience)

What Motivated Vancouver's First Zoning Codes?

Sat, 17 Jun 2017 09:07:48 -0700

Ask a random person what the purpose of zoning is, and they’ll probably mention that it keeps unpleasant or dangerous things away from homes. You wouldn’t want to live next to a garbage dump or concrete factory, right?

Most people agree that that’s a good thing, but zoning codes do a lot more than that. If you could ask the people who wrote our first zoning codes, you’d quickly learn that separating industry from homes was very far down their list of goals. Separating certain homes – specifically apartment homes – from other homes was the primary motivation for Vancouver’s first zoning code.

Point Grey

Our story starts in the 1920s, with 3 municipalities where Vancouver lies today: The City of Vancouver, Point Grey, and South Vancouver. Point Grey encompassed not just modern-day Point Grey, but most of the West Side. Point Grey was the first municipality in Canada to introduce zoning in 1922. The chairman of the Point Grey Town Planning Commission was eager to boast of the zoning bylaw’s “successes,” specifically, how it excluded lower-cost housing:

Such by-laws as these served, in no uncertain way, to implement the ideals held by the residents that their municipality was to be one in which the best type of home could not only be built, but also adequately safeguarded from the encroachments of undesirable types of development. That the quality and type of dwelling within the municipality at the present time is of a very high order is indicated by the study of the “dwelling permits” over a five-year period. The average cost of residences over such a period, these “permits” show, was in excess of $4,100. At the present time over ninety per cent, of the municipality is zoned for one-family dwelling districts. Point Grey has no slum district. (A Plan for the City of Vancouver, p. 297)

Vancouver’s Interim Zoning Bylaw

By 1927, the City of Vancouver was seriously considering a zoning bylaw of their own. They had hired Harland Bartholomew, a prominent urban planning consultant, to devise a comprehensive plan for Vancouver. But these things take time, and time was in short supply. Well before Bartholomew could finish his official plan, Vancouver commissioned him to write a temporary bylaw to stop the most pressing danger to the city: apartment buildings. His instructions were unambiguous:

“When Bartholomew asked what abuses he should consider in the interim zoning by-law of 1927 he was preparing, the chairman replied that ’the only serious abuse… is the intrusion of undesirable apartment houses into residential districts’” (Zoning and the Single-Family Landscape, p. 60)

Bartholomew was happy to state the motivation for the interim bylaw in official documents:

…largely to prevent the intrusion of apartment houses in single or two-family residential areas, an interim zoning by-law was prepared and approved by the Town Planning Commission, recommended to the Council, and became law on 5th February, 1927. (A Plan for the City of Vancouver, p. 211)

Vancouver’s First Comprehensive Plan

Finally in 1930 the three municipalities had been amalgamated into one and Bartholomew’s official Plan for the City of Vancouver was released. Bartholomew’s plan was remarkably comprehensive, spanning 310 pages. Thankfully for us, many of those pages were used to describe the intent of the plan and not just the regulatory details. Town Planning Commission chairman Arthur Smith set the tone by praising Point Grey’s early bylaw, explicitly segregating Vancouver by class, and noting the promotion of single-family homes as a major goal:

The wise foresight which Point Grey has used in planning at an early stage of its growth should provide Vancouver with one of the most desirable residential districts possessed by any city on the Continent, and those who have to gain their livelihood by manual labor should find in Hastings Townsite, and in a replanned South Vancouver, a place where they can build up modest homes which should differ only in size from that of the more opulent employers. The retention of Vancouver as a city of single family homes has always been close to the heart of those engaged in the preparation of this plan. (A Plan for the City of Vancouver, p. 26)

Bartholomew later clarified his preference for single-family homes in an appeal to authority:

As has been mentioned, Vancouver is largely a city of one-family homes and is surrounded by similar development in the adjoining municipalities. Large areas are now available for such development, though a considerable proportion has yet to be served by utilities. That the one-family dwelling is the desirable unit for happy living is the general concensus (sic) of opinion of all authorities. (A Plan for the City of Vancouver, p. 233-234)

Neighbourhood stores

Bartholomew was also keen to keep stores out of residential neighbourhoods:

The scattering of stores promiscuously throughout residence districts has done considerable damage to the city’s appearance. The nearly universal custom of building stores out to the street line has hurt the appearance of a good many residence streets and at the same time has injured adjoining lots by making them less desirable for living purposes and reducing their saleable value. The zoning by-law will remedy this condition and tend to prevent residence districts from becoming blighted. (A Plan for the City of Vancouver, p. 247)

The contrast with modern-day Vancouver is remarkable, given that neighbourhood stores built before Bartholomew’s Plan are now many neighbourhoods’ most cherished jewels. Who would prefer the East Side without the Marché St. George, or the West Side without Arbutus Coffee? Bartholomew sought to completely eradicate shops and meeting places from residential neighbourhoods, without asking whether people might want to live near them.

The Marché St. George in East Van

The Wilder Snail in Strathcona

Cardero Bottega in the West End

Neighbourhood stores might seem like a small nuisance at worst, but no: in a later report for the City of Vancouver, Bartholomew clarified his belief that apartments and stores were the “worst offenders” motivating his plan:

Had it not been for the widespread intrusion of commercial and industrial uses and the building of multiple dwellings in single-family residential districts just after World War I, the institution of Town Planning in Vancouver would have been delayed or retarded for several years. The encroachment of apartments and of retail stores, the latter almost always extending to the street line, were the worst offenders. (A Preliminary Report upon Zoning, p. 1)

If only Bartholomew had stopped there. Even though apartments and stores were his focus, the plan is littered with other “menaces” that he intended to keep out of Vancouver. Unfortunately, some of those are now recognized as aspects of Vancouver’s best pre-zoning urbanism.

Small Lots

Small residential lots were also verboten in Bartholomew’s Vancouver:

The minimum dimensions for lots shall be forty (40) feet for width and one hundred and twenty (120) feet for depth, and in no case shall a rectangular or irregular shaped lot contain less than forty-eight hundred (4800) square feet. The custom of laying out 25-foot lots is productive of building conditions which are not a credit to the city. (A Plan for the City of Vancouver, p. 276)

Small residential lots were to be shunned, because they might attract the kind of builder who couldn’t afford a bigger lot. Nowadays, that sort of affordability is more appreciated and many people enjoy the fine-grained urbanism that small lots bring to neighbourhoods. Fortunately, Vancouver’s Strathcona neighbourhood was developed with narrow lots before Bartholomew’s ideas were adopted:

Strathcona apartment buildings on 25-foot lots

Strathcona 4-plex on a 25-foot lot

Front yards

Bartholomew’s plan also mandated very large yards for residential buildings. Previously, many buildings were built fairly close to the sidewalk – but now houses and low-rise apartment buildings were required to have a front yard 20 feet deep. This might seem like a small change but its effects cannot be understated. Many low-rise buildings from before this rule take up much less land per household – and many later buildings are saddled with incredibly underused yards.

Pre-zoning homes in Riley Park

Post-zoning homes in Riley Park

Pre-zoning Mount Pleasant apartments

Post-zoning apartments in Mount Pleasant. About half as dense (by FAR)!

Legacy

Bartholomew’s plan was not adopted in full, but the aspects mentioned above were. Even though it was written nearly 90 years ago, its outline is still clearly visible in Vancouver’s modern zoning code. Some change has occurred and mostly for the better: the “single-family” zones have been revised and now permit one basement suite and a (very small) laneway house in addition to the main suite. But that “Great House Reserve” still takes up the vast majority of our residential land, still forbids apartment buildings and townhouses, and its boundaries still remain mostly unchanged:

Conclusion

The primary motivation for Vancouver’s first zoning code was to prevent the construction of apartment buildings and stores in residential neighbourhoods. Apartment buildings were seen by Bartholomew as an afterthought at best, and a major nuisance at worst. Stores in residential neighbourhoods were only seen as a blight. Even after ensuring the homogeneity of residential neighbourhoods, Bartholomew imposed extravagantly large minimum lot sizes and yards.

Vancouver’s formative urban plan could be more accurately described as a suburban plan, designed by men with a profoundly anti-urban bias. It would be amusing if we weren’t still living in its shadow.

Works Cited

A Plan for the City of Vancouver including Point Grey and South Vancouver. Harland Bartholomew and Associates, 1930
A Preliminary Report Upon Zoning Vancouver, British Columbia. Harland Bartholomew and Associates, 1946
Zoning and the single-family landscape: large new houses and neighbourhood change in Vancouver. Barbara A. Pettit, 1993