justinpombrio.net

Lindenmayer Systems

Mon, 16 Feb 2026 00:00:00 -0500

Let me show you how to use Lindenmayer systems to produce beautiful images like this one:

Turtles

We’ll start by talking about turtles.

Some of you are nodding along like this is an obvious starting point; others are rather confused. In 1967(!) some brilliant engineers designed an educational programming language called Logo. It lets you make turtle graphics drawings by telling a “turtle” with a “pen” where to walk on the screen, drawing a line as it goes. I just learned when writing this that there were also physical programmable turtles that would draw on paper!

So yeah, we’re startings with turtles. Logo turtles had all sorts of fancy commands, but ours will need just three:

f means “walk forward one unit”
- means “turn left a quarter turn”
+ means “turn right a quarter turn”

The turtle starts in the center of the screen facing up. So the program -f+ff will drawn an “L”:

The turtle’s path is drawn as a gradient from white to yellow to green. If you have an active imagination you can pretend that the green dot at the end is a turtle.

And the program f+f--f+f+f will draw an “F”:

Angles

What if you want your turtle to turn at angles other than a quarter turn? We could go the route that Logo did, and give an arbitrary angle for every turn. But for the sorts of drawing we’re going to do we can get away with something simpler: we’ll declare up front what angle - and + should turn. (Measured in turns, of course.)

For example, to draw a hexagon we could use the angle 1/6 and run the turtle program f+f+f+f+f+f+:

Aristid Lindenmayer

Drawing a detailed image this way would take a really long sequence of -, +, and f. Again, Logo solved this in a general way by adding loops to the language, but we’re going to do something a lot more specialized. We’re going to use an idea by Aristid Lindenmayer in 1968, now called “Lindenmayer Systems” or “L-Systems”.

(This timing—Logo in 1967 and L-Systems in 1968—is suspicious, isn’t it? I’m not aware of any explicit cross-polination of ideas between Lindenmayer and the Logo creators, but wouldn’t be surprised if one partially inspired the other.)

Lindenmayer’s idea is to have a start string, and some production rules that replace a letter with a sequence of instructions. You start with the start string and repeatedly replace letters according to the production rules for some number of iterations. Then you have the turtle follow the (now very long) sequence of instructions.

This will be easier to follow with an example.

One pretty picture that can be drawn with an L-system is the dragon curve. Its rules are:

start: R
productions:
    R -> Rf+L
    L -> Rf-L
angle: 1/4

Iterations of the dragon curve look like this:

The 0th iteration starts with the start string, so it’s just R.
The 1st iteration replaces that R according to the production rule R -> Rf+L, so it becomes Rf+L.
The 2nd iteration replaces both the R and L according to the production rules, giving Rf+Lf+Rf-L.
And so on.

We get to choose how many iterations to do. Let’s say we do 9 iterations. Then we get:

Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf+
Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf+
Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf-
Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf+
Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf+
Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf-
Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf-
Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf+
Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf+
Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf+
Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf-
Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf-
Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf+
Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf-
Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf-
Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf+
Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf+
Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf+
Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf-
Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf+
Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf+
Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf-
Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf-
Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf-
Rf+Lf+Rf-Lf+Rf+Lf-Rf-Lf+Rf+Lf+Rf-Lf-Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf+
Rf+Lf-Rf-Lf-Rf+Lf+Rf-Lf-Rf+Lf-Rf-L

Now we just need to tell the turtle to follow these instructions. But… what’s it supposed to do with all the Rs and Ls? Simple: it will just ignore them. Erasing them from the string gives:

f+f+f-f+f+f-f-f+f+f+f-f-f+f-f-f+f+f+f-f+f+f-f-f-f+f+f-f-f+f-
f-f+f+f+f-f+f+f-f-f+f+f+f-f-f+f-f-f-f+f+f-f+f+f-f-f-f+f+f-f-
f+f-f-f+f+f+f-f+f+f-f-f+f+f+f-f-f+f-f-f+f+f+f-f+f+f-f-f-f+f+
f-f-f+f-f-f-f+f+f-f+f+f-f-f+f+f+f-f-f+f-f-f-f+f+f-f+f+f-f-f-
f+f+f-f-f+f-f-f+f+f+f-f+f+f-f-f+f+f+f-f-f+f-f-f+f+f+f-f+f+f-
f-f-f+f+f-f-f+f-f-f+f+f+f-f+f+f-f-f+f+f+f-f-f+f-f-f-f+f+f-f+
f+f-f-f-f+f+f-f-f+f-f-f-f+f+f-f+f+f-f-f+f+f+f-f-f+f-f-f+f+f+
f-f+f+f-f-f-f+f+f-f-f+f-f-f-f+f+f-f+f+f-f-f+f+f+f-f-f+f-f-f-
f+f+f-f+f+f-f-f-f+f+f-f-f+f-f-f+f+f+f-f+f+f-f-f+f+f+f-f-f+f-
f-f+f+f+f-f+f+f-f-f-f+f+f-f-f+f-f-f+f+f+f-f+f+f-f-f+f+f+f-f-
f+f-f-f-f+f+f-f+f+f-f-f-f+f+f-f-f+f-f-f+f+f+f-f+f+f-f-f+f+f+
f-f-f+f-f-f+f+f+f-f+f+f-f-f-f+f+f-f-f+f-f-f-f+f+f-f+f+f-f-f+
f+f+f-f-f+f-f-f-f+f+f-f+f+f-f-f-f+f+f-f-f+f-f-f-f+f+f-f+f+f-
f-f+f+f+f-f-f+f-f-f+f+f+f-f+f+f-f-f-f+f+f-f-f+f-f-f+f+f+f-f+
f+f-f-f+f+f+f-f-f+f-f-f-f+f+f-f+f+f-f-f-f+f+f-f-f+f-f-f-f+f+
f-f+f+f-f-f+f+f+f-f-f+f-f-f+f+f+f-f+f+f-f-f-f+f+f-f-f+f-f-f-
f+f+f-f+f+f-f-f+f+f+f-f-f+f-f-f-f+f+f-f+f+f-f-f-f+f+f-f-f+f-
f-

If we tell our little turtle to follow those instructions, it will draw:

What happens if we go further? After about 22 iterations (producing about 8 million instructions), the turtle is making turns smaller than a pixel on the screen, so the path it took is no longer directly visible: you just see the color gradient it was drawn with. Adding more iterations after that doesn’t make the image look any different (beyond rotating), it has stabilized. It looks like this:

Code

I implemented all this as a Rust library. Here’s the code for that dragon curve:

LindenmayerSystem {
    start: "R",
    rules: &[('R', "Rf+L"), ('L', "Rf-L")],
    angle: 0.25,
    implicit_f: false,
}

The extra field implicit_f tells the turtle what to do with leftover letters like L and R in the expanded program. For the dragon curve, like I said above, we should just erase them. But sometimes it’s more convenient to replace them with f; setting implicit_f: true would do so.

(There’s a math question here: can you express any L-system with implicit_f: true as an L-system with implicit_f: false and vice-versa? I don’t know the answer.)

Here are a couple more L-systems, to give you a sense what they look like. First David Hilbert’s space filling curve:

LindenmayerSystem {
    start: "A",
    rules: &[('A', "+Bf-AfA-fB+"), ('B', "-Af+BfB+fA-")],
    angle: 0.25,
    implicit_f: false,
}

Which looks like this, at 5 iterations:

And here’s Helge von Koch’s snowflake, at 3 iterations:

LindenmayerSystem {
    start: "X-X-X-X-X-X",
    rules: &[('X', "X-X++X-X")],
    angle: 1.0 / 6.0,
    implicit_f: true,
}

Gradient

So far all the pictures have been drawn with the same color gradient, but you might want to use others. The gradients I’ve implemented fall into three categories:

CET perceptually uniform color maps. These are color gradients designed to be perceptually uniform. The gradient used in all the examples so far is CET-L10.
Color gradients built out of combinators for things like scaling, cycling, and sawtoothing. These are all in OK-LAB color space.
A very fancy color gradient based on the 3D Hilbert curve. The idea is to (i) draw a cube in OK-LAB color space; (ii) trace a 3D Hilbert curve through the cube; (iii) dye the curve according to the cube’s colors; then (iv) straighten the curve and then lay it out on the curve that you’re drawing.

The last approach makes for very textured pictures since the 3D Hilbert curve changes color so quickly. (It covers all the colors on the cube, which is most possible colors.) For example, here’s the dragon curve with a 3D-Hilbert curve color gradient:

Pretty Pictures

Putting this all together, you can draw some very pretty pictures.

A curve from J. Arioni in 2017:

A haphazard curve I made that’s pretty nonetheless:

Moore’s curve:

An “s curve” of my devising.

UPDATE: I said in the readme about this curve that “It’s very simple, so I wouldn’t be surprised if I wasn’t the first to find it.” Indeed, someone emailed me to point out it was discovered by Knuth and Davis under the name “Terdragon curve”: it can be viewed as a variant of the dragon curve.

A curve from Dieter K. Steemann on Pinterest:

Wunderlich’s third space filling curve:

Sierpinski’s triangle:

Sierpinski’s space filling curve:

UPDATE: Some new curves. Thanks internet commenters!

Koch’s “quadratic island”:

D. M. McKenna’s SquaRecurve:

Repository

The code that generated these images is at https://github.com/justinpombrio/lindenmayer. Feel free to play around with it! I’m especially interested if anyone knows of interesting space-filling curves that I’ve missed.

A couple blocks from where I live, about a year ago, a PhD student at Tufts university named Rümeysa Öztürk was taken by masked, plainclothes DHS officers and held at an ICE detainment facility for six weeks for purely political reasons (she wrote a pro-Palestine op-ed in 2024). Since then, ICE used tear gas, unprovoked, on peaceful protesters while in Portland against the wishes of the cities’ elected officials, and shot Alex Pretti and Renée Good to death. The weaponization of ICE and other federal agencies against those politically opposed to the administration needs to stop.

Imagining a Language without Booleans

Mon, 22 Sep 2025 00:00:00 -0400

Let’s start where I started. Just thinking about if statements.

An if with an else can produce a value:

// C
int x = -5;
int abs_x = x > 0 ? x : -x;

# Python
x = -5
abs_x = x if x > 0 else -x

// Rust
let x = -5;
let abs_x = if x > 0 { x } else { -x };

What about an if without an else?

// C
int x = -5;
int abs_x = x > 0 ? x; // Syntax error!

# Python
x = -5
abs_x = x if x > 0 # Syntax error!

// Rust
let x = -5;
let abs_x = if x > 0 { x }; // Type error -- evaluates to ()!

No dice.

Optional `if`

I realized that there is a meaningful value for if (x > 0) x, though. It’s Option<i32>:

// Hypothetical-Lang

let x = -5;
let pos_x = if (x > 0) x; // = None

let y = 7;
let pos_y = if (y > 0) y; // = Some(7)

(This hypothetical language will look a lot like Rust, but with different syntax for if since we’re giving it a different semantics.)

In general, if the conditional evaluates to true then the if evalutes to Some, otherwise it evaluates to None:

if (true)  e  ⟼  Some(e)
if (false) e  ⟼  None

So the type of if is that it takes a bool and a T, and produces an Option<T>:

if (bool) T  :  Option<T>

(That is to say: if expr_1 has type bool and expr_2 has type T, then if (expr_1) expr_2 has type Option<T>.)

This generalized if could be used to perform an operation that’s only valid when the conditional is true:

fn strip_prefix(string: &str, prefix: &str) -> Option<&str> {
    if (string.starts_with(prefix)) &string[prefix.len()..]
}

or in conjunction with a method like filter_map:

let numbers = vec![9.0, -20.0, 16.0, -16.0];
let square_roots = numbers
    .into_iter()
    .filter_map(|x| if (x > 0) x.sqrt());
assert_eq!(square_roots.next(), Some(3.0));
assert_eq!(square_roots.next(), Some(4.0));
assert_eq!(square_roots.next(), None);

Optional `else`

There’s a matching interpretation for else. It’s type is:

Option<T> else T  :  T

and its evaluation rules are that:

None    else e  ⟼  e
Some(x) else e  ⟼  x

So it supplies a default for an option:

fn get_name(personal_info: &HashMap<String, String>) -> &str {
    personal_info.get("name") else "Whoever you are"
}

(This is exactly the behavior of Rust’s unwrap_or_else(), except that you don’t need to manually wrap the expression in a closure.)

Of course, you usually pair an else with an if. This continues to work like you’d expect, as you can check using the type rules:

let abs_num = if (num > 0) num else -num;
                 --------  ---
                   bool    i32
              ----------------      ----
                Option<i32>         i32
              ---------------------------
                          i32

Notice what we’ve done: we’ve turned both if and else into binary operators.

Optional `and` and `or`

Can we go further? What about and and or – can they work on options? They would take options, and return options.

or will take the first option if it’s Some, or otherwise take the second. You could use it to try multiple options each of which may or may not succeed:

fn get_color(attrs: &HashMap<String, String>) -> Option<&str> {
    attrs.get("color") or attrs.get("colour")
}

and will take the second option, so long as the first option was Some. You could use it to try something but only if a condition holds. For example, if you wanted to parse a hex color like "#a52a2a" if a string starts with #, or a color name like "brown" otherwise, you could write:

fn parse_color(color: &str) -> Option<Color> {
    (color.starts_with("#") and parse_color_hex(color))
    or parse_color_name(color)
}

More formally, and and or would have the evaluation rules:

None    and e  ⟼  None
Some(x) and e  ⟼  e

None    or  e  ⟼  e
Some(x) or  e  ⟼  Some(x)

and the type rules:

Option<A> and Option<B>  :  Option<B>
Option<A> or  Option<A>  :  Option<A>

(This makes and and or equivalent to Rust’s and_then and or_else.)

If you have good mathematical intuition you might notice the asymmetry between the type rules for and and or and suspect that we’re missing some sort of generalization. Don’t worry, it will come.

But booleans?

But wait! You still need to be able to use and and or inside an if’s condition, for example if (x >= 0 and x < len). That doesn’t work if and produces an option!

There’s an easy fix, though. There’s an equivalence between booleans and options:

bool  = Option<()>
true  = Some(())
false = None

So we could eliminate booleans from our hypothetical language, and always use the equivalent options instead.

Putting it all Together

Now let’s imagine what a programming language without booleans might look like.

We just talked about replacing bool with Option<()>, because those types are equivalent. But there’s a further equivalence with results, as well:

Option<T> = Result<T, ()>
bool      = Option<()>
          = Result<(), ()>

If we’re going to use the equivalence between booleans and options, we might as well go all the way and use the equivalence between options and results. Results everywhere!

Since they’ll be so ubiquitous, let’s invent a short syntax for them. T ? E can mean “either a successful value T, or an error E”. What Rust would call Result<T, E>.

Then bool is a type alias for ()?(). Except… that looks like an ASCII drawing of a fly’s face. So let’s call the unit value (and the unit type) nil instead. Then we have:

bool  = nil?nil

true  = Ok(nil)
false = Err(nil)

These will be aliases built into the language.

Writing all of the type rules from before but using results instead of options, we have:

if (A?E) B   :  B?E
A?E else A   :  A
A?E and B?E  :  B?E
A?E or  A?F  :  A?F
not A?E      :  E?A

(There’s the nice symmetry that was lacking before.)

And writing all of the evaluation rules:

if (Ok(x))  e  ⟼  Ok(e)
if (Err(x)) e  ⟼  Err(x)

Ok(x)  else e  ⟼  x
Err(x) else e  ⟼  e

Ok(x)  and e   ⟼  e
Err(x) and e   ⟼  Err(x)

Ok(x)  or  e   ⟼  Ok(x)
Err(x) or  e   ⟼  e

not Ok(x)      ⟼  Err(x)
not Err(x)     ⟼  Ok(x)

(I’m not really sure whether if (A?E) B is the right type, or if if should require its conditional’s error to be nil, like if (A?nil) B.)

Now we can start to imagine what conditionals in this language might look like.

`else if`

We’ve replaced if and else with binary operators, so we should verify that else if still works the way it’s supposed to:

let sign = if (n > 0)
    1
else if (n < 0)
    -1
else
    0;

It does, so long as else is right associative. That is, the grouping needs to be like this:

    if (n > 0) 1 else if (n < 0) -1 else 0
    ------------      -------------
                      --------------------
    --------------------------------------

`or if`

Surprisingly there’s another way to write multi-way conditionals:

let sign = if (n > 0)
    1
or if (n < 0)
    -1
else
    0;

This behaves exactly the same as else if! It’s relying on the or binding tighter than the else:

    if (n > 0) 1 or if (n < 0) -1 else 0
    ------------    -------------
    -----------------------------
    ------------------------------------

It took me a bit to get past my “WTF” reaction, but now I kind of like saying “or if”.

`true` and `false`

Remember that true and false are simply shorthands for Ok(nil) and Err(nil). So you can write:

fn delete_file(path: FilePath) -> nil ? IOError {
    if (not file_exists(path)) return true;

    ...
}

and:

fn pop(vec: &mut Vec<i32>) -> i32 ? nil {
    if (vec.len == 0) return false;

    ...
}

`is`

It’s very useful to be able to bind a pattern inside a conditional. Rust uses the syntax if let for this; let’s use the syntax is instead:

fn parse_and_clamp(s: &str, min: i32, max: i32) -> i32 ? nil {
    if (parse_num(s) is Ok(n)) {
        if (n < min) min
        or if (n > max) max
        else n
    }
}

`try / else`

Python lets you put an else clause after a for loop. The else clause runs if the for loop doesn’t break. For example:

for elem in list:
    if predicate(elem):
        print("Found!", elem)
        break
else:
    print("Not found :-(")

There’s a natural extension of this in our hypothetical language – a for loop can break with a value, and prodcues Ok if it does:

fn find(list: &Vec<i32>, predicate: fn(i32) -> bool) -> i32 ? nil {
    for elem in list {
        if (predicate(elem)) break elem;
    }
}

This can be composed with our existing else construct:

fn find_with_default(
    list: &Vec<i32>,
    predicate: fn(i32) -> bool,
    default: i32
) -> i32 {
    for elem in list {
        if (predicate(elem)) break elem;
    } else default
}

Fewer `Ok`s and `Err`s

Many functions which would require wrapping their result in Ok or Err or Some if they were written in Rust, no longer require that. For example, parsing a boolean in Rust:

fn parse_bool(s: &str) -> Option<bool> {
    if s == "true" {
        Some(true)
    } else if s == "false" {
        Some(false)
    } else {
        None
    }
}

vs. in our hypothetical language:

fn parse_bool(s: &str) -> bool ? nil {
    if (s == "true") true
    or if (s == "false") false
}

Real Messy Examples

All the examples so far have been short. Let’s look at some real code, code from deep inside the guts of my personal projects. No need to understand what it does (I certainly don’t remember the details), we just need to rewrite it in this hypothetical language and see how it compares.

Here’s one snippet:

if !node.can_have_children(s) {
    return None;
}
if let Some(last_child) = node.last_child(s) {
    Some(Location(AtNode(last_child)))
} else {
    Some(Location(BelowNode(node)))
}

which can now be written as:

if (node.can_have_children(s)) {
    if (node.last_child(s) is Ok(last_child))
        Location(AtNode(last_child))
    else
        Location(BelowNode(node))
}

And other example:

if let Some(menu) = &mut self.active_menu {
    if let Some(key_prog) = menu.lookup(key) {
        return Some(KeyLookupResult::KeyProg(key_prog));
    }
    if let Some(ch) = key.as_plain_char() {
        if menu.execute(MenuSelectionCmd::Insert(ch)) {
            return Some(KeyLookupResult::Redisplay);
        }
    }
} else {
    let layer = self.composite_layer(doc_name);
    let keymap = layer.keymaps.get(&KeymapLabel::Mode(mode));
    if let Some(key_prog) = keymap.lookup(key, None) {
        return Some(KeyLookupResult::KeyProg(key_prog));
    }
    if mode == Mode::Text {
        if let Some(ch) = key.as_plain_char() {
            return Some(KeyLookupResult::InsertChar(ch));
        }
    }
}
None

would be written as:

if (&mut self.active_menu is Ok(menu)) {
    if (menu.lookup(key) is Ok(key_prog)) {
        KeyLookupResult::KeyProg(key_prog)
    } or if (key.as_plain_char() is Ok(ch)
           and menu.execute(MenuSelectionCmd::Insert(ch))) {
        KeyLookupResult::Redisplay
    }
} else {
    let layer = self.composite_layer(doc_name);
    let keymap = layer.keymaps.get(&KeymapLabel::Mode(mode));
    if (keymap.lookup(key, false) is Ok(key_prog)) {
        KeyLookupResult::KeyProg(key_prog)
    } or if (mode == Mode::Text and key.as_plain_char() is Ok(ch)) {
        KeyLookupResult::InsertChar(ch)
    }
}

Notice that this loses the need for wrapping the results in Some and the need for early return statements.

Conclusion

This sure seems to be a coherent design! It will certainly influence how I think about conditionals.

The closest thing I’ve seen is fallible expressions in Verse, but those are pretty different because they (i) don’t assign a value to an if without an else, and (ii) involve speculative execution. Let me know if you’ve seen anything more similar.

UPDATES:

I mentioned Verse, but commenters point out that its ideas are much older, originally called “goal directed evaluation” and developed in SNOBOL and Icon in the sixties and seventies.
Many dynamically typed languages allow and and or to act on values other than booleans, making them behave much like this post suggests.
This post used to say that not a?e had type nil?nil. Commenter dpercy points out that it could instead have type e?a. This seems so right that I edited the post to make it so.
Jimmy Koppel says of this post “So basically, Rust’s or_else and friends, but as built-in syntax.” Yup, exactly that. Plus the related realization that if and else can be binary operators.

Discussion was on r/programming and lobste.rs.

JJ Cheat Sheet

Tue, 11 Feb 2025 00:00:00 -0500

Have a cheat sheet for Jujutsu.

I’ve been learning Jujutsu a.k.a. jj, a version control system that’s compatible with git repos. It’s clicked for me in a way that git hasn’t even after many years of use.

The best way to learn something is to teach it, so I wrote a reference and cheat sheet for jj with the help of a friend:

The reference describes the state space of a jj repository and how it changes when you fetch and push.
The cheat sheet visually shows what all of the common editing operations do to the repo state.

While there have been several excellent tutorials on Jujutsu, I haven’t seen anything that quite fills these roles. And importantly, they each fit on a page, so it’s possible to print it out double sided and keep it on your desk while learning jj!

While you can of course approach jj any way you please, if you’re truly new to it I would suggest first reading a tutorial first to grok some of the basics. Steve Klabnik’s tutorial, and Kuba Martin’s introduction are both excellent.

(A word about why you should read a tutorial first. If you’re coming from git, you may think of a repo as a collection of branches, or a merge conflict as part of an operation. These intuitions will stand in the way of you learning jj, and a tutorial is a better way to unlearn them than a reference.)

Without further ado, here’s the reference and cheat sheet. It’s written in Typst and the source is on Github.

Typst as a Language

Sat, 30 Nov 2024 00:00:00 -0500

Investigating the Typst programming language.

Typst is a modern typesetting system, and a competitor to LaTeX. You can roughly divide it into two parts: it’s a programming language glued to a layout engine.

The layout engine deals with	The programming language deals with
- Margins	- Data representation
- Padding	- Cyclic data
- Subscripts	- Aliasing
- Floating Figures	- Mutability
- Numbered references	- Garbage collection
- Justified text	- Control flow
- Right-to-left languages	- Functions
- Hypenation points in words	- Composability

This post investigates the programming language inside Typst. Since typesetting systems have very different requirements from most programming languages, Typst lacks some common programming language features but also includes unique features I haven’t seen elsewhere. This makes it a fascinating case study.

(On the other hand, if you’d like to learn about Typst as a layout engine, here’s an in depth post from that perspective.)

I originally wrote this post in Typst itself. If you’d like to view it that way, here’s the PDF and the source.

Overview

The most basic way to use Typst is as a markup language that’s reminiscent of Markdown, plus the ability to write math inside $s:

= Heading

First paragraph, with _emphasis_
and *stronger emphasis*.

Second paragraph, with fancy math
$sqrt(5^2 - 4^2) = 3$.

➔

All of this is content, which directly describes what to render to the screen. In this blog post, we’ll focus instead on code. Here’s an example with both content and code:

1 + 2 = #(1 + 2)

➔

The + on the left is content—it appears in the output—while the + inside the code #(...) performs addition.

Ultimately everything you write in Typst is in one of three modes:

Markup Mode: The default mode, that every file starts in. If you’re in code mode, you can get back to markup mode with [...].
Math Mode: Enter math mode with $...$ .
Code Mode: Enter code mode with #variable or #(expression) or #{multiple lines of code}.

Since we’re interested in the programming language of Typst, we’ll mostly be using Code Mode.

Values

Let’s talk about the different kinds of values Typst has.

Content

The most pervasive kind of value in Typst is content; it’s what you get when you write in markup mode or math mode, and it’s what is ultimately rendered.

Typst has a lot of builtin functions from content to content. For example, _underscores_ are a shorthand for a call to the function emph, which (by default) italicizes the content it’s given:

_Underscores_ are a shorthand for
#emph[a call to emph].

➔

So emph is a function from content to content. Notice that its argument is passed in square brackets [...] to mark the argument as markup.

I’ll split the rest of Typst’s values into primitive values, which cannot contain other values, and compound values which can.

Primitive Values

Typst has the standard primitives you’d expect: none, booleans, integers, floats, and strings. (You might be surprised by the strings since we already have content; I’ll get to that in a minute.)

It also has some measurement types specialized for typesetting:

Length (e.g. 2pt, 3mm). Used for things like the size of a font or the margins of a page.
Angle (e.g. 10deg). Used for shapes and rotations and such.
Fraction (e.g. 2fr), ratio (e.g. 50%), and relative (e.g. 100% - 5mm). Specifies how big something else should be compared to other things. I won’t say more because it’s more of a layout topic than a programming language topic.
Colors can be specified in a variety of color spaces, including rgb and hsl and oklab. It’s really great that perceptually uniform color spaces like oklab and cielab are built in.

Coming back to strings. Strings are different from content! Content has styling, such as font and size and color, while strings don’t.

We haven’t seen a string yet; how can you construct one? Since Typst wants to work as a convenient markup language, just putting double quotes in markup mode won’t produce a string, it will produce quote marks. This covers the common case where you want to write down what someone said and have it be shown in matching quote marks:

"We're _almost_ there," she said.

➔

Instead, you need to be in code mode to construct a string:

#"This is a _string_."

➔

Notice how the underscores in the string were rendered as-is. A string is a sequence of Unicode characters; only content has style.

There’s something funny about this example. I told you that this is an example of a string (which is true), but that only content is rendered to the screen (also true). How then is this string getting shown?

Implicit Coercion to Content

What’s happening is that the string is getting implicitly converted into content. This implicit conversion actually happened in an even earlier example too:

1 + 2 = #(1 + 2)

➔

The 1 on the left is content, while the 1 on the right is a number, and 1 + 2 on the right produces the number 3. Because the code block is inside content, the number it evalutes to is converted into content. Here’s an example where this conversion is more visible:

3.0 = #3.0

➔

The conversion removes the redundant .0, while the content keeps it. (As a more extreme example, 3.x is perfectly fine content, but #3.x is an error because 3 doesn’t have a field called x.)

So strings and numbers are implicitly converted into content when they’re used inside content. In fact, all values can be converted into content. Here’s an array, and how it’s printed as content:

#(3.0, 17in, "banana",
  rgb(50, 100, 150))

➔

(Tangent: it’s a little weird that 3.0 is displayed differently in an array than when it’s standalone. Note the .0 and the syntax highlighting when it’s in the array. My guess is that Typst assumes that if you put #3.0 in content you mean to render it as a number for readers, but if you put the array #(3.0,) in content you mean to render it for debugging purposes. And it’s helpful when debugging to get syntax highlighting and to be explicit about decimal points in order to distinguish integers from floats.)

Compound Values

Let’s look more at compound values like that array. Typst has just two kinds of compound values, which it calls arrays and dictionaries. These are completely standard data structures that almost every language uses. Unfortunately, what they’re called hasn’t exactly been standardized. Here’s a table showing the names for three common data types in various languages:

Language	Fixed Length Array	Growable Array	Hash Map
Typst	-	array	dictionary
Python	-	list	dict
JavaScript	TypedArray	array	object
Java	array	ArrayList	HashMap
C++	array	vector	unordered_map
C#	array	List	Dictionary
Go	array	slice	map
Rust	array	Vec	HashMap
PHP	-	-	array
Ruby	-	Array	Hash
OCaml	Array	Dynarray	HashTbl
Racket	vector	-	hash

The precise definitions for those columns are:

Fixed-length array: An array stored contiguously in memory, of a fixed size, with constant-time indexing.
Growable array: A array stored contiguously in memory, together with a length and a capacity. Also has constant-time indexing. The capacity is how big the array is, and the length is the number of elements that are currently used. If you append to this array and the additional elements fit within the unused capacity, they’re filled in and the length is increased. However if the new length would be greater than the capacity, the array is re-allocated with a (multiplicatively) larger size. The fact that the backing array grows exponentially ensures that appending to it is amortized constant time.
Hash map: A mapping from keys to values. The values are stored in a contiguous array in memory, where the index into the array is determined by the hash of the corresponding key. If the array gets too full, it’s re-allocated with a multiplicatively larger size, ensuring amortized constant time insertion. Some languages, including Typst, require the keys to be strings. (This description doesn’t say what happens if multiple keys have the same hash. There are a few ways to deal with that, all of which change the picture slightly.)

So Typst’s array is a growable array and its dictionary is a hash map.

Typst arrays are written in parentheses and accessed with .at():

#{
  let array = (9, 4, -1)
  array.at(1) + array.at(2)
}

➔

You can also modify the array using .at(), like array.at(0) = 17.

(Tangent: this is a bit unusual. If you write let x = array.at(0), at is a function call that returns the number 9. But if you write array.at(0) = 17, it’s magically setting a value instead of calling a function. Most languages avoid using syntax that looks like function calls in the left hand side of assignments.)

You can also use destructuring in assignment like so:

#{
  let array = (9, 4, -1)

  // Shift elements left one, cycling around
  ((array.at(0), array.at(1), array.at(2)) =
   (array.at(1), array.at(2), array.at(0)))

  array.at(0) + array.at(1)
}

➔

(The extra set of parentheses is needed because the assignment spans two lines. If it were all on one line, you could remove the outer parentheses.)

A dictionary is also written in parentheses, with colons to separate the key from the value:

#{
  let dict = (x: 0.5, y: 2.5)
  dict.x + dict.y
}

➔

You can set a field of a dictionary using familiar syntax:

#{
  let dict = (x: 0.5, y: 2.5)
  dict.x = 5.5
  dict.x - dict.y
}

➔

User-Defined Types

Most languages allow users to define their own types of values (e.g. struct or class). Typst doesn’t support this, and I think that’s a reasonable decision. User-defined types help structure complex programs, but there will be fewer of those in Typst than in a general purpose language.

Control Flow

Control flow is the rules for determining which code will run after which other code. Control flow in Typst is straightforward. It has runtime errors, which can’t be caught. It has the usual control flow constructs if, while, for, return, break, and continue, all of which work in the standard way. There’s a lot of questions in programming language design that are still up in the air, but these constructs we’ve figured out:

#{
  let array = (0, 1, 2, -100, 4)
  let sum = 0
  for x in array {
    if x < 0 { break }
    sum += x
  }
  sum
}

➔

Short-Circuiting

There are another couple control flow constructs that you might not realize are control flow constructs: and and or (spelled as && and || in many languages for historical reasons.

Say you want to check if an array is either empty or starts with none. You would likely write this:

array.len() == 0 or array.at(0) == none

If the array is empty, it’s imperative that array.at(0) == none is not evaluated, because it would raise an error. So x or y first evaluates x, and if true it never evaluates y. Likewise, x and y evaluates x, and if false it never evalutes y.

This is a form of control flow. In fact, and and or together are as expressive as if! You can take any if/else statement:

if CONDITION {
  CONSEQUENCE
} else {
  ALTERNATIVE
}

and convert it into code that only uses and and or but behaves exactly the same:

let _ = CONDITION and {
  CONSEQUENCE
  true
} or {
  ALTERNATIVE
  true
}

(There are some fiddly details in there; understanding them isn’t particularly enlightening. The important thing is to know that and and or can express everything that if can.)

Joining Content

I said that control flow constructs like if and for in Typst were completely standard. That’s mostly true, though they have an interesting interaction with content.

You can use a for loop to perform mutation, like the earlier example that mutates sum += x. You can also use it to join multiple pieces of content together (taking an example from the docs):

#{
  let books = (
    Shakespeare: "Hamlet",
    Homer: "The Odyssey",
    Austen: "Persuasion",
  )
  for (author, title) in books {
    [#author wrote _#(title)_. ]
  }
}

➔

You can also make the whole body of the for loop be content by surrounding it by [...] instead of {...}:

#{
  let books = (
    Shakespeare: "Hamlet",
    Homer: "The Odyssey",
    Austen: "Persuasion",
  )
  for (author, title) in books [
    #author wrote _#(title)_.
  ]
}

➔

(In general, everything that expects {...} including if and while and functions can be given [...] instead.)

This joining behavior is similar to:

Quasiquotation in lisps, where Typst code is to Typst content as lisp macros are to lisp code.
inline for in Zig, where Typst code is to Typst content as Zig comptime code is to Zig runtime code.

That’s all I have to say about control flow in Typst. It’s overall, thankfully, very straightforward.

Abstractions

An abstraction is when you hide irrelevant details of something, so that people who use that thing don’t need to think about them every time they use it. They only need to think about a smaller set of things, called its interface.

For example, a car is an abstraction. To use a car, you need to understand its interface, which includes things like using the gas pedal to accelerate and the steering wheel to turn. You do not need to know all the gnarly details about how the car works like VVT (variable valve timing), ABS (anti-lock braking systems), DCT (dual clutch transmission), APC (adaptive power control), or LSD (limited-slip differentials). Which is fortunate because one of those is made up.

(And yes, abstractions can leak. If your catalytic converter starts leaking, suddenly you need to know what a catalytic converter is, or find someone who does.)

Typst has two forms of abstraction, both of which are common in programming languages.

Functions

The first form of abstraction is functions. They’re written with let syntax and generally work like you’d expect:

#{
  let add-one(n) = {
    n + 1
  }
  add-one(2)
}

➔

As you can see in that example, functions don’t need an explicit return (though you can use it for control flow). If the statements in a function produce multiple pieces of content and there’s no explicit return statement, that content is joined together and implicitly returned:

#{
  let adoption-message(n) = {
    [You have chosen to
     adopt #n kittens.]
    if n > 2 [
      That's a lot of kittens.
    ]
    [Remember to love and
     care for your kittens!]
  }

  adoption-message(3)
}

➔

There’s no sensible way to use return statements in this function, so it’s natural that Typst makes them optional.

There are a couple conveniences for functions. First, there are three kinds of arguments a function can take:

Regular positional arguments. Defined like let f(x, y) = ... and called like f(10, 20).
Optional named arguments that take on a default value. Defined like let f(x: 100) = ... and called like f(x: 101).
An “argument sink” for functions that take any number of arguments. Defined like f(..remaining-args) and called like f(1, 2, 3).

Second, there’s a shorthand for passing content to a function. If you put brackets [...] after the arguments passed to the function, the content in the brackets is passed as the function’s last positional argument. This simple trick is very convenient:

#{
  let repeat(times: 2, stuff) = {
    for i in range(times) {
      stuff
    }
  }
  repeat(times: 3)[
    There's no place like home.
  ]
}

➔

One of the things mentioned in this section is going to turn out to be very important later, and essential for how Typst deals with typesetting.

Modules

Typst’s other form of abstraction is modules. It conflates modules with files, which I think is reasonable for its aims. It’s already dealing with all of typesetting, it should keep the language simple. There are 2.5 ways to “import” a module:

#import "foo.typ" — puts foo in scope. Then you can write foo.helper to access what’s defined with let helper = ... in “foo.typ”.
#import "foo.typ": helper — puts helper directly in scope, so that you don’t need to prefix it with foo.helper every time you use it.
#include "foo.typ" — inlines the content from “foo.typ”. Importantly, style settings that were set by “foo.typ” don’t contaminate the current file; you only get its content. (Much more on style settings later.)

This is almost well done, except for the fact that modules don’t give you any way to mark items as public or private! Thus every let statement in the whole module is exported, even ones like let horrible-fiddly-details-no-one-needs-to-know. There’s an open issue with a good discussion of the tradeoffs of different approaches, but no decision yet about how to move forward.

You can effectively make some items private if you’re willing to wrap the whole module in a let that defines the public items:

// helper-mod.typ
#let (wow, double-wow) = {
  let priv-star = [☆]

  let pub-wow(msg)        = [#priv-star #msg]
  let pub-double-wow(msg) = [#priv-star #msg #priv-star]

  (pub-wow, pub-double-wow)
}

// main-file.typ
#import "helper-mod.typ": double-wow
#double-wow[The stars are out!]

Still, it’s disappointing that every let in a module is public. This means modules aren’t really an abstraction, since the implementation details leak out unless you go out of your way to use a pattern like the above to hide them.

Mutation

Let’s talk about mutation.

Earlier, when I wrote the code dict.x = 5.5, it probably looked innocent to you. I, on the other hand, was laughing maniacally.

Mutation in programming languages is a bargain with the Devil. It’s so easy to implement and seems innocuous, but then the Devil will:

Create cycles, with dict.x = dict. This makes printing values harder (do you print ... at a particular recursion depth, like JS does?), makes serializing and de-serializing values harder, and is the reason you can’t use pure reference counting as a form of garbage collection.
Invalidate an iterator by modifying an array while it’s being iterated over: for x in array { array.pop() }. Is this undefined behavior that can result in memory corruption like in C++, or is there a runtime exception for it like Java’s ConcurrentModificationException, or is it prevented by the compiler like in Rust, or does it have some fixed behavior?
Modify a dictionary as it’s being read in another thread. Is this undefined behavior like in Go, or guaranteed to produce an exception, or prevented by the compiler like in Rust?
Use global mutable variables, which is now generally acknowledged as bad practice.
Generally make it harder to understand and test code, because a function call could modify just about anything. Say you have a bunch of function calls in a row like f() g() h() i(). If i() isn’t behaving the way you want it to, it could be because the behavior of i() depends on a value that’s modified by a function q(), which was called by p(), which was called by g(). I like the term “spooky action at a distance” for this.

Let’s try out some of these examples in Typst, to see how it deals with mutation.

Cycles

#{
  let dict = (x: 1, y: 2)
  dict.x = dict
  dict
}

➔

Woah! That’s unexpected. No cycle, instead just a copy of the original dictionary inside the new one.

Let’s look at more examples and try to spot a pattern in what’s going on.

Iterator Invalidation

#{
  let array = (1, 2, 3, 4, 5)
  for x in array [
    (#x, #array.pop())
  ]
  parbreak()
  [Final array len: #array.len()]
}

➔

So it did pop from array until it became empty, but at the same time the for loop iterated over the original array.

Global Mutable Variables

#{
  let x = 2
  x = 3
  let my-func() = {
    x = 5
  }
  //my-func()
  x
}

➔

We’re actually not allowed to call my-func(). It errors with the message “variables from outside the function are read-only and cannot be modified”. So there are only sort of global mutable variables. You can mutate them outside of functions, and you can access them inside functions, but each function only sees the value the variable had when it was defined:

#{
  let x = 1
  let f() = x

  x += 1
  let g() = x

  f() + g()
}

➔

Mutating Function Arguments

#{
  let modify-dict(dict) = {
    dict.x = 100
  }
  let dict = (x: 1, y: 2)
  modify-dict(dict)
  dict.x + dict.y
}

➔

Ok, that’s really strange. (Unless you’re an R or Swift programmer, in which case this seems completely ordinary and you’re wondering why anyone would think it was strange.) What’s going on?

Value Semantics

Typst has what’s called value semantics.

You can think of value semantics as always copying a value before passing it around:

When you say dict.x = dict, the dict on the right is a copy of the original dictionary.
When you say for x in array, the for loop gets a copy of the array, and iterates over that copy.
When you pass dict to the poorly named modify-dict function, it receives a copy of dict and modifies that.

This gives a lot of advantages. It “foils the Devil”:

You can’t construct cycles. All data is ultimately tree-shaped. This makes printing, serializing, and de-serializing data easier, and it makes pure reference counting a correct garbage collection strategy.
You don’t need to deal with iterator invalidation. It can’t happen; you’re always iterating over the original collection.
Multi-threading becomes easy. The latest release of Typst came with multi-threading support, and it didn’t sound like a Herculean effort. Contrast this to the multi year saga of removing Python’s GIL.
Global variables aren’t mutable from the perspective of a function. For any given function, each global variable is a constant with a fixed value.
It’s easier to reason about code. You know that a function can’t have side effects that influence how later functions behave. As a specific example, this means that the order in which you run test cases can’t matter.

At this point you might be thinking that this sounds great, but it must be really expensive to copy every value each time it’s used. Fortunately, while “copy every value each time it’s used” is a correct mental model of Value Semantics, it’s not the only possible implementation. Typst uses a more efficient “copy-on-write” implementation.

The downside of using copy-on-write is that the performance is unpredictable. As a programmer it isn’t clear when you’re going to accidentally initiate a deep copy. A value gets copied if and only if there is more than one reference to it, but it’s not always obvious if that’s the case.

An alternative is to explicitly distinguish uniquely owned values from references, like the Hylo language does. That is, every variable is either a uniquely owned value that you can modify freely (and no one else can see your changes because you’re the unique owner), or a shared reference that you can’t modify (unless you explicitly copy it to turn it into an owned value). This makes programs more complicated, but makes the places where copies happen explicit.

That’s the tradeoff: copies are either implicit and difficult to predict, or explicit and easy to predict. The fact that Typst is a typesetting system has a couple effects on this tradeoff:

Typst should aim for simplicity, because users should be thinking about typesetting instead of programming.
Copies in Typst are likely to be cheaper than in most other languages. The largest data that’s going to get passed around is content, but content is immutable so it never needs to be deep-copied.

Both of these differences make “implicit but difficult to predict” more attractive, so I think Typst was right to choose that option. (And I think Hylo, being a general purpose language, was right to choose the other option.)

Overall, I think value semantics is a really good fit for Typst.

Unique Features

Most of what we’ve seen so far has been pretty run of the mill. But typesetting has some distinctive challenges that set it apart from other programming domains, and now we’ll look at how Typst handles those challenges.

First Challenge: Tons of Style Options

You should be able to change the text font. As well as the font size, and the spacing between letters, and whether ligatures are enabled, and… well there are a lot of options. How can they all be organized and configured?

Typst solves this problem with one of the features we’ve already seen: arguments with default values. Take text styling. All text that’s rendered in the document is made through calls to a function called text(), which is called implicitly:

Hello world

// Is shorthand for:
#text("Hello world")

➔

There are a lot of options about how the text should be rendered, and they’re configured by (optional) named arguments with default values. Thirty two of them. For example, the weight argument has a default value of “regular”, but you can pass “semibold” to make it bolder (but not “bold” bold):

// Defaults to "regular" weight
Hello world

#text(weight: "semibold", "Hello world")

➔

We talked about text(), but there are many other implicit functions with styles to be configured, like heading() for section headings and enum() for numbered lists. Using named arguments has an advantage over having a single enormous global set of styling options: it provides organization by grouping them by the function they apply to.

Second Challenge: Persistent Settings

What if we want to change the font size of the whole document? Or just part of the document? It would be beyond tedious to wrap every piece of text in the entire document in a call to text() with a size argument.

Typst solves this with a feature called set, which changes the default value of an argument:

"Regular text

#set text(size: 8pt)
Small text

More small text

➔

set can be used on any of the builtin “element” functions that render things, like text(), header(), enum(), and image(). It can’t be used on user-defined functions, though.

set may look like it’s simply setting a global parameter, but it’s not! set never changes anything outside of its scope (the braces {...} or brackets [...] it’s inside of):

#let disclaimer(org) = [
  #set text(size: 8pt)
  These claims have not
  been evaluated by #org.
]

Zyverra instantly heals broken bones.
#disclaimer[the FDA]
Buy Zyverra today!

➔

See that the font size change only effected the disclaimer itself, not the text that came after it.

The fact that set doesn’t “leak out” may seem minor, but it ensures a strong property. If you have some code like this:

Some text.
#some-function()
Some more text.

It is impossible for some-function() to change the font size (or any other property) of the text after it. This is an incredibly valuable guarantee that makes Typst easier to reason about as a user, easier to test, and easier to debug.

I’ve been hand-waving so far, but here’s a precise way to think about set. It has three rules. The first rule is that you first partially evaluate the document, reducing everything down to set statements plus content. For example, this code:

#set text(fill: yellow)
#let scream = {
  set text(fill: red)
  [Aaaagh!]
}
#set text(fill: blue)
#scream Noooooo! #scream

partially evaluates to:

#set text(fill: yellow)
#set text(fill: blue)
#{
  set text(fill: red)
  [Aaaagh!]
}
Noooooo!
#{
  set text(fill: red)
  [Aaaagh!]
}

The second rule is that content is only affected by sets in enclosing scopes (surrounding braces {...} and brackets [...]). Thus #set text(red) won’t apply to “Noooooo!”.

The third rule is that if more than one set statement applies, the last one wins. (And if an argument is given directly, like text(fill: green)[Yessss!], that takes priority over all set statements.)

Putting these together, you should be able to predict what that example is supposed to produce.

Here it is:

#set text(fill: yellow)
#let scream = {
  set text(fill: red)
  [Aaaagh!]
}
#set text(fill: blue)
#scream Noooooo! #scream

➔

Third Challenge: Beyond the Builtin Parameters

What if you want to do something that isn’t covered by one of the builtin parameters to an element function? For example, say you want top-level headings to be centered and in smallcaps.

Typst handles this with a feature called show. It lets you invoke a callback every time one of the bulitin element functions is called. Taking an example from a previous version of the Typst docs:

#show heading.where(level: 1): it => [
  #set align(center)
  #set text(
    13pt,
    weight: "regular"
  )
  #smallcaps(it)
]

= Smallcaps Heading
Putting your headings in smallcaps
makes them look sophisticated.

➔

Breaking this down into pieces:

show heading means we’re going to bind a callback to be invoked on calls to the builtin heading function, which is called whenever you write = Some Heading, == Some Heading, etc.
.where(level: 1) means that we shouldn’t invoke the callback on every call to heading, only on the ones whose level argument is 1. (So it will be invoked on = Some Heading but not on == Some Heading.)
it => [ ... ] is a closure (a.k.a. lambda expression, a.k.a. anonymous function). By convention, the argument is called it, but you can use any name you like. It’s bound to the content returned by the original heading() function. Thus if you used the closure it => it, the show statement would leave the heading unchanged. Instead, our example’s show statement renders the heading differently by setting it’s alignment and weight, and putting it in smallcaps.

I described show statements as simply being callbacks, but actually they’re a lot more complicated than that. They’re more like macros, with a bunch of special cases for how they’re invoked and applied. As an example of some of this complicated behavior, if you write #show heading: it => [some text] then some text comes out bold despite the fact that the callback discards its argument. I kind of want to criticize it for being overly complex, but despite having a PhD in this stuff I don’t have an alternative that’s unambiguously better.

show has the same scoping rules as set: it never affects anything outside of its scope. Again, this enables very powerful reasoning about your document. If you have some text that’s too small, and you’re wondering what changed the font size, the only possibilities are:

There’s an explicit size argument passed to the text function (duh).
There’s a call to set text(size: ...) in the current scope or an enclosing scope.
There’s a call to show text in the current scope or an enclosing scope.

Fourth Challenge: Tweak an Element

Say you want to put little flowers “✿” around each of your headings. There’s no heading(flowers: true) parameter, nor are there heading(prefix: "✿", suffix: "✿") parameters, so you can’t use set for this.

Let’s try show instead:

#show heading: it => [
  ✿#(it)✿
]

== Flowers
make for soothing headings

➔

That sure didn’t work! The trouble, I’m guessing, is that it is already in a block() which forces it to be on its own line, separating it from the flowers.

Let’s try working with the text of the heading directly, which is accessible as it.body (because body is one of the arguments to heading()):

#show heading: it => [
  ✿#(it.body)✿
]

== Flowers
make for soothing headings

➔

Now we have the opposite problem: we didn’t put the heading in a block(), so it isn’t on its own line. Fixing that:

"#show heading: it => [
  #block[✿#(it.body)✿]
]

== Flowers
make for soothing headings

➔

Looks good.

But now we’ve introduced a bug! It’s hard to see because it hasn’t been triggered yet. But say we enable numbering on our headings:

#set heading(numbering: "1.")

== A Heading
before the `show` statement
will be numbered.

#show heading: it => [
  #block[✿#(it.body)✿]
]

== A Heading
after the `show` statement
will not be numbered.

➔

This is because we constructed the heading while ignoring it.numbering. The only general way around this sort of issue is to reconstruct all of the behavior of the heading() function in our show statement. This is the suggested solution on the Typst forums for how to make certain changes to headings or to term lists. This isn’t ideal, as one of the most common reasons to reach for show is to slightly tweak how an element is rendered. It shouldn’t be difficult to do so robustly.

Next I’ll give suggestions for how Typst might be able to improve on this and some other problems.

Suggestions

Evolving a programming language is hard, because you want your users’ existing programs to continue to work indefinitely while improving the language, and those goals are frequently in conflict. Fortunately I’m not a Typst developer, so I can ignore the realities of the situation and dream up whatever I like. Consider these suggestions in light of that!

More Powerful Show

We talked earlier about the fact that if you want to make a small tweak to how an element is rendered, like putting flowers around headings, you may need to write the formatting for that element from scratch, instead of being able to tweak the existing formatting.

As a second example that I actually encountered: if you want to make the terms in a term list not be bold, you have to say how to format each term/definition pair, including the spacing between them, even if you don’t want to change that.

I had an idea for handling these situations by making it legal to write recursive show statements. But I spoke to a core Typst dev who suggested a much simpler approach. Allow show statements to apply not just to element functions, but to their arguments, like so:

show heading > body: it => [✿#it;✿].

This says that in every call to heading(), the body argument (call it it) should be replaced with [✿#it;✿]. Since it’s only modifying the body and not the entire heading, it doesn’t run into the issue with losing the block() wrapper that caused us trouble earlier.

Private Module Items

The fact that #importing a module lets you access every #let in the entire module is an abstraction leakage: some of those #lets are almost certainly implementation details. There’s a discussion of this that covers the tradeoffs between the various possible solutions. I like the suggestion by one commenter:

Everything in a module remains public, unless there’s at least one use of the export keyword. If so, only exported lets are public, and everything else is private. This has the advantage of being backwards compatible except for the introduction of the keyword, and it allows users to be lazy about public/private distinctions in places like modules local to their own projects where privacy doesn’t matter as much.

(The commenter suggested the keyword export, but it could just as well be pub.)

`set` and `show` for User-Defined Functions

For this section we enter a vignette…

You’re a geneticist in the year 2026, and you’re writing a paper. Typst has caught on, and you’re excited to use it for the first time. Your paper is going to mention a lot of genome sequences, and you’re pleasantly surprised to find a ready made package with a sequence() function for displaying them. It has all sorts of formatting options:

type (“DNA” or “RNA”, default “DNA”). Determines whether thymine “T” or uracil “U” should be used.
order (“sense” or “antisense” or none, default none). The order of the sequence: “sense” for 5’ to 3’; “antisense” for 3’ to 5’; none to omit the 3’ and 5’.
codon-sep (string, default “ “). The string used to separate codons. Use an empty string for no separation.
fasta (boolean, default false). Use the standard fasta formatting. Overrides order and codon-sep.

You test it out and see that sequence(type: "RNA", codon-sep: "-", order: "sense", "ATGTGTGGC) produces:

5'-AUG-UGU-GGC-3'

Perfect.

(I know squat about genetics, so I’m hoping this is at least believable if not perfect.)

Then you get to work:

Your paper has 137 genome sequences in it. You want all of them to be separated by dashes, so you write at the top of your paper set sequence(codon-sep: "-"). And they’re all in “sense” order, so you add set sequence(order: "sense").
Your third section is about RNA, and it’s in its own module, so you set sequence(type: "RNA") at the top of the module.
You have an appendix that lists many sequences in fasta format for reference at the end, so you set sequence(fasta: true) in it.

All of this nicely separates content from presentation. The same sequence will be displayed differently depending on which section it’s in. You just write sequence("ATGTGTGGC") and get (e.g.) 5'-AUG-UGU-GGC-3' automatically.

This is good. You smile.

Exiting the vignette…

This story assumes that set works for user-defined functions like sequence, but today it doesn’t. If this story were about Typst today, the geneticist would have had to either:

Manually add the appropriate arguments to all 137 calls to sequence(), even if they were the same throughout the whole document, or
Write a bunch of helper functions, one for every combination of arguments to sequence, and always use those helper functions instead of calling sequence() directly.

But once you’ve learned how set and show work, it’s entirely natural to try to use them on a user-defined function like sequence instead of just on element functions. So my suggestion is to allow this. This is purely backwards compatible, as all current uses would be an (uncatchable) error.

Conclusion

I’m impressed by the design of Typst. Overall, its language is very minimal. Its use of value semantics makes it predictable and easy to debug. And it has a few features for dealing with the complications of typesetting like set and show. (And also context, which I didn’t discuss in this post.)

And this is exactly what you want out of a typesetting system: a programming language that mostly gets out of your way, except to make a few hard things easier.

A Twist on Wadler's Printer

Fri, 23 Feb 2024 00:00:00 -0500

I present a more expressive variant of Wadler’s “Prettier Printer”.

What’s a pretty printer?

You’ve probably used a code formatter like gofmt or rustfmt or JS prettier. These tools work in two steps: (i) parse the source code in a file, and (ii) print it out nicely. Step (ii) is pretty printing.

Pretty printing is the reverse of parsing. Parsing turns linear text into a tree (a parse tree, which after a bit of post-processing becomes an abstract syntax tree (AST)). Pretty printing is the reverse: it turns the tree-shaped AST into linear text.

One of the most commonly used pretty printing algorithms is Philip Wadler’s “A Prettier Printer” [1]. Hereafter I’ll call it Prettier. Prettier is reasonably expressive while being extremely fast (linear time) and simple (50 lines of code), which probably accounts for its popularity.

Implementing Prettier in a strict language. (Click to expand.)

Prettier is written in Haskell and relies on the language being lazily evaluated. If you code it directly in another language, you'll get something that runs in exponential time! But there's an easy modification to the algorithm for strict languages described in "Strictly Pretty" [2]. The code in this post will do essentially the same thing.

In this post, I’ll:

describe how Prettier behaves
describe a variation of it I came up with that’s just as fast while being more expressive
walk through implementing it in Rust, and writing a Json printer using it

Wadler’s Prettier Printer

Say we have a program we want to print. For simplicity of exposition, just a short list: [hello, world] (imagine that “hello” and “world” are variables). There are a couple ways you might want to print this. Either all on one line:

[ hello, world ]

or on separate lines:

[
    hello,
    world
]

Each of these possibilities is called a layout. A pretty printer will have (i) a language in which you can express a variety of possible layouts, and (ii) an algorithm for picking the “best” possible layout from that set. What “best” means depends on the printer, but for Prettier it’s simply “try not to make lines too long” (canonically 80 characters, but you can set that to whatever width you want).

In our tiny example, there are only two possible layouts, but in general there can be many more. Exponentially more, in fact, since alternatives can be nested inside each other. This gives a very large space of possibilities to explore. Prettier’s algorithm explores it in linear time by greedily resolving alternatives one line at a time. We’ll see that in detail later, but for now let’s return to the easier problem of how to express the two layouts above.

Those two layouts can be expressed in Prettier using a combination of:

txt for the text of each token like “[” and “hello”. I’ll typically omit the txt and just write the text directly in quotes, so the examples don’t get too noisy.
nl for newlines.
indent(i, x) to indent every newline in x by i spaces.
x & y to concatenate things together.

I’ll call a mix of these operations a notation.

I'm using different words than Wadler does in his paper.

If you're referencing his paper, he calls the above `text`, `line`, `nest`, and `<>`. I say "indent" instead of "nest" because that's a better word for what it does; "nl" (newline) instead of "line" because "line" could mean a lot of other things; and "&" because you can use that as an operator in Rust. Additionally, what I call a "notation" Wadler calls a "document". I find that a particularly unpleasant word choice because "document" sounds like the thing you wanted to print in the first place, not the pretty printing expression you've converted it to to display it. Unfortunately, "document" is pretty well embedded as a term in the literature now. Fortunately I'm writing a blog post and can call it whatever I please.

We can write a notation that prints as the first layout like so:

"[" & " " & "hello" & "," & " " & "world" & " " & "]"

⇨

[ hello, world ]

And a notation that prints as the second layout:

"[" & indent(4, nl & "hello" & "," & nl & "world") & nl & "]"

⇨

[
    hello,
    world
]

Notice how much these two notations have in common! If you replace every nl with a space " ", the second notation becomes almost identical to the first:

"[" & indent(4, " " & "hello" & "," & " " & "world") & " " & "]"

The only difference is the indent, which no longer matters because there are no newlines to indent.

This is the key to how Prettier represents choices. There’s just one additional notation operator:

group(x) means “Display x with every nl in it replaced by a space, if that would fit on the current line. Otherwise, display x as-is.”

Using group, we can express a choice between the two above layouts with a single notation:

group("[" & indent(4, nl & "hello" & "," & nl & "world") & nl & "]")

Then you can print this notation with various max line widths. If the max width is 80 chars, the group sees that you can replace newlines with spaces and things will fit on one line, so it prints:

[ hello, world ]

but if the max width is 10 it sees that things don’t fit on one line so it prints:

[
    hello,
    world
]

And that’s all! Prettier is basically these five operations, plus a fairly simple and fast algorithm for printing a notation.

My Twist on the Printer

I’ve found a slight variation on this that’s more expressive, while being able to use essentially the same simple and fast printing algorithm.

Instead of group, there are two operations:

x | y for a choice between arbitrary notations x and y. The printer will display x if it fits on the current line, or y otherwise. There are some rules about the properties of x and y that you should follow when writing choices, which will be described later!
flat(x) forces every choice inside of it to pick its leftmost option. It’s called flat because this will typically “flatten” the layout to a single line.

This is only a little bit novel.

Wadler's paper also describes this choice operation and defines `group` in terms of it, but purposefully doesn't expose it to the user because they could break the invariants the printer relies on. It's only the `flat` operation that's new. Wadler's paper has "flatten", which replaces newlines with spaces, while my "flat" picks the first option of each choice.

This is better than group because it’s more powerful. It allows you to specify choices between layouts that differ in more than just whitespace. For example, rustfmt would format our “hello world” example with a trailing comma:

[
    hello,
    world,
]

The trailing comma looks a little silly, but it's nice for git diffs.

If you were to add a third line to the list while using a trailing comma:

[
    hello,
    world,
    turtle,
]

then the diff is only one line:

+    turtle,

But if you were to add that third line while not using trailing commas, then the diff is two lines:

-    world
+    world,
+    turtle

The point is it's at least sensible to put a trailing comma on lists that are split across multiple lines, so it would be nice to support that.

But it would be stupid to put a trailing comma on single line lists!

// stupid:
[ hello, world, ]

So we’d like a printer that can express choices between layouts that differ in more complex ways than simply “newline → space”. There are various ways to slightly extend what Prettier can express, piecemeal. Or we can go all in and use the two operations I describe above, which give a whole lot of power all at once. I like to go all in 😃.

The choice operator will let us eliminate the trailing comma from the single-line layout. As a bonus, we can also remove the spaces just inside the brackets of the single-line layout, giving:

[hello, world]

[
    hello,
    world,
]

To express these two alternatives, all we do is take the single-line notation we want and the multi-line notation we want, and join them with |:

"[" & "hello" & "," & " " & "world" & "]"
|
"[" & indent(4, nl & "hello" & "," & nl & "world" & ",") & nl & "]"

This example doesn’t need flat, but flat comes in handy when we’re writing a notation but don’t know what might appear inside of it. For example, if we’re writing a notation for a generic list containing unknown notations $X and $Y, the single-line layout would want to enforce that the things inside it don’t span multiple lines. It could do so using flat:

"[" & flat($X) & "," & " " & flat($Y) & "]"
|
"[" & indent(4, nl & $X & "," & nl & $Y & ",") & nl & "]"

The Rule

With great power comes great responsibility. The choice | operator gives great power, in that you can make a choice between arbitrary notations. But the printer can’t feasibly check all exponentially many possibilities (remember that choices can be deeply nested), so it must rely on the choices obeying a rule.

The Rule. In every choice x | y, the shortest possible first line of y must be at least as short as every possible first line of x.

The printing algorithm, which I’ll start describing soon, will rely on this choice for correctly determining whether things fit on a line.

There’s another softer rule, which is that if you have a choice x | y then x should not contain forced newlines. If you obey this rule, then flat does what its name says and collapses the entire layout onto a single line. If you don’t the printer will continue to work and flat will continue to pick the first option, but that first option might span multiple lines.

Implementation

Let’s walk through a quick prototype of the printing algorithm in Rust. Then we’ll use it to print Json.

Dependencies

We’ll initialize a rust project with cargo init --lib, then add two dependencies to Cargo.toml:

[dependencies]
unicode-width = "0.1.5"

[dev-dependencies]
serde_json = "1.0"

unicode-width is to get accurate string widths for monospaced terminal fonts including full-width Unicode characters. I’m pretty sure this is an impossible problem but we can at least try.
serde_json is for parsing Json. It’s only a dev-dependency because the pretty printing library won’t rely on it. It’s only needed for the Json parser we’ll build on top of the library later.

src/lib.rs

The library’s going to be split into two modules: one defines a data type for notations, and the other gives the pretty printing algorithm:

mod notation;
mod print;

pub use notation::{flat, indent, nl, txt, Notation};
pub use print::pretty_print;

You might notice that we didn’t export any operations for concat or choice. This is because they’ll be defined with the bitwise operators & and |. (If you really don’t like overloaded operators, feel free to write functions for these instead. And then update the use sites, where you’ll see how clunky it can get.)

src/notation.rs

The notation enum has exactly the operators described in this post, though with a couple interesting details I’ll discuss after showing the code:

use std::ops::{BitAnd, BitOr};
use std::rc::Rc;

#[derive(Debug, Clone)]
pub struct Notation(pub(crate) Rc<NotationInner>);

#[derive(Debug, Clone)]
pub enum NotationInner {
    Newline,
    Text(String, u32),
    Flat(Notation),
    Indent(u32, Notation),
    Concat(Notation, Notation),
    Choice(Notation, Notation),
}

/// Display a newline
pub fn nl() -> Notation {
    Notation(Rc::new(NotationInner::Newline))
}  

/// Display text exactly as-is. The text should not contain a newline!
pub fn txt(s: impl ToString) -> Notation {
    let string = s.to_string();
    let width = unicode_width::UnicodeWidthStr::width(&string as &str) as u32;
    Notation(Rc::new(NotationInner::Text(string, width)))
}

/// Use the leftmost option of every choice in the contained Notation.
/// If the contained Notation follows the recommendation of not
/// putting newlines in the left-most options of choices, then this
/// `flat` will be displayed all on one line.
pub fn flat(notation: Notation) -> Notation {
    Notation(Rc::new(NotationInner::Flat(notation)))
}

/// Increase the indentation level of the contained notation by the
/// given width. The indentation level determines the number of spaces
/// put after `Newline`s. (It therefore doesn't affect the first line
/// of a notation.)
pub fn indent(indent: u32, notation: Notation) -> Notation {
    Notation(Rc::new(NotationInner::Indent(indent, notation)))
}

impl BitAnd<Notation> for Notation {
    type Output = Notation;

    /// Display both notations. The first character of the right
    /// notation immediately follows the last character of the
    /// left notation.
    fn bitand(self, other: Notation) -> Notation {
        Notation(Rc::new(NotationInner::Concat(self, other)))
    }
}

impl BitOr<Notation> for Notation {
    type Output = Notation;

    /// If inside a `flat`, _or_ the first line of the left notation
    /// fits within the required width, then display the left
    /// notation. Otherwise, display the right notation.
    fn bitor(self, other: Notation) -> Notation {
        Notation(Rc::new(NotationInner::Choice(self, other)))
    }
}

Sub-notations that are used in multiple places must be shared, so they’re stored in an Rc (ref-counted). This sharing is very important: without it the notation becomes exponentially larger. For example, notice that the final “hello world” notation repeated “hello” and “world” twice. If you start nesting notations like this, these repetitions multiply together. So it’s important to share them by having multiple references to the same notation on the heap, which is what Rc does. Prettier was written in Haskell, which shares by default, so it didn’t need anything like Rc.

The Text variant contains the width of the string in columns. It makes sense to compute this up front because it will accessed multiple times when determining whether things fit on a line.

Dive into the rabbit hole of string widths.

The width of a string in a monospace font is different from how many bytes it is, because `á` is multiple bytes but width 1, and it's different from the number of Unicode code points or grapheme clusters because "漢" is one character/code-point/grapheme-cluster but has width 2.

The model we're relying on here is: "when you print characters in a monospace font they'll all be printed in that font, that font will only contain single-width and double-width glyphs, and these widths can be determined from the character alone without knowing the OS, terminal, and installed Unicode version". This model is false, but we'll pray it's sufficiently true to be useful and that the `unicode_width` library does a good job at approximating it.

src/print.rs

Here’s the core algorithmic idea from Wadler. When printing a document, we’re going to break it up into a stack of chunks containing everything we haven’t yet printed. Each chunk keeps a reference to a notation, together with the accumulated indentation at that notation and whether it’s inside a flat:

use crate::notation::{Notation, NotationInner};

#[derive(Debug, Clone, Copy)]
struct Chunk<'a> {
    notation: &'a Notation,
    indent: u32,
    flat: bool,
}

A few methods for modifying chunks will come in handy later:

impl<'a> Chunk<'a> {
    fn with_notation(self, notation: &'a Notation) -> Chunk<'a> {
        Chunk {
            notation,
            indent: self.indent,
            flat: self.flat,
        }
    }
    
    fn indented(self, indent: u32) -> Chunk<'a> {
        Chunk {
            notation: self.notation,
            indent: self.indent + indent,
            flat: self.flat,
        }
    }
    
    fn flat(self) -> Chunk<'a> {
        Chunk {
            notation: self.notation,
            indent: self.indent,
            flat: true,
        }
    }
}

The printer needs just three things:

struct PrettyPrinter<'a> {
    /// Maximum line width that we'll try to stay within
    width: u32,
    /// Current column position
    col: u32,
    /// A stack of chunks to print. The _top_ of the stack is the
    /// _end_ of the vector, which represents the _earliest_ part
    /// of the document to print.
    chunks: Vec<Chunk<'a>>,
}

To initialize a Printer, set the starting column to 0 and the chunk stack to contain a single chunk whose notation represents the entire document to print:

impl<'a> PrettyPrinter<'a> {
    fn new(notation: &'a Notation, width: u32) -> PrettyPrinter<'a> {
        let chunk = Chunk {
            notation,
            indent: 0,
            flat: false,
        };
        PrettyPrinter {
            width,
            col: 0,
            chunks: vec![chunk],
        }
    }
}

Here’s the method for printing:

impl<'a> PrettyPrinter<'a> {
fn print(&mut self) -> String {
    use NotationInner::*;

    let mut output = String::new();
    while let Some(chunk) = self.chunks.pop() {
        match chunk.notation.0.as_ref() {
            Text(text, width) => {
                output.push_str(text);
                self.col += width;
            }
            Newline => {
                output.push('\n');
                for _ in 0..chunk.indent {
                    output.push(' ');
                }
                self.col = chunk.indent;
            }
            Flat(x) =>
                self.chunks.push(chunk.with_notation(x).flat()),
            Indent(i, x) =>
                self.chunks.push(chunk.with_notation(x).indented(*i)),
            Concat(x, y) => {
                self.chunks.push(chunk.with_notation(y));
                self.chunks.push(chunk.with_notation(x));
            }
            Choice(x, y) => {
                if chunk.flat || self.fits(chunk.with_notation(x)) {
                    self.chunks.push(chunk.with_notation(x));
                } else {
                    self.chunks.push(chunk.with_notation(y));
                }
            }
        }
    }
    output
}
}

Walking through each case:

Text prints the text and adds the text’s width to col.
Newline prints a newline, followed by spaces for the current indentation level. It also updates col.
Flat and Indent set the chunk’s flat/indentation and push it back onto the stack so that we’ll recur on it.
Concat(x, y) splits the chunk into two chunks and pushes them onto the stack. It has to push first y and then x so that x is on the top of the stack and we process it next.
Choice(x, y) is the one interesting case. If we’re inside flat, we pick x. Otherwise we pick x if it fits on the current line or y otherwise. This calls self.fits() which does the hard work.

The fits method is where the sausage gets made. We’re going to essentially scan ahead to see whether things will fit on the current line, but there are some important details.

impl<'a> PrettyPrinter<'a> {
    fn fits(&self, chunk: Chunk<'a>) -> bool {
        use NotationInner::*;
  
        let mut remaining = if self.col <= self.width {         
            self.width - self.col
        } else {
            return false;
        };
        let mut stack = vec![chunk];
        let mut chunks = &self.chunks as &[Chunk];

        loop {
            let chunk = match stack.pop() {
                Some(chunk) => chunk,
                None => match chunks.split_last() {
                    None => return true,
                    Some((chunk, more_chunks)) => {
                        chunks = more_chunks;
                        *chunk
                    }
                },
            };

            match chunk.notation.0.as_ref() {
                Newline => return true,
                Text(_text, text_width) => {
                    if *text_width <= remaining {
                        remaining -= *text_width;
                    } else {
                        return false;
                    }
                }
                Flat(x) => stack.push(chunk.with_notation(x).flat()),
                Indent(i, x) =>
                    stack.push(chunk.with_notation(x).indented(*i)),
                Concat(x, y) => {
                    stack.push(chunk.with_notation(y));
                    stack.push(chunk.with_notation(x));
                }
                Choice(x, y) => {
                    if chunk.flat {
                        stack.push(chunk.with_notation(x));
                    } else {
                        // Relies on the rule that for every choice
                        // `x | y`, the first line of `y` is no longer
                        // than the first line of `x`.
                        stack.push(chunk.with_notation(y));
                    }
                }
            }
        }
    }
}

Let’s walk through this piece by piece.

The amount of space remaining on the current line is the max line width self.width minus the current column self.col, though if we’re already past the width limit we can return false because things really don’t fit.

let mut remaining = if self.col <= self.width {         
    self.width - self.col
} else {
    return false;
};

Just like in print(), we want to keep a stack of chunks to process. Though now we’re just going to scan ahead to see if stuff fits on the current line, without actually printing anything. We therefore don’t want to modify self.chunks because modifications should not persist after the fits() method finishes. However, we don’t want to copy self.chunks because that would be inefficient. So instead we’ll keep a working stack called stack and a reference to the self.chunks that we haven’t yet processed called chunks:

let mut stack = vec![chunk];
let mut chunks = &self.chunks as &[Chunk];

The first thing the loop will do is “get the next chunk”. Since there are two places it might come from this is a little complicated, but all this code says is “take a chunk from the stack, or if that’s empty take it from chunks, or if that’s empty too return true”:

loop {
    let chunk = match stack.pop() {
        Some(chunk) => chunk,
        None => match chunks.split_last() {
            None => return true,
            Some((chunk, more_chunks)) => {
                chunks = more_chunks;
                *chunk
            }
        },
    };

Then the body of the loop processes the chunk to check whether it fits on the line:

match chunk.notation.0.as_ref() {
    Newline => return true,
    Text(_text, text_width) => {
        if *text_width <= remaining {
            remaining -= *text_width;
        } else {
            return false;
        }
    }
    Flat(x) => stack.push(chunk.with_notation(x).flat()),
    Indent(i, x) => stack.push(chunk.with_notation(x).indented(*i)),
    Concat(x, y) => {
        stack.push(chunk.with_notation(y));
        stack.push(chunk.with_notation(x));
    }
    Choice(x, y) => {
        if chunk.flat {
            stack.push(chunk.with_notation(x));
        } else {
            // Relies on the rule that for every choice
            // `x | y`, the first line of `y` is no longer
            // than the first line of `x`.
            stack.push(chunk.with_notation(y));
        }
    }
}

If we hit a newline, there’s nothing more on the line so things must have fit and we can return true.
When we encounter Text, we check if it fits in the remaining space. If it does, we subtract its width from the remaining space. If it doesn’t, we return false.
We handle Flat, Indent, and Concat just like in the print() method, though here we’re pushing to stack instead of self.chunks.
The Choice(x, y) case is the crux of the whole algorithm. When we encounter this choice, it means that we were already checking whether the first option of a choice fits on the current line, because that’s the only situation where fits() is invoked. While doing so, we started looking ahead and encountered this second choice Choice(x, y). There might be a tradeoff between these two choices: perhaps either one can fit on one line but not both at once. We are greedily resolving this tradeoff in favor of the first choice, by asking whether there’s any way the second choice can be resolved such that the first choice fits on the line. This is where The Rule comes in! We ignore x and only check y, by the reasoning that “if y doesn’t fit, then x wouldn’t fit either, so it’s safe to only check if y fits.”

And that’s it! That’s the whole printer.

examples/json.rs

To test our newly completed pretty printing library, let’s add an example usage in example/json.rs for printing Json. It will give functions for constructing Notations for each kind of Json value (null, number, list, etc.):

use pretty::{flat, indent, nl, pretty_print, txt, Notation};

pub fn json_null() -> Notation {
    txt("null")
}

pub fn json_bool(b: bool) -> Notation {
    if b {
        txt("true")
    } else {
        txt("false")
    }
}

pub fn json_string(s: &str) -> Notation {
    // TODO: escape sequences
    txt(format!("\"{}\"", s))
}

pub fn json_number(n: impl ToString) -> Notation {
    txt(n)
}

fn comma_sep_single_line(elems: &[Notation]) -> Notation {
    let mut list = flat(elems[0].clone());
    for elem in &elems[1..] {
        list = list & txt(", ") & flat(elem.clone());
    }
    list
}

fn comma_sep_multi_line(elems: &[Notation]) -> Notation {
    let mut list = elems[0].clone();
    for elem in &elems[1..] {
        list = list & txt(", ") & nl() & elem.clone();
    }
    list
}

fn surrounded(open: &str, elems: &[Notation], closed: &str) -> Notation {
    if elems.is_empty() {
        return txt(open) & txt(closed);
    }

    let single_line = txt(open) & comma_sep_single_line(elems) & txt(closed);
    let multi_line = txt(open) & indent(4, nl() & comma_sep_multi_line(elems)) & nl() & txt(closed);
    single_line | multi_line
}

pub fn json_array(
    elems: impl IntoIterator<Item = Notation>
) -> Notation {
    let elems = elems.into_iter().collect::<Vec<_>>();
    surrounded("[", &elems, "]")
}

fn json_object_entry(key: String, value: Notation) -> Notation {
    json_string(&key) & txt(": ") & value
}

pub fn json_object(
    entries: impl IntoIterator<Item = (String, Notation)>
) -> Notation {
    let entries = entries
        .into_iter()
        .map(|(key, val)| json_object_entry(key, val))
        .collect::<Vec<_>>();
    surrounded("{", &entries, "}")
}

We could call these methods manually:

json_array([json_string("hello"), json_string("world")])

but it would be tedious to build a large enough example to be interesting.

Instead let’s make a Json code formatter that uses serde_json to parse Json and then re-prints it with our pretty printer. serde_json has a representation of Json called serde_json::Value. We’ll need a function to convert that to a Notation:

fn json_to_notation(json: serde_json::Value) -> Notation {
    use serde_json::Value::{Array, Bool, Null, Number, Object, String};

    match json {
        Null => json_null(),
        Bool(b) => json_bool(b),
        Number(n) => json_number(n),
        String(s) => json_string(&s),
        Array(elems) =>
            json_array(elems.into_iter().map(json_to_notation)),
        Object(entries) => json_object(
            entries
                .into_iter()
                .map(|(key, val)| (key, json_to_notation(val))),
        ),
    }
}

And now we can glue everything together to re-format a Json file. Let’s record timing info to tell how long each phase takes:

const WIDTH: u32 = 120;

fn main() {
    use std::env;
    use std::fs::File;
    use std::io::BufReader;
    use std::time::Instant;

    // Get the filename to parse from the command line args
    let env_args = env::args().collect::<Vec<_>>();
    if env_args.len() != 2 {
        panic!("Usage: cargo run --release --example json FILENAME.json");
    }
    let filename = &env_args[1];

    // Parse the file into json using serde
    let start = Instant::now();
    let file = File::open(filename).unwrap();
    let reader = BufReader::new(file);
    let json = serde_json::from_reader(reader).unwrap();
    let ms_to_parse = start.elapsed().as_millis();

    // Convert it to a Notation
    let start = Instant::now();
    let notation = json_to_notation(json);
    let ms_to_construct = start.elapsed().as_millis();

    // Pretty print the Notation
    let start = Instant::now();
    let output = pretty_print(&notation, WIDTH);
    let ms_to_pretty_print = start.elapsed().as_millis();

    // Print it to the terminal
    let start = Instant::now();
    println!("{}", output);
    let ms_to_output = start.elapsed().as_millis();

    // Print timing info to stderr
    eprintln!("Time to parse file as Json:    {} ms", ms_to_parse);
    eprintln!("Time to construct Notation:    {} ms", ms_to_construct);
    eprintln!("Time to pretty print Notation: {} ms", ms_to_pretty_print);
    eprintln!("Time to print to terminal:     {} ms", ms_to_output);
}

Running this on a big Json file of Pokémon on my laptop:

cargo run --release --example json pokemon.json

gives:

{
    "abilities": [
        {
            "ability": {"name": "overgrow", "url": "https://pokeapi.co/api/v2/ability/65/"}, 
            "is_hidden": false, 
            "slot": 1
        }, 
        {
            "ability": {"name": "chlorophyll", "url": "https://pokeapi.co/api/v2/ability/34/"}, 
            "is_hidden": true, 
            "slot": 3
        }
    ], 
    "base_experience": 64, 
    "cries": {
        "latest": "https://raw.githubusercontent.com/PokeAPI/cries/main/cries/pokemon/latest/1.ogg", 
        "legacy": "https://raw.githubusercontent.com/PokeAPI/cries/main/cries/pokemon/legacy/1.ogg"
    },
    "forms": [{"name": "bulbasaur", "url": "https://pokeapi.co/api/v2/pokemon-form/1/"}], 
    "game_indices": [
        {"game_index": 153, "version": {"name": "red", "url": "https://pokeapi.co/api/v2/version/1/"}}, 
        {"game_index": 153, "version": {"name": "blue", "url": "https://pokeapi.co/api/v2/version/2/"}}, 
        {"game_index": 153, "version": {"name": "yellow", "url": "https://pokeapi.co/api/v2/version/3/"}}, 
        {"game_index": 1, "version": {"name": "gold", "url": "https://pokeapi.co/api/v2/version/4/"}}, 
        ...
        {"game_index": 1, "version": {"name": "white-2", "url": "https://pokeapi.co/api/v2/version/22/"}}
    ], 
    "height": 7, 
    "held_items": [], 
    "id": 1,    
    "is_default": true, 
    "location_area_encounters": "https://pokeapi.co/api/v2/pokemon/1/encounters", 
    "moves": [
        {
            "move": {"name": "razor-wind", "url": "https://pokeapi.co/api/v2/move/13/"}, 
            "version_group_details": [
                {
                    "level_learned_at": 0, 
                    "move_learn_method": {"name": "egg", "url": "https://pokeapi.co/api/v2/move-learn-method/2/"}, 
                    "version_group": {"name": "gold-silver", "url": "https://pokeapi.co/api/v2/version-group/3/"}
                }, 
                {
                    "level_learned_at": 0, 
                    "move_learn_method": {"name": "egg", "url": "https://pokeapi.co/api/v2/move-learn-method/2/"}, 
                    "version_group": {"name": "crystal", "url": "https://pokeapi.co/api/v2/version-group/4/"}
                }
            ]
        },
...
6737 lines of Json
...
Time to parse file as Json:    2 ms
Time to construct Notation:    8 ms
Time to pretty print Notation: 2 ms
Time to print to terminal:     36 ms

(This is printed at width 120 to make the output more interesting by having more things fit on a line, then hackily squeezed into my website’s rather narrow code blocks.)

So:

Only 2ms to parse the file. serde_json is fast.
It takes longer to construct the Notation than to print it! I suspect that most of the time comes from the Rcs. There are faster techniques for sharing, but they’re pretty heavy handed so I won’t show them here. But here’s a branch that does arena allocation.
The parsing and printing together is much faster than outputting to the terminal.

So there it is: a fairly simple, fairly fast, fairly expressive pretty printer. It’s Wadler’s Prettier Printer, with a twist.

The full code is on Github.

Citations

[1] Philip Wadler. “A Prettier Printer”. The Fun of Programming, Cornerstones of Computing, 2003. link.

[2] Christian Lindig. “Strictly Pretty”. Informally published, 2000. Link.

justinpombrio.net

Lindenmayer Systems

Turtles

Angles

Aristid Lindenmayer

Code

Gradient

Pretty Pictures

Repository

Imagining a Language without Booleans

Optional if

Optional else

Optional and and or

But booleans?

Putting it all Together

else if

or if

true and false

is

try / else

Fewer Oks and Errs

Real Messy Examples

Conclusion

JJ Cheat Sheet

Typst as a Language

Overview

Values

Content

Primitive Values

Implicit Coercion to Content

Compound Values

User-Defined Types

Control Flow

Short-Circuiting

Joining Content

Abstractions

Functions

Modules

Mutation

Cycles

Iterator Invalidation

Global Mutable Variables

Mutating Function Arguments

Value Semantics

Unique Features

First Challenge: Tons of Style Options

Second Challenge: Persistent Settings

Third Challenge: Beyond the Builtin Parameters

Fourth Challenge: Tweak an Element

Suggestions

More Powerful Show

Private Module Items

set and show for User-Defined Functions

Conclusion

A Twist on Wadler's Printer

What’s a pretty printer?

Wadler’s Prettier Printer

My Twist on the Printer

The Rule

Implementation

Dependencies

src/lib.rs

src/notation.rs

src/print.rs

examples/json.rs

Citations

Optional `if`

Optional `else`

Optional `and` and `or`

`else if`

`or if`

`true` and `false`

`is`

`try / else`

Fewer `Ok`s and `Err`s

`set` and `show` for User-Defined Functions