Breandan's Blog

Computation graphs and graph computation

2020-06-30T00:00:00+00:00

Research has begun to reveal many algorithms can be expressed as matrix multiplication, suggesting an unrealized connection between linear algebra and computer science. I speculate graphs are the missing piece of the puzzle. Graphs are not only useful as cognitive aides, but are suitable data structures for a wide variety of tasks, particularly on modern parallel processing hardware.

In this essay, I explore the virtues of graphs, algebra, types, and show how these concepts can help us reason about programs. I propose a computational primitive based on graph signal processing, linking software engineering, graphs, and linear algebra. Finally, I share my predictions for the path ahead, which I consider to be the start of an exciting new chapter in computing history.

n.b.: None of these ideas are mine alone. Shoulders of giants. Follow the links and use landscape mode for optimal reading experience.

Biographical details
Graph applications
Inductive languages
Inductive graphs
Graph languages
Dynamical systems on graphs
Efficient implementations
Partial evaluation and program synthesis
Future roadmap

New decade, new delusions

Over the last decade, I bet on some strange ideas. A lot of people I looked up to at the time laughed at me. I’ll bet they aren’t laughing anymore. I ought to thank them one day, because their laughter gave me a lot of motivation. I’ve said some idiotic things to be sure, but I’ve also made some laughable predictions that were correct. Lesson learned: aim straighter.

In 2012, I was in Austin sitting next to an ex-poker player named Amir who was singing Hinton’s praises. Hypnotized by his technicolor slides, I quit my job in a hurry and started an educational project using speech recognition and restricted Boltzmann machines. It never panned out, but I learned a lot about ASR and Android audio. Still love that idea.

In 2017, I started writing a book on the ethics of automation and predicted mass unemployment and social unrest. Although I got the causes wrong (pandemic, go figure), the information economy and confirmation bias takes were all dead right. Sadly, this is now driving the world completely insane. Don’t say I warned you, go out and fix our broken systems. The world needs more engineers who care.

In 2017, I witnessed the birth of differentiable programming, which I stole from Chris Olah and turned into a master’s thesis. Had a lot of trouble convincing people that classical programs could be made differentiable, but look at the proceedings of any machine learning conference today and you’ll find dozens of papers on differentiable sorting and rendering and simulation. Don’t thank me, thank Chris and the Theano guys.

In 2018, I correctly predicted Microsoft would acquire GitHub to mine code. Why MS and not Google? I’ll bet they tried, but Google’s leadership had fantasies of AGI and besides JetBrains, MS were the only ones who gave a damn about developers. Now ML4SE is a thriving research area and showing up in real products, much to the chagrin of those who believed ML was a fad. I suspect their hype filter blinded them to the value those tools provide.

Prediction: MS will acquire GH within five years. If the #ML4Code stuff delivers for MS, acquisition is highly likely. Although it would have been cheaper a few years ago. https://t.co/5ZMtiRtifD https://t.co/TaxkArm5ps
— breandan (@breandan) May 7, 2018

But to heck with everything I’ve said! If I had just one idea to share with these ML people, it would be types. Beat that drum as loud as I could. Types are the best tool we know for synthetic reasoning. If you want to build provably correct systems that scale on real-world applications, use types. Not everyone is convinced yet, but mark my words, types are coming. Whoever figures out how to connect types and learning will be the next Barbara Liskov or Frances Allen.

This year, I predicted the pandemic weeks before the lockdown, exited the market, and turned down a job at Google. Some people called me crazy. Now I’m going all-in on some new ideas (none of which are mine). I’m making some big bets and some will be wrong, but I see the very same spark of genius in them.

Everything old is new again

As a kid, I was given a book on the history of mathematics. I remember it had some interesting puzzles, including one with some bridges in a town divided by rivers, once inhabited by a man called Euler. Was there a tour crossing each bridge exactly once? Was it possible to tell without checking every path? I remember spending days trying to figure out the answer.

In the late 90s, my mom and I went to Ireland. I remember visiting Trinity College, and learning about a mathematician called Hamilton who discovered a famous formula connecting algebra and geometry, and carved it onto a bridge. We later visited the bridge, and the tour guide pointed out the stone, which we touched for good luck. The Irish have a thing for stones.

In 2007, I was applying to college and took the train from Boston to South Bend, Indiana, home of the Fighting Irish. Wandering about, I picked up a magazine article by a Hungarian mathematician called Barabási then at Notre Dame, who had some interesting things to say about complex networks. Later in 2009, while studying in Rochester, I carpooled with a nice professor, and learned complex networks are found in brains, languages and many marvelous places.

Fast forward to 2017. I was lured by the siren song of algorithmic differentiation. Olivier Breleux presented Myia and Buche. Matt Johnson gave a talk on Autograd. I met Chris Olah in Long Beach, who gave me the idea to study differentiable programming. I stole his idea, dressed it up in Kotlin and traded it for a POPL workshop paper and later a Master’s thesis. Our contributions were using algebra, shape inference and presenting AD as term rewriting.

In 2019, I joined a lab with a nice professor at McGill applying knowledge graphs to software engineering. Like logical reasoning, knowledge graphs are an idea from the first wave of AI in the 1960s and 70s which have been revived and studied in light of recent progress in the field. I believe this is an important area of research with a lot of potential. Knowledge and traceability plays a big role in software engineering, and it’s the bread-and-butter of a good IDE. The world needs better IDEs if we’re ever going to untangle this mess we’re in.

This Spring, I took a fascinating seminar on Graph Representation Learning. A lot of delightful graph theory has been worked out over the last decade. PageRank turned into power iteration. People have discovered many interesting connections to linear algebra, including Weisfeiler-Lehman graph kernels, graph Laplacians, Krylov methods, and spectral graph theory. These ideas have deepened our understanding of graph signal processing and its applications for learning and program analysis. More on that later.

What are graphs?

Graphs are general-purpose data structures used to represent a variety of data types and procedural phenomena. Unlike most sequential languages, graphs are capable of expressing a much richer family of relations between entities, and are a natural fit for many problems in computer science, physics, biology and mathematics. Consider the following hierarchy of data structures, all of which are graphs with increasing expressive power:

Sets: datasets, multisets, posets, alphabets
Sequences: Lists, strings, arrays, linear function composition
Trees: Abstract syntax, XML, phylogeny, decision trees
DAGs: Git, citations, dependency graphs, workflows, control flow, MLPs
Directed graphs: State machines, λ-calculus, the web, call graphs, RNNs
Hypergraphs: Knowledge, Zettelkasten, categories, physics, hypernetworks

As we realized in Kotlin∇, directed graphs can be used to model mathematical expressions, as well as other formal languages, including source code, intermediate representations and binary artifacts. Not only can graphs be used to describe extant human knowledge, many recent examples have shown that machines can “grow” trees and graphs for various applications, such as program synthesis, mathematical deduction and physical simulation. Recent neuro-symbolic applications have shown promising early results in graph synthesis:

Learning to Represent Programs with Graphs, Allamanis et al., 2018
Deep Learning for Symbolic Mathematics, Lample and Charton, 2019.
Discovering Symbolic Models from Deep Learning with Inductive Biases, Cranmer et al., 2020.
Symbolic Pregression: Discovering Physical Laws from Raw Distorted Video (Udrescu & Tegmark, 2020).
DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning, Ellis et al., 2020.
Strong Generalization and Efficiency in Neural Programs, Li et al., 2020.
Neural Execution of Graph Algorithms, Veličković et al. (2020)

The field of natural language processing has also developed a rich set of graph-based representations, such as constituency, dependency, link and other and other typed attribute grammars which can be used to reason about syntactic and semantic relations between natural language entities. Research has begun to show many practical applications for such grammars in the extraction and organization of human knowledge stored in large text corpora. Those graphs can be further processed into ontologies for logical reasoning.

Using coreference resolution and entity alignment techniques, we can reconstruct internally consistent relations between entities, which capture cross-corpus consensus in natural language datasets. When stored in knowledge graphs, these relations can be used for information retrieval and question answering, e.g. on wikis and other content management systems. Recent techniques have shown promise in automatic knowledge base construction (cf. Reddy et al., 2016).

Lo and behold, the key idea behind knowledge graphs is our old friend, types. Knowledge graphs are multi-relational graphs whose nodes and edges possess a type. Two entities can be related by multiple types, and each type can relate many pairs of entities. We can index an entity based on its type for knowledge retrieval, and use types to reason about compound queries, e.g. “Which company has a direct flight from a port city to a capital city?”, which would otherwise be difficult to answer without a type system.

Induction introduction!

In this section, we will review some important concepts from Chomskyan linguistics, including finite automata, abstract rewriting systems, and λ-calculus. Readers already familiar with these concepts will gain a newfound appreciation for how each one shares a common thread and can be modeled using the same underlying abstractions.

Regular languages

One thing that always fascinated me is the idea of inductively defined languages, also known as recursive, or structural induction. Consider a very simple language that accepts strings of the form 0, 1, 100, 101, 1001, 1010, et cetera, but rejects 011, 110, 1011, or any string containing 11. The → symbol denotes a “production”. The | symbol, which we read as “or”, is just shorthand for defining multiple productions on a single line:

true → 1
term → 0 | 10 | ε
expr → term | expr term

We have two sets of productions, those which can be expanded, called “nonterminals”, and those which can be expanded no further, called “terminals”. Notice how each non-terminal occurs at most once in any single production. This property guarantees the language is recognizable by a special kind of graph, called a finite state machine. As their name suggests, FSMs contain a finite set of states, with labeled transitions between them:

Finite Automaton	Library Courtesy Bell
	Please ring the bell once and wait for assistance.

Imagine a library desk: you can wait quietly and eventually you will be served. Or, you can ring the bell once and wait quietly to be served. Should no one arrive after a while, you may press the bell again and continue waiting. Though you must never ring the bell twice, lest you disturb the patrons and be tossed out.

Regular languages can also model nested repetition. Consider a slightly more complicated language, given by the regular expression (0(01)*)*(10)*. The *, or Kleene star, means, “accept zero or more of the previous token”.

Backus-Naur Grammar	Nondeterminstic Finite Automaton
`t → ε \| 0 a → 10 \| a 10 b → 0 \| b 01 \| b 0 ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎`

Note here, a single symbol may have multiple transitions from the same state. Called a nondeterminsic finite automaton (NFA), this machine can occupy multiple states simultaneously. While no more powerful than their determinstic cousins, NFAs often require far fewer states to recognize the same language. One way to implement an NFA is to simulate the superposition of all states, by cloning the machine whenever such a transition occurs. More on that later.

Arithmetic

Now suppose we have a slightly more expressive language that accepts well-formed arithmetic expressions with up to two variables, in either infix or unary prefix form. In this language, a non-terminal may occur twice inside a single production – an expr can be composed of two subexprs:

term → 1 | 0 | x | y
  op → + | - | ·
expr → term | op expr | expr op expr

This is an example of a context-free language (CFL). We can represent strings in this language using a special kind of graph, called a syntax tree. Each time we expand an expr with a production rule, this generates a rooted subtree on op, whose branch(es) are exprs. Typically, syntax trees are inverted, with branches extending downwards, like so:

Syntax Tree	Peach Tree

While syntax trees can be interpreted computationally, they do not actually perform computation unless evaluated. To [partially] evaluate a syntax tree, we will now introduce some pattern matching rules. Instead of just allowing terminals to occur on the right-hand side of a production, suppose we also allow terminals on the left, and applying a rule can shrink a string in our language. Here, we use capital letters on the same line to indicate an exact match, e.g. a rule U + V → V + U would replace x + y with y + x:

                                         E + E → +E
                                         E · E → ·E
                  E + 1 | 1 + E | +1 | -0 | ·1 → 1
                         E + 0 | 0 + E | E - 0 → E
  E - E | E · 0 | 0 · E | 0 - E | +0 | -1 | ·0 → 0

If we must add two identical expressions, why evaluate them twice? If we need to multiply an expression by 0, why evaluate it at all? Instead, we will try to simplify these patterns whenever we encounter them. This is known as a rewrite system, which we can think of as grafting or pruning the branches of a tree. Some say, “all trees are DAGs, but not all DAGs are trees”. I prefer to think of a DAG as a tree with a gemel:

Rewrite Rule	Deformed Tree

Let us now introduce a new operator, Dₓ, and some corresponding rules. In effect, these rules will push Dₓ as far towards the leaves as possible, while rewriting terms along the way. We will also introduce some terminal rewrites:

[R0]       term → Dₓ(term)
[R1]      Dₓ(x) → 1  
[R2]      Dₓ(y) → 0  
[R3]    Dₓ(U+V) → Dₓ(U) + Dₓ(V)  
[R4]    Dₓ(U·V) → U·Dₓ(V) + Dₓ(U)·V  
[R5]     Dₓ(+U) → +Dₓ(U)
[R6]     Dₓ(-U) → -Dₓ(U)
[R7]     Dₓ(·U) → +U·Dₓ(U)
[R8]      Dₓ(1) → 0
[R9]      Dₓ(0) → 0

Although we assign an ordering R0-R9 for notational convenience, an initial string, once given to this system, will always converge to the same result, no matter in which order we perform the substitutions (proof required):

Term Confluence	Ottawa-St. Lawrence Confluence

This feature, called confluence, is an important property of some rewrite systems: regardless of the substitution order, we will eventually arrive at the same result. If all strings in a language reduce to a form which can be simplified no further, we call such systems strongly normalizing, or terminating. If a rewriting system is both confluent and terminating it is said to be convergent.

λ-calculus

So far, the languages we have seen are capable of generating and simplifying arithmetic expressions, but are by themselves incapable of performing arithmetic, since they cannot evaluate arbitrary arithmetic expressions. We will now consider a language which can encode and evaluate any arithmetic expression:

expr → var | func | appl
func → (λ var.expr)
appl → (expr expr)

To evaluate an expr in this language, we need a single substitution rule. The notation expr[var → val], we read as, “within expr, var becomes val”:

(λ var.expr) val → (expr[var → val])

For example, applying the above rule to the expression (λy.y z) a yields a z. With this seemingly trivial addition, our language is now powerful enough to encode any computable function! Known as the pure untyped λ-calculus, this system is equivalent to an idealized computer with infinite memory.

While grammatically compact, computation in the λ-calculus is not particularly terse. In order to perform any computation, we will need a way to encode values. For example, we can encode the boolean algebra like so:

[D1]           λx.λy.x = T     "true"
[D2]           λx.λy.y = F     "false"
[D3]       λp.λq.p q p = &     "and"
[D4]       λp.λq.p p q = |     "or"
[D5]    λp.λa.λb.p b a = !     "not"

To evaluate a boolean expression !T, we will first need to encode it as a λ-expression. We can then evaluate it using the λ-calculus as follows:

  (           !          ) T
→ (λp.λa.λb.    p     b a) T   [D5]
→ (   λa.λb.    T     b a)     [p → T]
→ (   λa.λb.(λx.λy.x) b a)     [D1]
→ (   λa.λb.(   λy.b)   a)     [x → b]
→ (   λa.λb.(   λy.b)    )     [y → a]
→ (   λa.λb.b            )     [y →  ]
→ (   F                  )     [D2]

We have reached a terminal, and can recurse no further. This particular program is decidable. What about others? Let us consider an undecidable example:

(λg.(λx.g (x x)) (λx.g (x x))) f
    (λx.f (x x)) (λx.f (x x))                        [g → f]
        f (λx.f (x x))(λx.f (x x))                   [f → λx.f(x x)]
        f     f (λx.f (x x))(λx.f (x x))             [f → λx.f(x x)]
        f     f     f (λx.f (x x))(λx.f (x x))       [f → λx.f(x x)]
        ...                 (λx.f (x x))(λx.f (x x)) [f → λx.f(x x)]

This pattern is Curry’s (1930) famous fixed point combinator and the cornerstone of recursion, called Y. Unlike its typed cousin, the untyped λ-calculus is not strongly normalizing and thus not guaranteed to converge. Were it convergent, it would not be Turing-complete. This hard choice between decidability and universality is one which no computational language can avoid.

The λ-calculus, can also be interpreted graphically. I refer the curious reader to some promising proposals which have attempted to formalize this perspective:

An Algorithm for Optimal Lambda Calculus Reduction, (Lample 1990)
A Graphical Notation for the Lambda Calculus (Keenan, 1996)
Visual lambda calculus (Massalõgin, 2008)
Graphic lambda calculus (Buliga, 2013)
Lambda Diagrams (Tromp, 2014)

Cellular automata

The elementary cellular automaton is another string rewrite system consisting of a one dimensional binary array, and a 3-cell grammar. Note there are $2^{2^3} = 256$ possible rules for rewriting the tape. It turns out even in this tiny space, there exist remarkable automata. Consider the following rewrite system:

current pattern	`111`	`110`	`101`	`100`	`011`	`010`	`001`	`000`
next pattern	` 0 `	` 1 `	` 1 `	` 0 `	` 1 `	` 1 `	` 1 `	` 0 `

We can think of this machine as sliding over the tape, and replacing the centermost cell in each matching substring with the second value. Depending on the initial state and rewrite pattern, cellular autoamta can produce many visually interesting patterns. Some have spent a great deal of effort cataloguing families of CA and their behavior. Following Robinson (1987), we can also define an ECA inductively, using the following recurrence relation:

\[a_i^{(t)} = \sum_j s(j)a_{(i-j)}^{t-1} \mod m\]

This characterization might remind us of a certain operation from digital signal processing, called a discrete convolution. We read $f * g$ as “$f$ convolved by $g$”:

\[(f * g)[n] = \sum_m f[m]g[n-m]\]

Here $f$ is our state and $g$ is called a “kernel”. Similar to the λ-calculus, this language also is known to be universal. Disregarding efficiency, we could encode any computable function as an initial state and mechanically apply Rule 110 to simulate a TM, λ-calculus, or any other TC system for that matter.

Cellular automata can also be interpreted as a graph rewriting system, although the benefits of this perspective are not as clear. Unlike string rewriting, graph substitution is much more difficult to implement efficiently, as pattern matching amounts to subgraph isomorphism, which is NP-complete. While there are some optimizations to mitigate this problem, graph grammars do not appear to confer any additional computational benefits. Nevertheless, it is conceptually interesting.

Graphs, inductively

Just like grammars, we can define graphs themselves inductively. As many graph algorithms are recursive, this choice considerably simplifies their implementation. Take one definition of an unlabeled directed graph, proposed by Erwig (2001). Here, the notation list → [item] is shorthand for list → item list, where item is some terminal, and list is just a list of items:

vertex  → int
adj     → [vertex]
context → (adj, vertex, adj)
graph   → empty | context & graph

Erwig defines a graph in four parts. First, we have a vertex, which is simply an integer. Next we have a list of vertices, adj, called an adjacency list. The context is a 3-tuple containing a vertex and symmetric references to its inbound and outbound neighbors, respectively. Finally, we have the inductive case: a graph is either (1) empty, or (2) a context and a graph.

String	Graph
`([3], 4, [1, 3]) & ([1, 2, 4], 3, [4] ) & ([1], 2, [1, 3]) & ([2, 4], 1, [2, 3])`

Let us consider a directed graph implementation in Kotlin. We do not store inbound neighbors, and attempt to define a vertex as a closed neighborhood:

open class Graph(val vertices: Set<Vertex>) { ... }
data class Vertex(neighbors: Set<Vertex>): Graph(this + neighbors)
//                                               ↳ Compile error!

Note the coinductive definition, which creates problems right off the bat. Since this is not accessible inside the constructor, we cannot have cycles or closed neighborhoods, unless we delay edge instantiation until after construction:

class Graph(val vertices: Set<Vertex>) { ... }
class Vertex(adjacencyMap: (Vertex) -> Set<Vertex>) {
  constructor(neighbors: Set<Vertex> = setOf()) : this({ neighbors })
  val neighbors = adjacencyMap(this).toSet()
}

We can now call Vertex() { setOf(it) } to create loops and closed neighborhoods. This definition admits a nice k-nearest neighbors implementation, allowing us to compute the k-hop transitive closure of a vertex or set of vertices:

tailrec fun Vertex.neighbors(k: Int = 0, vertices: Set<Vertex> =
                             neighbors + this): Set<Vertex> =
  if (k == 0 || vertices.neighbors() == vertices) vertices
  else knn(k - 1, vertices + vertices.neighbors() + this)

fun Set<Vertex>.neighbors() = flatMap { it.neighbors() }.toSet()

// Removes all vertices outside the set
fun Set<Vertex>.closure(): Set<Vertex> =
  map { vertex -> Vertex(neighbors.filter { it in this }) }.toSet()

fun Vertex.neighborhood(k: Int = 0) = Graph(neighbors(k).closure())

Another useful representation for a graph, which we will describe in further detail below, is a matrix. We can define the adjacency, degree, and Laplacian matrices like so:

val Graph.adjacency = Mat(vertices.size, vertices.size).also { adj ->
  vertices.forEach { v -> v.neighbors.forEach { n -> adj[v, n] = 1 } }
}

val Graph.degree = Mat(vertices.size, vertices.size).also { deg ->
  vertices.forEach { v -> deg[v, v] = v.neighbors.size }
}

val Graph.laplacian = degree - adjacency

These matrices have some important applications in algebraic and spectral graph theory, which we will have more to stay about later.

Weisfeiler-Lehman

Let us consider an algorithm called the Weisfeiler-Lehman isomorphism test, on which my colleague David Bieber has written a nice piece. I’ll focus on its implementation. First, we need a pooling operator, which will aggregate all neighbors in a node’s neighborhood using some summary statistic:

fun Graph.poolBy(statistic: Set<Vertex>.() -> Int): Map<Vertex, Int> =
  nodes.map { it to statistic(it.neighbors()) }.toMap()

Next, we’ll define a histogram, which just counts each node’s neighborhood:

val Graph.histogram: Map<Vertex, Int> = poolBy { size }

Now we’re ready to define the Weisfeiler-Lehman operator, which recursively hashes the labels until fixpoint termination:

tailrec fun Graph.wl(labels: Map<Vertex, Int>): Map<Vertex, Int> {
  val next = poolBy { map { labels[it]!! }.sorted().hash() }
  return if (labels == next) labels else wl(next)
}

With one round, we’re just comparing the degree histogram. We compute the hash of the entire graph by hashing the multiset of WL labels:

fun Graph.hash() = wl(histogram).values.sorted().hash()

Finally, we can define a test to detect if one graph is isomorphic to another:

fun Graph.isIsomorphicTo(that: Graph) =
  this.nodes.size == that.nodes.size && 
  this.numOfEdges == that.numOfEdges && 
  this.hash() == that.hash()

This algorithm works on many graphs encountered in the wild, however it cannot distinguish two regular graphs with an identical number of vertices and edges. Nevertheless, it is appealing for its simplicity and exemplifies a simple “message passing” algorithm, which we will revisit later. For a complete implementation and other inductive graph algorithms, such as Barabási’s preferential attachment algorithm, check out Kaliningraph.

Graph Diameter

A graph’s diameter is the length of the longest shortest path between any two of its vertices. Let us define the augmented adjacency matrix as $A + A^\intercal + \mathbb{1}$, or:

val Graph.A_AUG = adjacency.let { it + it.transpose() } + ONES

To compute the diameter of a connected graph $G$, we can simply power the augmented adjacency matrix until it contains no zeros:

/* (A')ⁿ[a, b] counts the number of walks between vertices a, b of
 * length n. Let i be the smallest natural number such that (A')ⁱ
 * has no zeros. i is the length of the longest shortest path in G.
 */

tailrec fun Graph.slowDiameter(i: Int = 1, walks: Mat = A_AUG): Int =
  if (walks.data.all { 0 < it }) i 
  else slowDiameter(i = i + 1, walks = walks * A_AUG)

If we consider the complexity of this procedure, we note it takes $\mathcal O(M \mid G\mid)$ time, where $M$ is the complexity of matrix multiplication, and $\mathcal O(Q\mid G \mid^2)$ space, where $Q$ is the number of bits required for a single entry in A_AUG. Since we only care whether or not the entries are zero, we can instead cast A_AUG to $\mathbb B^{n\times n}$ and run binary search for the smallest i yielding a matrix with no zeros:

tailrec fun Graph.fastDiameter(i: Int = 1, 
    prev: Mat = A_AUG, next: Mat = prev): Int =
  if (next.isFull) slowDiameter(i / 2, prev)
  else fastDiameter(i = 2 * i, prev = next, next = next * next)

Our improved procedure fastDiameter runs in $\mathcal O(M\log_2\mid G\mid)$ time. An iterative version of this procedure may be found in Booth and Lipton (1981).

Graph Neural Networks

A graph neural network is like a graph, but whose edges are neural networks. In its simplest form, the inference step can be defined as a matrix recurrence relation $\mathbf H^t := σ(\mathbf A \mathbf H^{t-1} \mathbf W^t + \mathbf H^{t-1} \mathbf W^t)$ following Hamilton (2020):

tailrec fun gnn(
  // Number of message passing rounds
  t: Int = fastDiameter(),
  // Matrix of node representations ℝ^{|V|xd}
  H: Mat,
  // (Trainable) weight matrix ℝ^{dxd}
  W: Mat = randomMatrix(H.numCols),
  // Bias term ℝ^{dxd}
  b: Mat = randomMatrix(size, H.numCols),
  // Nonlinearity ℝ^{*} -> ℝ^{*}
  σ: (Mat) -> Mat = { it.elwise { tanh(it) } },
  // Layer normalization ℝ^{*} -> ℝ^{*}
  z: (Mat) -> Mat = { it.meanNorm() },
  // Message ℝ^{*} -> ℝ^{*}
  m: Graph<G, E, V>.(Mat) -> Mat = { σ(z(A * it * W + it * W + b)) }
): Mat = if(t == 0) H else gnn(t = t - 1, H = m(H), W = W, b = b)

The important thing to note here is that this message passing procedure is a recurrence relation, which like the graph grammar and WL algorithm seen earlier, can be defined inductively. Hamilton et al. (2017) also consider induction in the context of representation learning, although their definition is more closely related to the concept of generalization. It would be interesting to explore the connection between induction in these two settings, and we will have more to say about matrix recurrence relations in a bit.

Graph languages

Approximately 20% of the human cerebral cortex is devoted to visual processing. By using visual representations, language designers can tap into powerful pattern matching abilities which are often underutilized by linear symbolic writing systems. Graphs are one such example which have found many applications as reasoning and communication devices in various domain-specific languages:

Language	Example
Finite automata
Tensor networks
Causal graphs
Category theory
Penrose notation
Feynman diagrams

As Bradley (2019) vividly portrays in her writing, we can think of a matrix as not just a two-dimensional array, but a function on a vector space. This perspective can be depicted using a bipartite graph:

Not only do matrices correspond to graphs, graphs also correspond to matrices. One way to think of a graph is just a boolean matrix, or real matrix for weighted graphs. Consider an adjacency matrix containing nodes V, and edges E, where:

\[\begin{align*} \mathbf A \in \mathbb B^{|V|\times|V|} \text{ where } \mathbf A[u, v] = \begin{cases} 1,& \text{if } u, v \in E \\ 0,& \text{otherwise} \end{cases} \end{align*}\]

Geometric	Matrix

Note the lower triangular structure of the adjacency matrix, indicating it contains no cycles, a property that is not immediately obvious from the naïve geometric layout. Any graph whose adjacency matrix can be reordered into triangular form is a directed acyclic graph. Called a topological ordering, this algorithm can be implemented by repeatedly squaring the adjacency matrix.

Both the geometric and matrix representations impose an extrinsic perspective on graphs, each with their own advantages and drawbacks. 2D renderings can be visually compelling, but require solving a minimal crossing number or similar optimization to make connectivity plain to the naked eye. While graph drawing is an active field of research, matrices can often reveal symmetries that are not obvious from a naïve graph layout (and vis versa).

Matrices are problematic for some reasons. Primarily, by treating a graph as a matrix, we impose an ordering over all vertices which is often arbitrary. Note also its sparsity, and consider the size of the matrix required to store even small graphs. While problematic, this can be overcome with certain optimizations. Despite these issues, matrices and are a natural representation choice for many graph algorithms, particularly on modern parallel processing hardware.

Just like matrices, we can also think of a graph as a function, or transition system, which carries information from one state to the next - given a state or set of states, the graph tells us which other states are reachable. Recent work in graph theory has revealed a fascinating duality between graphs and linear algebra, holding many important insights for dynamical processes on graphs.

Graphs, computationally

What happens when we take a square matrix $\mathbb{R}^{n\times n}$ and raise it to a power? Which kinds of matrices converge and what are their asymptotics? This is a very fertile line of inquiry which has occupied engineers for the better part of the last century, with important applications in statistical phyics, control theory, and deep learning. Linear algebra gives us many tricks for designing the matrix and normalizing the product to promote convergence.

One way to interpret this is as follows: each time we multiply a matrix by a vector $\mathbb{R}^{n}$, we are effectively simulating a dynamical system at discrete time steps. This method is known as power iteration or the Krylov method in linear algebra. In the limit, we are seeking fixpoints, or eigenvectors, which are these islands of stability in our dynamical system. If we initialize our state at such a point, the transition matrix will send us straight back to where we started.

\[f(x, y) = \begin{bmatrix} \frac{cos(x+2y)}{x} & 0 \\ 0 & \frac{sin(x-2y)}{y} \end{bmatrix} * \begin{bmatrix}x\\y\end{bmatrix} = \begin{bmatrix}cos(x+2y)\\sin(x-2y)\end{bmatrix}\]

Locating a point where $f(x, y) = f\circ f(x, y)$, indicates the trajectory has terminated. There exists a famous theorem from Brouwer: given a set $S$ which contains its limits (i.e. closed), whose contents are all a finite distance apart (i.e. bounded), every continuous function from the set onto itself ($f: S \rightarrow S$) will map at least one of its members $x: S$, called a fixpoint, onto itself ($f(x)=x$). Such points describe the asymptotic behavior of our function.

First, let’s get some definitions out of the way.

  𝔹 → True | False
  𝔻 → 1 | ... | 9
  ℕ → 𝔻 | 𝔻0  | 𝔻ℕ
  ℤ → 0 | ℕ   | -ℕ
  ℚ → ℕ | ℤ/ℕ
  ℝ → ℕ | ℕ.ℕ | -ℝ
  ℂ → ℝ + ℝi
  ℍ → ℝ + ℝi + ℝj + ℝk
  T → 𝔹 | ℕ | ℤ | ℚ | ℝ | ℂ | ℍ
  n → ℕ
vec → [Tⁿ]
mat → [[Tⁿ]ⁿ]

We can think of the Krylov method as either a matrix-matrix or matrix-vector product, or a recurrence relation with some normalization:

Krylov Method

There exists in St. Petersburg a naval research facility, known as the Krylov Shipbuilding Research Institute, which houses the world's largest full ocean depth hydraulic pressure tank. Capable of simulating in excess of 20,000 PSI, the DK-1000 is used to test deepwater submersible vessels. At such pressure, even water itself undergoes ~5% compression. Before inserting your personal submarine, you may wish to perform a finite element analysis to check hull integrity. Instabilities in the stiffness matrix may produce disappointing results.

Grammar	Example
`mmp → mat \| mat * mmp mvp → (mmp) * vec`	$(\mathbf{M}\mathbf{M})\mathbf{v}, (\mathbf{M}\mathbf{M}\mathbf{M})\mathbf{v}, \ldots$
`mvp → mat * vec \| mat * (mvp)`	$\mathbf{M}(\mathbf{M}\mathbf{v}), \mathbf{M}(\mathbf{M}(\mathbf{M}\mathbf{v})), \ldots$
`fun → mat * vec / ‖ mat * vec ‖ rec → fun \| mat * rec / ‖ mat * rec ‖`	$\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}, \frac{\mathbf{M}\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}}{\\|\mathbf{M}\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}\\|}, \frac{\mathbf{M}\frac{\mathbf{M}\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}}{\\|\mathbf{M}\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}\\|}}{\\|\mathbf{M}\frac{\mathbf{M}\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}}{\\|\mathbf{M}\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}\\|}\\|}, \ldots$

Regrouping the order of matrix multiplication offers various computational benefits, and adding normalization prevents singularities from emerging. Alternate normalization schemes have been developed for various applications in graphs. This sequence forms the so-called Krylov matrix (Krylov, 1931):

\[\mathbf{K}_{i} = \begin{bmatrix}\mathbf{v} & \mathbf{M}\mathbf{v} & \mathbf{M}^{2}\mathbf{v} & \cdots & \mathbf{M}^{i-1}\mathbf{v} \end{bmatrix}\]

There exists a famous theorem known as the Perron-Frobenius theorem, which states that if $\mathbf M \in \mathcal T^{n \times n}$, then $\mathbf M$ has a unique largest eigenvalue $\lambda \in \mathcal T$ and dominant eigenvector $\mathbf{q} \in \mathcal T^{n}$. It has long been known that under some weak assumptions, $\lim_{i\rightarrow \infty} \mathbf{M}^i \mathbf{v} = c\mathbf{q}$ where $c$ is some constant. We are primarily interested in determinstic transition systems, where $\mathcal T \in \{\mathbb B, \mathbb N\}$.

The Krylov methods have important applications for studying dynamical systems and graph signal processing. Researchers are just beginning to understand how eigenvalues of the graph Laplacian affect the asymptotics of dynamical processes on graphs. We have already seen one example of these in the WL algorithm. Another example of graph computation can be found in Valiant (1975), who shows a CFL parsing algorithm which is equivalent to matrix multiplication.

TIL: CFL parsing can be reduced to boolean matrix multiplication (Valiant, 1975), known to be subcubic (Strassen, 1969), and later proven an asymptotic lower bound (Lee, 1997). This admits efficient GPGPU implementation (Azimov, 2017) in @YaccConstructor https://t.co/3Vbml0v6b9
— breandan (@breandan) June 28, 2020

Yet another example of graph computation can be found in Reps et al. (2016), who show that boolean matrix algebra can be used for abstract interpretation. By representing control flow graphs as boolean matrix expressions, they show how to apply root-finding techniques like Newton’s method (first observed by Esparza et al. (2010)) to dataflow analysis, e.g. for determining which states are reachable from some starting configuration by computing their transitive closure:

Newton's method has some amazing applications for program analysis. Reps et al. (2016) show a mapping between control flow graphs and boolean matrix expressions. Graph reachability amounts to finding fixed points of a semiring equation. What a goldmine! https://t.co/BFCZiJ1b6n pic.twitter.com/Jd86bEXiIu
— breandan (@breandan) July 12, 2020

We could spend all day listing various matrix algorithms for graph computation. Certainly, far better writers have done them justice. Instead, let’s just give some simple examples of dynamical processes on graphs.

Examples

What happens if we define arithmetic operations on graphs? How could we define these operations in a way that allows us to perform computation? As we already saw, one way to represent a directed graph is just a square matrix whose non-zero entries indicate edges between nodes. Just like real matrices in linear algebra, we can add, subtract, multiply and exponentiate them. Other composable operations are also possible.

We will now show a few examples simulating a state machine using the Krylov method. For illustrative purposes, the state simply holds a vector of binary or integer values, although we can imagine it carrying other “messages” around the graph in a similar manner, using another algebra. Here, we will use the boolean algebra for matrix multiplication, where + corresponds to logical disjunction (∨), and * corresponds to logical conjunction (∧):

┌───┬───┬─────┬─────┐
│ x │ y │ x*y │ x+y │        Boolean Matrix Multiplication
├───┼───┼─────┼─────┤ ┌─       ─┐ ┌─ ─┐ ┌─                     ─┐
│ 0 │ 0 │  0  │  0  │ │ a  b  c │ │ j │ │ a * j + b * k + c * l │
│ 0 │ 1 │  0  │  1  │ │ d  e  f │*│ k │=│ d * j + e * k + f * l │
│ 1 │ 0 │  0  │  1  │ │ g  h  i │ │ l │ │ g * j + h * k + i * l │
│ 1 │ 1 │  1  │  1  │ └─       ─┘ └─ ─┘ └─                     ─┘
└───┴───┴─────┴─────┘

Linear chains

Let’s iterate through a linked list. To do so, we will initialize the pointer to the head of the list, and use multiplication to advance the pointer by a single element. We add an implicit self-loop to the final node, and halt whenever a fixpoint is detected. This structure is known as an absorbing Markov chain.

Graph	Matrix	S	S'
	`a b c ┌──────── a │ 0 0 0 b │ 1 0 0 c │ 0 1 1`	`1 0 0`	`0 1 0`
	`a b c ┌──────── a │ 0 0 0 b │ 1 0 0 c │ 0 1 1`	`0 1 0`	`0 0 1`
	`a b c ┌──────── a │ 0 0 0 b │ 1 0 0 c │ 0 1 1`	`0 0 1`	`0 0 1`

Nondeterminstic finite automata

Simulating a DFA using a matrix can be inefficient since we only ever inhabit one state at a time. The real benefit of using matrices comes when simulating nondeterminstic finite automata, seen earlier.

Formally, an NFA is a 5-tuple $\langle Q, \Sigma, \Delta, q_0, F \rangle$, where $Q$ is a finite set of states, $\Sigma$ is the alphabet, $\Delta :Q\times (\Sigma \cup \{\epsilon \})\rightarrow P(Q)$ is the transition function, $q_0 \in Q$ is the initial state and $F \subseteq Q$ are the terminal states. An NFA can be represented as a labeled transition system, or directed graph whose adjacency matrix is defined by the transition function, with edge labels representing symbols from the alphabet and self-loops for each terminal state, both omitted for brevity.

Typical implementations often require cloning the NFA when multiple transitions are valid, which can be inefficient. Instead of cloning the machine, we can simulate the superposition of all states using a single data structure:

Graph	Matrix	S	S'
	`a b c d ┌─────────── a │ 0 0 0 0 b │ 1 0 0 0 c │ 1 0 0 0 d │ 0 1 1 1`	`1 0 0 0`	`0 1 1 0`
	`a b c d ┌─────────── a │ 0 0 0 0 b │ 1 0 0 0 c │ 1 0 0 0 d │ 0 1 1 1`	`0 1 1 0`	`0 0 0 1`
	`a b c d ┌─────────── a │ 0 0 0 0 b │ 1 0 0 0 c │ 1 0 0 0 d │ 0 1 1 1`	`0 0 0 1`	`0 0 0 1`

We encode the accept state as a self cycle in order to detect the fixpoint criterion $S_{t+1} = S_{t}$, after which we halt execution.

Dataflow graphs

Suppose we have the function f(a, b) = (a + b) * b and want to evaluate f(2, 3). For operators, we will need two tricks. First, all operators will retain their state, i.e. 1s along all operator diagonals. Second, when applying the operator, we will combine values using the operator instead of performing a sum.

Graph	Matrix	S	S'
	`a b + * ┌─────────── a │ 0 0 0 0 b │ 0 0 0 0 + │ 1 1 1 0 * │ 0 1 1 1`	`2 3 0 0`	`0 0 5 3`
	`a b + * ┌─────────── a │ 0 0 0 0 b │ 0 0 0 0 + │ 1 1 1 0 * │ 0 1 1 1`	`0 0 5 3`	`0 0 0 15`
	`a b + * ┌─────────── a │ 0 0 0 0 b │ 0 0 0 0 + │ 1 1 1 0 * │ 0 1 1 1`	`0 0 0 15`	`0 0 0 15`

Did you know? Arithmetic expressions can be efficiently parallelized using matrix arithmetic (Miller et al., 1987): https://t.co/9Tr9hImPFA pic.twitter.com/8vBv9phssk
— breandan (@breandan) July 15, 2020

The author was very excited to discover this technique while playing with matrices one day, only later to discover it was described 33 years earlier by Miller et al. (1987). Miller was inspired by Valiant et al.’s (1983) work in arithmetic circuit complexity, who was in turn inspired by Borodin et al.’s (1982) work on matrix computation. This line of research has recently been revisited by Nisan and Wigderson (1997) and later Klivans and Shpilka (2003) which seeks to understand how circuit size and depth affects learning complexity.

Graphs, efficiently

Due to their well-studied algebraic properties, graphs are suitable data structures for a wide variety of applications. Finding a reduction to a known graph problem can save years of effort, but graph algorithms can be challenging to implement efficiently, as dozens of libraries and compiler frameworks have found. Why have efficient implementations proven so difficult, and what has changed?

One issue hindering efficient graph representation is their space complexity. Suppose we have a graph with $10^5=100,000$ nodes, but only a single edge. We will need $10^{5\times 2}$ bits, or about 1 GB to store its adjacency matrix, where an equivalent adjacency list would only consume $\lceil 2\log_2 10^5 \rceil = 34$ bits. Most graphs are similarly sparse. But how do you multiply adjacency lists? One solution is to use a sparse matrix, which is spatially denser proportional to its sparsity and can be linearly faster on parallel computing architectures.

Perhaps the more significant barrier to widespread adoption of graph algorithms is their time complexity. Many interesting problems on graphs are NP-complete, including Hamiltonian path detection, TSP and subgraph isomorphism. Many of those problems have approximations which are often tolerable, but even if exact solutions are needed, CS theory is primarily concerned with worst-case complexity, which seldom or rarely occurs in practice. Natural instances can often be solved quickly using heuristic-guided search, such as SAT or SMT solvers.

Most graph algorithms are currently implemented using object oriented or algebraic data types as we saw previously. While conceptually simple to grasp, this approach is computationally inefficient. We would instead prefer a high level API backed by a pure BLAS implementation. As numerous papers have shown, finding an efficient matrix representation opens the path to optimized execution on GPUs or SIMD-capable hardware. For example, all of the following automata can be greatly accelerated using sparse matrix arithmetic on modern hardware:

Suppose we want to access the source code of a program from within the program itself. How could we accomplish that? There is a famous theorem by Kleene which gives us a clue how to construct a self-replicating program. More specifically, we need an intermediate representation, or reified computation graph (i.e. runtime-accessible IR). Given any variable y, we need some method y.graph() which programmatically returns its transitive closure, including upstream dependencies and downstream dependents. Depending on scope and granularity, this graph can expand very quickly, so efficiency is key.

With the advent of staged metaprogramming in domain-specific languages like TensorFlow and MetaOCaml, such graphs are available to introspect at runtime. By tracing all operations (e.g. using operator overloading) on an intermediate data structure (e.g. stack, AST, or DAG), these DSLs are able to embed a programming language in another language. At periodic intervals, they may perform certain optimizations (e.g. constant propagation, common subexpression elimination) and emit an intermediate language (e.g. CUDA, webasm) for optimized execution on special hardware, such as a GPU or TPU.

This #GraphBLAS stuff is super exciting. Most graph algorithms can be expressed as linear algebra. Sparse matrix SIMD-backed graph algorithms lets us process orders-of-magnitude larger graphs. Similar to AD tools like Theano et al., this will give a huge boost to network science.
— breandan (@breandan) June 29, 2020

Recent work in linear algebra and sparse matrix representations for graphs shows us how to treat many recursive graph algorithms as pure matrix arithmetic, thus benefiting from SIMD acceleration. Researchers are just beginning to explore how these techniques can be used to transform general-purpose programs into graphs. We anticipate this effort will require further engineering to develop an efficient encoder, but see no fundamental obstacle for a common analysis framework or graph-based execution scheme.

Programs as graphs

Graphs are not only useful as data structures for representing programs, but we can think of the act of computation itself as traversing a graph on a binary configuration space. Each tick of the clock corresponds to one matrix multiplication on a boolean tape.

Futamura (1983) shows us that programs can be decomposed into two inputs: static and dynamic. While long considered a theoretical distinction, partial evaluation has been successfully operationalized in several general purpose and domain-specific languages using this observation.

\[\mathbf P: I_{\text{static}} \times I_{\text{dynamic}} \rightarrow O\]

Programs can be viewed as simply functions mapping inputs to output, and executing the program amounts to running a matrix dynamical system to completion. Consider the static case, in which we have all information available at compile-time. In order to evaluate the program, we can just multiply the program $\mathbf P: \mathbb B^{\lvert S\rvert \times \lvert S\rvert}$ by the state $S$ until termination:

    [P]────────────────────────────────           } Program
      ╲          ╲          ╲          ╲
[S₀]───*───[S₁]───*───[S₂]───*───[..]───*───[Sₜ]  } TM tape

Now consider the dynamic case, where the matrix $\mathbf P$ at each time step might be governed by another program:

        [Q]─────────────────────                  } Dynamics
          ╲          ╲          ╲
    [P₀]───*───[P₁]───*───[..]───*───[Pₜ₋₁]       } Program
      ╲          ╲          ╲          ╲
[S₀]───*───[S₁]───*───[S₂]───*───[..]───*───[Sₜ]  } TM tape

We might also imagine the dynamic inputs as being generated by successively higher order programs. Parts of these may be stored elsewhere in memory.

                     ⋮
            [R₀]─────────                         } World model
              ╲          ╲
        [Q₀]───*───[..]───*───[Pₜ₋₂]              } Dynamics
          ╲          ╲          ╲
    [P₀]───*───[P₁]───*───[..]───*───[Pₜ₋₁]       } Program
      ╲          ╲          ╲          ╲
[S₀]───*───[S₁]───*───[S₂]───*───[..]───*───[Sₜ]  } TM tape

What about programs of varying length? It may be the case we want to learn programs where $t$ varies. The key is, we can choose an upper bound on $t$, and search for fixed points, i.e. halt whenever $S_t = S_{t+1}$.

There will always be some program, at the interface of the machine and the real world, which must be approximated. One question worth asking is how large does $\lvert S\rvert$ need to be in order to do so? If it is very large, this procedure might well be intractable. Time complexity appears to be at worst $\mathcal{O}(tn^{2.37})$, using CW matmuls, although considerably better if $\mathbf P$ is sparse.

Program synthesis

Many people have asked me, “Why should developers care about automatic differentiation?” Yes, we can use it to build machine learning systems. Yes, it has specialized applications in robotics, space travel, and physical simulation. But does it really matter for software engineers?

I have been thinking carefully about this question, and although it is not yet fully clear to me, I am starting to see how some pieces fit together. A more complete picture will require more research, engineering and rethinking the role of software, compilers and machine learning.

    [P]────────────────────────────────           } Program
      ╲          ╲          ╲          ╲
[S₀]───*───[S₁]───*───[S₂]───*───[..]───*───[Sₜ]  } TM tape

Consider the static case seen above. Since the matrix $\mathbf P$ is fixed throughout execution, to learn $\mathbf P$, we need to solve the following minimization problem:

\[\underset{P}{\text{argmin}}\sum_{i \sim I_{static}}\mathcal L(P^t S^i_0, S_t)\]

One issue with this formulation is we must rely on a loss over $S_t$, which is often too sparse and generalizes poorly. It may be the case that many interesting program synthesis problems have optimal substructure, so we should be making “progress” towards a goal state, and might be able to define a cross-entropy loss over intermediate states to guide the search process. This intuition stems from RL and needs to be explored in further depth.

Some, including Gaunt et al., (2016), have shown gradient is not very effective, as the space of boolean circuits is littered with islands which have zero gradient (some results have suggested the TerpreT problem is surmountable by applying various smoothing tricks). However Gaunt’s representation is also relatively complex – effectively, they are trying to learn a recursively enumerable language using something like a Neural Turing Machine (Graves et al., 2014).

More recent work, including that of Lample et al., (2019), demonstrated gradient is effective for learning programs belonging to the class of context-free languages. This space is often much more tractable to search through and generate synthetic training data. Furthermore, this appears to be well within the reach of modern language models, i.e. pointer networks and transformers.

In the last year, a number of interesting results in differentiable architecture search started to emerge. DARTS (Liu et al., 2019) proposes to use gradient to search through the space of directed graphs. The authors first perform a continuous relaxation of the discrete graph, by reweighting the output of each potential edge by a hyperparameter, optimizing over the space of edges using gradient descent, then taking a softmax to discretize the output graph.

Solar-Lezma (2020) calls this latter approach, “program extraction”, where a network implicitly or explicitly parameterizes a function, which after training, can be decoded into a symbolic expression. This perspective also aligns with Ian Goodfellow’s notion of deep networks as performing computation, where each layer represents a residual step in a parallel program:

"Can neural networks be made to reason?" Conversation with Ian Goodfellow (@goodfellow_ian). Full version: https://t.co/3MYC8jWjwl pic.twitter.com/tGcDwgZPA1
— Lex Fridman (@lexfridman) May 20, 2019

A less charitable interpretation is that Goodfellow is simply using a metaphor to explain deep learning to lay audience, but I prefer to think he is communicating something deeper about the role of recurrent nonlinear function approximators as computational primitives, where adding depth effectively increases serial processing capacity and layer width increases bandwidth. There may be an interesting connection between this idea and arithmetic circuit complexity (Klivans and Shpilka, 2003).

Roadmap

Much work lies ahead for the interested reader. Before we can claim to have a unification of graph linear algebra and computer science, at least three technical hurdles will need to be cleared. First is theoretical: we must show that binary matrix mutiplication is Turing-equivalent. Second, we must show a proof-of-concept via binary recompilation. Third, we must develop a robust toolchain for compiling and introspecting a wide variety of graph programs.

While it would be sufficient to prove boolean matrix multiplication corresponds to Peano Arithmetic, a constructive proof taking physics into consideration is needed. Given some universal language $\mathcal L$, and a program implementing a boolean vector function $\mathcal V: \mathbb B^i \rightarrow \mathbb B^o \in \mathcal L$, we must derive a transformation $\mathcal T_\mathcal L: \mathcal V \rightarrow \mathcal M$, which maps $\mathcal V$ to a boolean matrix function $\mathcal M: \mathbb B^{j \times k} \times \mathbb B^{l\times m}$, while preserving asymptotic complexity $\mathcal O(\mathcal M) \lt \mathcal O(\mathcal V)$, i.e. which is no worse than a constant factor in space or time. Clearly, the identity function $\mathcal I(\mathcal V)$ is a valid candidate for $\mathcal T_{\mathcal L}$. But as recent GPGPU research has shown, we can do much better.

n.b. Not saying anything about the workload, just the architecture - Software 1.0 may still be the dominant paradigm. I'm saying there is a binary translation from load/store/jump/branch instructions to sparse BLAS primitives which imposes no constraints on the programming model.
— breandan (@breandan) July 1, 2020

The second major hurdle to graph computation is developing a binary recompiler which translates programs into optimized BLAS instructions. The resulting program will eventually need to demonstrate performant execution across a variety of heterogenously-typed programs, e.g. Int, Float16, Float32, and physical SIMD devices. Developing the infrastructure for such a recompiler will be a major engineering undertaking in the next two decades as the world transitions to graph computing. Program induction will likely be a key step to accelerating these graphs on physical hardware.

The third and final hurdle is to develop a robust compiler toolchain for graph computation. At some point, users will be able to feed a short program into a source-to-source transpiler and have the program slightly rewritten with semantics preserving guarantees. This will require progress in abstract interpretation, programming tools, runtime instrumentation, as well as shape-safe libraries and frameworks. Ultimately, we hypothesize users will adopt a more declarative programming style with resource-aware and type-directed constraints. This step will require fundamental progress in program induction and consume the better half of the next century to fully realize.

References

Graph Representation Learning (Hamilton, 2020)
Graph Algorithms in the Language of Linear Algebra (Kepner & Gilbert, 2011)
Analysis of Boolean Functions (O’Donnell, 2014)
Spectral and Algebraic Graph Theory (Spielman, 2019)
GraphBLAS Mathematics (Kepner, 2017)
Term Rewriting and All That (Baader & Nipkow, 1998)
Representation of Events in Nerve Nets and Finite Automata (Kleene, 1951)
A Class of Models with the Potential to Represent Fundamental Physics (Wolfram, 2020)

Trust in Automation

2017-02-02T00:00:00+00:00

In which I discuss the trouble with trusting machines to take our jobs, curate our news feeds, drive our school buses, teach our children, and lots of boring stuff too difficult to bother doing ourselves. Oh, and quotes. Lots of quotes.

Houston, we have a problem. According to social media, a large fraction of our population will soon be unemployed. Not just unemployed, but unemployable. Thanks to rapid growth of automation and recent breakthroughs in machine learning, a majority of the world’s human labor will soon be economically obsolete. Too young to retire, and too old to retrain, there will be an enormous displacement of unskilled labor. This is not just idle speculation. Leading scientists and politicians have recognized the immediacy of this problem, and the importance of addressing it in our society.

Trains will move ~332 million passengers during the largest migration in history.

Automation does not just affect unskilled labor. Many jobs which require advanced degrees and years of experience are vulnerable, including a large number of doctors, lawyers, and financial analysts. Each of these professions does work that is already being learned, automated, and optimized by machines. Even mathematical research at the boundaries of our understanding can be automated. A growing number of mathematicians today use interactive proof assistants and automated theorem provers to verify proofs, and even derive new truths. But if current events are any indication, what is true and what is verifiable are entirely different matters.

And when your surpassing creations find the answers you asked for, you can’t understand their analysis and you can’t verify their answers. You have to take their word on faith—Or you use information theory to flatten it for you, to squash the tesseract into two dimensions and the Klein bottle into three, to simplify reality and pray to whatever Gods survived the millennium that your honorable twisting of the truth hasn’t ruptured any of its load-bearing pylons. You hire people like me; the crossbred progeny of profilers and proof assistants and information theorists…

In formal settings you’d call me Synthesist.

—Peter Watts, Blindsight (2006)

Peter Watts, in his breakout science-fiction novel, Blindsight, imagines the occupation of “synthesists”, professional interpreters who translate between AI and humans. In a future where most scientific breakthroughs are discovered by AI, synthesists, “explain the incomprehensible to the indifferent.” Watts’ protagonist, Siri Keaton, is a spacefaring science officer who encounters a Matrioshka brain outside the solar system. Siri joins a reconnaissance mission to collect observations and verify the true nature of this strange object. But as Siri soon realizes, not all truths can be verified.

Futurist sci-fi tends to fall into three broad categories. The optimists dream of a post-scarcity utopia where technology descends like manna from AI heaven, and we all travel into some digital promised land, en masse so to speak. The pessimists argue that robot overlords and megacorporations vie for control of a dystopian future where humans are mostly expendable. And the synergists suggest a hybrid future, where humans and machines co-exist in relative happiness, clinging to the hope we possess some vestigial importance to our metallic brethren. These thought experiments serve an important function as we grapple with the increasing effects of automation.

People think—wrongly—that speculative fiction is about predicting the future… What speculative fiction is really good at is not the future but the present—taking an aspect of it that troubles or is dangerous, and extending and extrapolating that aspect into something that allows the people of that time to see what they are doing from a different angle and from a different place. It’s cautionary.

—Neil Gaiman, Introduction to Fahrenheit 451 (2013)

Science fiction, or speculative fiction as some prefer, has a long history of anticipating current events, and takes lessons from past and present alike. In one remarkable passage of The Three Body Problem, a man called Von Neumann helps an ancient Chinese emperor build a computer to predict the movements of stars across the sky. With the emperor’s help, he trains millions of soldiers to form logic gates and memory buses, as they raise and lower flags and march around a vast plain. Wherever an error occurs, the emperor simply executes everyone involved and trains new replacements.

Qin Shi Huang lifted the sword to the sky, and shouted: “Computer Formation!” Four giant bronze cauldrons at the corners of the platform came to life simultaneously with roaring flames. A group of soldiers standing on the sloping side of the pyramid facing the phalanx chanted in unison: “Computer Formation!”

On the ground below, colors in the phalanx began to shift and move. Complicated and detailed circuit patterns appeared and gradually filled the entire formation. Ten minutes later, the army had made a thirty-six kilometer square computer motherboard…

“This is really interesting,” Qin Shi Huang said, pointing to the spectacular sight. “Each individual’s behavior is so simple, yet together, they can produce such a complex, great whole! Europeans criticize me for my tyrannical rule, claiming that I suppress creativity. But in reality, a large number of men yoked by severe discipline can also produce great wisdom when bound together as one.”

—Cixin Liu, The Three Body Problem (2008)

Cixin Liu, a sci-fi writer¹ in China, imagines the development of a Kardashev Type-II civilization through allegories. In this example, it is not difficult to see millions of factory workers across China building the devices that will soon replace them. But China is not the only country facing pressure from machines. Many countries with large manufacturing sectors are hugely threatened by the destabilizing presence of automation. As soon as you teach a robot to sew sweaters more cheaply than paying a human, suddenly every sweater-factory can run around-the-clock, displacing thousands of workers overnight. While they’re at it, why bother shipping goods half way around the world?

As the cost of labor goes up and the cost of machinery goes down, at some point, it’ll be cheaper to use machines than people. With the increase in productivity, the GDP goes up, but so does unemployment. What do you do? … The best way is to reduce the time a certain portion of the population spends living, and then find ways to keep them busy.

—Jingfang Hao, Folding Beijing (2014)

But job displacement, while a major challenge, is not the real problem facing our species. As history has shown, humanity has survived dozens of technological upheavals. In the agricultural revolution, nomadic hunter-gatherers started breeding their prey, growing their forage, wheeling their food into little villages. The industrial revolution enlisted those farmers as factory workers and foremen in village-sized machines which consumed raw materials and produced smaller machines. Our ancestors saw sweeping social and economic change and still landed on the moon, despite their share of contemporary detractors. So what is the problem exactly?

For too many of us, it’s become safer to retreat into our own bubbles, whether in our neighborhoods or on college campuses, or places of worship, or especially our social media feeds, surrounded by people who look like us and share the same political outlook and never challenge our assumptions. The rise of naked partisanship, and increasing economic and regional stratification, the splintering of our media into a channel for every taste — all this makes this great sorting seem natural, even inevitable. And increasingly, we become so secure in our bubbles that we start accepting only information, whether it’s true or not, that fits our opinions, instead of basing our opinions on the evidence that is out there.

—Barack Obama, Farewell Address (2016)

At the dawn of the information age, we were convinced a newfangled technology called the “internet” would save us all from the uniformity of traditional media. The growth of the internet would give voices to the voiceless and choices to the choiceless. It was a new media frontier where consumers could create and curate content according to their own tastes and desires. No longer was television the sole source of your daily entertainment. Suddenly, you could read whatever you pleased and tweet whenever you sneezed. Isn’t it great? We can share new ideas and opinions with ease. Even your boss agrees, let’s retweet and reshare this with him overseas!

The internet ushered a great awakening in this new age of information. Politicians and philosophers from ancient Rome could only dream of the freedom that instant access to unlimited information would one day bring to all humankind. What they could not foresee, is how the internet would unleash a new kind of tyranny, one that would eclipse any government’s own misuse in the name of security. Instant access does not guarantee self-improvement, only the promise of easy gratification. Unlimited information does not reveal deeper truth, only an endless road of distractions. Without education, the internet is a tyranny of the mind. Without purpose, it is a prison.

“A prison?” you might say. $\begin{align} \large\textit{Then you will know the truth,} \\ \large\textit{and the truth will set you free.}\\ \textsf{—John, 8:32 (100 A.D., est.)}\end{align}$“Why, it’s full of shiny gadgets, great entertainment, and people who agree with me. That doesn’t sound so bad!” Those shiny gadgets are Skinner boxes. The entertainment? Viral memes, waiting to infect your mind and eat your attention span. Those other people? They’re just reflections who echo our opinions, inflate our egos, and confirm our biases. The machines are very good at keeping us fat and happy. The best part is, we don’t even need to ask. They can model our habits, predict our behavior, anticipate our desires. They can practically read our minds.

Not only can AI anticipate our desires, it can trigger our impulses. And if AI can stimulate our craving for sugar by activating a group of pixels in the correct sequence, why stop at predicting the price of sugar when we can train it to influence future demand? As long as we’re entertaining wild conspiracies, what prevents AI from generating fake news to influence public opinion, or helping elect leaders who are friendly to automation? Whether self-acting or sponsored by old-fashioned capitalism, AI has the potential to metastasize.

All this sounds rather alarmist, and perhaps it is. “Never trust anything you read on the internet,” the adults used to say. The problem is, the age of automation offers opportunity and oppression, education and entertainment, truth and fiction, all in equal measure. The problem is, each of these things begins to look exactly like the other. When information is cheap to produce and free to consume, there is no incentive for it to be trustworthy. How should a color-blind chap in the Matrix know which pill to swallow? As it happens, when dealing with black boxes that can read your mind, the problem becomes surprisingly difficult.

“If our brains were simple enough for us to understand them, we’d be so simple that we couldn’t.”

—Ian Stewart, The Collapse of Chaos (1994)

Researchers have poured millions of dollars into an area of machine learning known as explainable AI. So far as we can tell, it can’t be explained very well. Sure, we know how to build them using GPUs and terabytes of data. We know fancy words like backpropogation, convolutions, and hyperparameters. We know how to poke and prod them, try a zillion different parameters and sometimes they get more accurate. And we know that they work. For most people that is enough. But why does deep learning work so well? And why does one AI classify Amy as a threat to society and Sam as an upstanding citizen? Because her face looks kind of “criminal”?

Automated criminality inference based on facial images is a real thing, and not just in China. Although many flaws exist in criminality prediction, the underlying assumptions are probably correct. With certain technologies, if it works, somebody will find a way to use it. But even if AIs could explain their logic, the problem is not the algorithms themselves. The problem is those who are willing to apply them, regardless of whether they are right, in either the technical or the ethical sense. If some algorithm is 35% confident you will default on a home loan, the bank isn’t going to debug their lending bot until it starts loosing money. The incentives are not aligned in your favor.

Believe in Bias

If you think discrimination is bad today, just wait until the machines take over. They will discriminate based on the shade of your iris, the shape of your brow, the size of a tatoo, or any arbitrary collection of low-level traits whose presence triggers a subtle bias. Regardless whether such traits are truly predictive, it will matter not unless those who benefit have an incentive to fix the model. For most applications, AI just needs to be good enough to yield a positive marginal utility. Barring blatant discrimination on certain parameters like sex or skin color, most biases will fly under the radar.

Treating the world as software promotes fantasies of control. And the best kind of control is control without responsibility. Our unique position as authors of software used by millions gives us power, but we don’t accept that this should make us accountable. We’re programmers—who else is going to write the software that runs the world? To put it plainly, we are surprised that people seem to get mad at us for trying to help.

Fortunately we are smart people and have found a way out of this predicament. Instead of relying on algorithms, which we can be accused of manipulating for our benefit, we have turned to machine learning, an ingenious way of disclaiming responsibility for anything. Machine learning is like money laundering for bias. It’s a clean, mathematical apparatus that gives the status quo the aura of logical inevitability.

—Maciej Ceglowski, On The Moral Economy of Tech (2016)

As experience has shown, preventing bias when training an AI is difficult enough, never mind verifying its neutrality as a subject - if you know the AI exists begin with. But let’s say you have prior knowledge, motive, and a deep background in statistics. If there is an open API and the owner is not careful, you might be able to steal the model, or test it yourself. That’s a lot of “if”s. The average end-user has no hope of ever verifying the fairness of an AI, and no motive to do so unless they are unfairly targeted. If you are unfairly targeted by an AI, the incentives are not in your favor.

$\begin{align} \large\textit{Trust, but verify.}\\ \textsf{—}\href{https://en.wikipedia.org/wiki/Trust,_but_verify}{\textsf{Russian Proverb}}\end{align}$Suppose you are unfairly targeted by an AI. The egregious cases will be tested prior to release to avoid the appearance of discrimination. If you suspect yourself to be the victim of unfair discrimination in an AI decision, it will be a tough case to prove. First you will need to prove the AI exists and had a significant influence in making the decision. Then you will need to establish a statistically significant number of samples where prior discrimination occurred. Good luck getting a subpoena for terabytes of anonymized PII. Finally, you will need to justify why the decision was unfair and how it caused actual harm.

The problem is not algorithms. The problem is the people training AI, and the people they are trained to imitate. The people training life-critical systems are college dropouts and JavaScript developers with something called a “nanodegree” in self-driving cars. If you’re lucky, maybe some of them have an actual degree in statistics or something. Whether out of ignorance or malice, the developers of AI will make mistakes in our headlong pursuit of autonomy. There needs to be a minimum of regulatory oversight for AI that can take a human life. Or at the very least, a certifying body for data scientists, like medical boards and bar associations. In an industry that embraces agility, the bureaucratic cost of these options is highly unattractive.

$\begin{align} \large\textit{Past performance does not} \\ \large\textit{guarantee future results.}\\ \textsf{—}\href{https://www.sec.gov/news/press/2003-122.htm}{\textsf{Mandatory SEC Disclosure}} \textsf{ (2003)}\end{align}$By far the thornier problem, is the data used to train AI. Most AIs today are trained on human data. Imperfect data used to predict the outcome of a scenario no human has exactly seen before, produced by conditions we can only hope are similar during operation. We take great pains to ensure that the data used to train AI comes from the same distribution we expect to see in future operation. There is an entire scientific discipline devoted to correctly sampling, cleaning, and preparing data for machine learning. However, data science departments have budgets limited by economics. And enough data to predict accurate results is seldom enough to prevent unwanted bias.

In order to prevent unwanted bias, we should understand where it comes from. “Bias” in the statistical sense is not a bad word. Bias is just a property of an estimate. A biased estimator is a function which tends to over- or under-estimate the value of a parameter. You can have a biased estimator that is still accurate. You can have a biased estimator that is inaccurate. You can have a biased estimator that is precise. And you can have one that is imprecise. Though you are seldom lucky enough to have an estimator that is perfectly unbiased and precise. A frequent compromise in machine learning is trading variance for bias. Completely eliminating bias often requires sacrificing precision, and high precision usually comes at the price of additional bias.²

	Low Variance (High Precision)	High Variance (Low Precision)
Low Bias	Most Accurate
High Bias		Least Accurate

Suppose that a company has a slight bias towards hiring Asians for technical roles. The following words in this paragraph are completely hypothetical, although there is evidence to suggest Asians are disproportionately well-compensated in some professions. Asians are smart. Asians work hard. Smart companies want to hire smart, hard working people to build smarter products, and will pay them commensurately. Whether or not Asian bias exists, the results of hiring for diversity has higher variance, on a number of key performance metrics. Since hiring more Asians, productivity and profits have soared. As a business owner, what would you do?³

<?xml version=”1.0” encoding=”UTF-8” standalone=”no”?>

Which would you prefer: low bias and high precision, or no bias and low(er) precision?

In many ways, bias is a valuable heuristic. It allows us to encode large amounts of categorical information and take action without shuffling mountains of data whenever we need to make a decision. Even when bias predicts falsely, the cost of a false positive may outweigh the cost of a false negative on average. Imagine a hiring bot rejects a West Samoan programmer, who is a genius-level engineer. While she would have made an excellent hire, if the metrics can’t properly estimate the value of a candidate, the safest strategy is to hire from a well-known population over one with unproven or uncertain quality.

Society thrives on social bias. It is the glue that keeps cultures and organizations intact, in the midst of sweeping globalization. Many have biases of athletic prowess, charismatic personality, or academic aptitude. We are comfortable with these biases, proud of our tolerance and sophistication in applying them. Some have biases of religious affiliation, ethnic background or physical similarity. We call these biases shameful, whilst secretly holding them ourselves. But now we can quantify bias, and the numbers do not paint a pretty picture. There will always be more and less accurate biases. But we must exercise caution. For bias, accurate or contrived, can have long-lasting effects on a population.

Human decision makers are prone to hundreds of cognitive biases. Whether relevant, or remotely accurate in any objective measure, bias can have negative and positive effects on a human population. The trouble with bias in prediction, is that any success is deceptive. Through luck or skill⁴, whenever someone is slightly successful at prediction, they invariably want to exploit their newfound predictive abilities. And exploitation is a whole new ballgame.

For most of ML, the training data is a given, often presumed to be representative of the data against which the prediction model will be deployed, but not much else. With a few notable exceptions, ML abstracts away from the data generating mechanism, and hence sees the data as raw material from which predictions are to be extracted. Indeed, machine learning generally lacks the vocabulary to capture the distinction between observational data and randomized data…

Most of the prediction literature assumes that predictions are made by a passive observer with no influence in the phenomenon. On the other hand, most prediction systems are used to make decisions about how to intervene in a phenomenon. Often, the assumption of non-influence is quite reasonable — say if we predict whether or not it will rain in order to determine if we should carry an umbrella. In this case, whether or not we decide to carry an umbrella clearly doesn’t affect the weather. But at other times, matters are less clear…

―Omkar Muralidharan, et al., Causality in machine learning (2017)

Exploitation requires a deep understanding of cause and effect, or deep reserves of luck. This subject is challenging for even the scientifically trained, and one of the toughest problems in AI. There is an outstanding body of work which explores the role of causality in machine learning. In short, the same predictive machinery that landed humans on the moon is surprisingly easy to derail. The same biases that impair our reasoning, make it easy to contaminate for example, AI models trained by humans, on human data, for other humans to interpret.

Long after machines are calling the shots, our heuristic biases will continue to deflect their ballistic trajectories. Although we may never eliminate human bias, we can ensure machines are less prone to statistical ones like selection and verification bias. Those training AI in a new domain should be asking themselves three important questions:

Are we really measuring what we want to measure? (ie. Test validity)
- Many advertisers try to maximize clicks. This is a loosing battle.
- Objectives may change over the production lifetime of a model.
- A poorly chosen objective can have unintended consequences.
Is the training data accurate and free from hidden bias? (ie. Internal validity)
- People constantly forget (or conveniently overlook) confounds.
- If the data generator is biased, the model will encode its bias.
- The method of sampling may have hidden biases.
Does the training data generalize well in practice? (ie. External validity)
- Maybe the training data has grown stale over time.
- Maybe the model is missing data on some key demographic.
- Maybe the true population is not the population we bargained for.

These questions are only clues in the search for unwanted bias. By eliminating sources of statistical bias, we can guarantee that human biases are apparent for what they are, not hiding as statistical errors in disguise. But we must not complacently check lists and perform t-tests. AI promises great predictive power, but with great power comes great responsibility. And responsible machine learning requires us to analyze human incentives, vigorously challenge our assumptions and constantly evaluate the validity of trained models.

Bias is an inextricable part of the world - every die is slightly loaded, every deck slightly stacked in one way or another. Machine learning just formalizes bias, blessing it under the auspices of validity. Bias can be found everywhere from the smallest neuron in a neural network, to the largest ensemble in a prize-winning Kaggle submission. Neural networks are essentially computing the sum of various biases - as the network sees more data, it updates the biases. Over time, some grow stronger and some grow weaker. Like their biological cousins, artificial neurons are coincidence detectors - they cannot distinguish association from causation. Intelligence is hard-wired to attribute association to causation, causing all manner of false rituals and superstitious behavior.

Fifty thousand years ago there were these three guys spread out across the plain and they each heard something rustling in the grass. The first one thought it was a tiger, and he ran like hell, and it was a tiger but the guy got away. The second one thought the rustling was a tiger and he ran like hell, but it was only the wind and his friends all laughed at him for being such a chickenshit. But the third guy thought it was only the wind, so he shrugged it off and the tiger had him for dinner. And the same thing happened a million times across ten thousand generations - and after a while everyone was seeing tigers in the grass even when there weren’t any tigers, because even chickenshits have more kids than corpses do. And from those humble beginnings we learn to see faces in the clouds and portents in the stars, to see agency in randomness, because natural selection favours the paranoid. Even here in the 21st century we can make people more honest just by scribbling a pair of eyes on the wall with a Sharpie. Even now we are wired to believe that unseen things are watching us.

―Peter Watts, Echopraxia (2015)

There will always be more and less accurate biases, depending on what we are measuring, and what outcome we hope to predict. Even so, machine learning will only reflect the data we show it, not the possible confounds from an unseen parameter, nor the possibilities if the world were a different place. If we do not probe these hypotheses, we are doomed to flee tigers in the grass, build monuments to false causes, and leave greatness in poverty. How many Lincolns were shot, or Einsteins put to the sword in the name of some false bias?

There is a dangerous course ahead on the path to autonomy. If we maintain the bias that only those who produce capital deserve to receive it, today’s economic inequality will grow exponentially worse. In the same way that genetic or geographic origin once predetermined a human’s social mobility, economic fitness will determine one’s financial security. In such a future, only those who are fast enough, smart enough, or wealthy enough to accelerate the pace of automation will enjoy the prosperity and wellbeing it guarantees. This is a heavy price to pay for a small increase in convenience and safety.

This is not a matter of social welfare. This is a matter of prolonging our competitive advantage against automation. By applying the same biases from evolutionary biology to education and employment, we are vastly underutilizing human potential. We should seek every opportunity to provide creative agency and professional opportunity to people from all walks of life. We should prefer deaf applicants to work in noisy environments, and let elderly people take leadership positions. We should empower those with mental and physical disabilities to find roles where they can utilize their skills effectively. We should incentivize high-performers in low-paying jobs from teaching to healthcare, and give them tools to become more effective at school and in the workplace.

Work in Progress

AI sometimes gets a bad rap. We often hear about AI in very competitive terms: whenever bots are not stealing our jobs or out-maneuvering us at chess, they are badly malfunctioning or causing some intrusion of privacy. And while it is important to maintain a healthy skepticism of the dangerous role that AI poses to our society, I think we will look back at many early applications as lacking sufficient imagination.

The Independents, rooted in the farms and small towns of the West, were innovators, but of a conceptual kind, not the technical kind à la Alexander Bell…They intuited that the telephone’s paramount value was not as a better version of the telegraph or a more efficient means of commerce, but as the first social technology…

Typically, the rural telephone systems were giant party lines, allowing a whole community to chat with or listen to one another. Obviously there was no privacy, but there were benefits to communal telephony other than secure person-to-person communications. Farmers would use the telephone lines to carry their own musical performances…

And so, while the Bell Company may have invented the telephone, it clearly didn’t perceive the full spectrum of its uses. This is such a common affliction that we might name it “founder’s myopia”. Again and again in the development of technology, full appreciation of an invention’s potential importance falls to others…

—Tim Wu, The Master Switch: The Rise and Fall of Information Empires

Technologists typically want to settle old scores. Greek astronomers invented primitive computers to foretell celestial movement, but never dreamt their descendants would use them to sail the heavens. The developers of microwave technology didn’t intend radar to heat food. If you have a general algorithm which learns to maximize reward by repeatedly choosing from a set of actions, then clearly you have an algorithm which prints money.⁵ But more importantly, if you have such a technology, then you have a new form of intelligent life. Why teach it to print paper, when we could be teaching it to write books?

The currency of the future is data, and the computational resources to mine it for insights. To say that money will vanish in the post-scarcity economy would be hubris, but printing money is just a side-effect, a parlor-trick compared the possible applications which AI affords. You might as well print paperclips and trade them for timeshares on the cloud. Our goal should be improving the lives of human beings, and if money is a necessary means to do so, then let’s print money. Although I suspect printing money is like handing out fish. If our goal is truly improving the lives of human beings rather than manufacturing consumers, we will need to give people a stable livelihood and meaningful ways to pursue happiness in the age of automation.

The hope is that, in not too many years, human brains and computing machines will be coupled together very tightly, and that the resulting partnership will think as no human brain has ever thought and process data in a way not approached by the information-handling machines we know today.

—J. C. R. Licklider, Man-Computer Symbiosis (1960)

One of the major limitations of user interfaces is bandwidth - keyboards and screens can only exchange so much information with their users. But today’s computers have the ability to interact with their environment in brilliant new ways. From self-flying drones to virtual patients, and home appliances to smart assistants, machines are becoming increasingly perceptive, and increasingly conversant. Machines can see, hear, and understand natural language. They can recognize faces and speech, anticipate our intentions and assist with increasingly sophisticated tasks. We call these capabilities “artificial intelligence”. But a more apt name might be “augmented intelligence”.

There are thousands of exciting AI applications, from tracking the spread of infectious diseases to detecting fake news, from brain-computer interfaces for coma patients, to automated hypothesis testing for scientists. Applications that are transforming our relationship with technology and improving the lives of billions of people on earth. The problems we encounter in AI are the same problems we have been struggling with for the greater part of the 20th century. Education. Equal opportunity. Employment. The democratization of technology. Solving these problems will require a more comprehensive view of AI, one that transcends just classification, prediction or automation.

Progress begins when we stop using algorithms to just predict people’s habits, and start teaching them new ones. When we give people a fighting chance, by retraining those whose jobs are threatened by automation. When we identify where humans show promise and teaching them how to improve. When we use AI to prevent diseases rather than designing drugs to treat symptoms. Progress begins when we start fixing broken systems rather than exploiting their weakness. If you want to contribute to the survival of our species, get out of the prediction game, and get into the business of making progress.

Discuss this post on HN!

Footnotes

Arguably China’s most famous living sci-fi writer. ↩
Pedro Domingos, A Few Useful Things to Know about Machine Learning. ↩
I have no idea whether the prior assertions are true. But a careful reading will reveal no implied causal relationship between “bias”, “productivity” and “profit”. The point is, we can replace “bias”, “productivity” and “profit” with any variable X, Y and Z. You control X, and observe the effect on Y and Z. Suppose X is positively correlated with Y and Z, i.e. higher values of X correspond to frequently higher values of Y and Z. Lower values correspond with frequently lower Y and Z, with high variance for low values of X. If we want to maximize Z, what is our best strategy? ↩
Although usually the former. ↩
Many clever but unimaginative people contracted the same idea at once, so one’s chances of developing a predictive advantage, and their profits from exploiting it, are much slimmer. ↩

Tales of a Traveling Twentysomething

2016-12-27T00:00:00+00:00

In which I describe globetrotting on Cars2Go, death-defying feats of gallantry, learning how to deliver a baby, and other mildly interesting stories from a prodigal twentysomething technologist with first-world problems.

Earlier this year, I had the chance to go on sabbatical. So I went. Since then, I have spent time in many interesting places - some for work, some for conferences, some just for the hell of it. As the year is almost over, I thought I would share parts of my journey. Perhaps you may wish to visit some of these places. Some of them, you will not.

In January 2016, I visited the sister of a Brother who lives in Phoenix. She kindly showed me the city where she had recently moved from somewhere in Long Island. We ate breakfast at the Breakfast Club, and looked for a relative who drowned in the attack on Pearl Harbor. In the middle of their city they have planted a huge gun for the lost sailors of the Battleship Arizona.

In February, I flew to Atlanta, and was promptly greeted by a cheerful Hawaiian professional photographer, turned recreational Uber driver, who gave a white kid tips for how to survive on the streets of Atlanta over Italian pasta. I think he took pity on me sitting in the backseat of his Uber. Besides the bomb evacuation at DevNexus, it was a pretty nice trip.

From Google Earth, over the I-20 and I-85 interchange in downtown Atlanta.

Later that month, I boarded Adirondack #69 from New York to Montreal for ConFoo. After running for miles in the freezing sleet, wondering what I was doing with my life and nearly succumbing to hypothermia, I met some incredibly kind people, including a friend from Brasil, the organizers, a plugin developer, and lots of Canadians. Great conference.

Shortly after driving down to EclipseCon in Washington, DC, I met some friendly people from a top-secret Russian laboratory, Goldman Sachs, and a couple of lawyers for the CFTC (all completely unrelated). I also had the opportunity to visit the National Basilica, a gigantic statue of Albert Einstein and the National Museum of Natural History. Go there.

In April I had a layover in Hong Kong and decided to get Dim Sum. Instead, I wandered around and got caught in a thunderstorm. So I did what anyone would normally do and climbed the tallest mountain I could find. After nearly dying of electrocution, the stormclouds parted and I caught a glimpse of the sprawling cyberpunk nirvana that is Greater Hong Kong.

The 68th Hong Kong School Prize Concert for Sacred Music, at St. John's Cathedral.

The rain started back up again, so I ducked into an empty church to dry off. Suddenly, there a great commotion - hundreds of children, dripping and tripping all over the place. They came in busses and bunches, shuffling onto the benches. Thunder and pouring rain. Umbrellas and coats. Candles everywhere. Then, they began to sing. I never heard anything so beautiful.

Landing in Bengaluru, I was greeted by the sound of pandemonium. Bengaluru is the tech capital of India, and I was here to work. After a long series of business meetings, chasing down deliveries, hailing three-wheeled contraptions, and drinking my weight in water every day, I felt cautiously prepared for the ensuing conference festivities.

Vada pav, from Sagar Resturant in Bengaluru.

Our tiny booth was quickly overwhelmed by a multitude of curious people. And from dawn to dusk, we shook hands, we gave away shirts, we held quizzes. Following an amazing culinary extravaganza and near-death experience with food-poisoning on M.G. Road, I am convinced Indian food, like all delicious things, is best consumed in moderation.

Two of my colleagues took it upon themselves to personally drive me around Bengaluru, broker steaming cups of tea from passing chaiwallas, order samosa, vada pav, and chats from busy street vendors, visit parliamentary buildings, gold-laden temples, purchase a saree, navigate massive thoroughfares and narrow alleyways, negotiate access to gated compounds.

Two things stand out in my mind. Indian enthusiasm is contagious. As a North American, my people are cool, reserved. We work hard, and complain a lot. Indians are hard working, and will happily talk to you about anything under the sun. I guess everyone is the same, everyone is different. But I am deeply grateful for their kindness.

Street vendors on the outskirts of Dharavi, one of the largest slums in the world.

After a brief stint in Mumbai, I flew back to New York with a new set of problems. So I decided to ignore them and go for a prolonged bike trip through the Pacific Northwest. Turns out that was just a different set of problems. I took the Amtrak Cascades from Seattle to Vancouver, rented a bike there, packed my backpack, and headed for Alaska.

After several near-death collisions, I gave up and settled for Whistler. If you sit at a desk for twelve hours a day, don’t try to peddle a bike 80 miles uphill the next one. Thank me later. But Whistler has fresh air, LTE, and skiing in May, so I was happy. I watched the seaplanes take off over Green Lake and wondered what I was doing with my life. Such are the problems we have in the first world.

Seaplanes take off from Whistler, carrying passengers and cargo to Vancouver.

After taking stock of the situation, I decided to pursue the full-time occupation of chasing a girl halfway around the planet. That was a pretty good idea in hindsight. We met in Vancouver, in Austin, in New York, in Boston and Portland and most places in between. You name it and we were probably there for some event or meetup this May or June.

It was my first time visiting Portland. We went there for Open Source Bridge, spent some time exploring Washington Park, and saw a spectacular Japanese Garden, recently restored and beautifully manicured, with a traveling bonsai exhibit. They have a Gilded Era mansion on the hill which is well-worth the visit, if just for the view. We also saw an extraordinary production of A Streetcar Named Desire at Portland Center Stage.

Planes. Trains. Automobiles. Every week was somewhere new. The daily milieu of new faces and scenes provided a welcome diversion to the reality of my situation. I chatted with Iranian barbers in Vancouver, brunched with captive bears on Grouse Mountain, had good ramen on Robson, explored castaway shipwrecks on False Creek, sat on the largest chair in San Fransisco, tested the longest slide in New York.

The Cheng-Shi. Her captain was "A sailor of the mind". Sadly, not a sailor of the sea.

How does one afford to travel so much? The answer is, as frequent flyers will explain in hushed voices, with lots of credit cards. By the interest earned from delinquent debtors and with a little help from Uncle Sam, US banks will simply hand you fake currency, if you pay your credit card bills in a timely fashion, that may be used at various airlines and hotel chains.

In June, I boarded UA 149 to São Paulo International Airport with a couple hundred other Brazilians. That plane never arrived in São Paulo International Airport. Nor did any of the subsequent flights we were on. Instead, they abandoned us behind customs in Rio Galeão without food or water for eight hours. To my amazement, United Airlines did not seem to see any problem with all this. The Brazilians just shrugged and laughed.

In São Paulo I had some amazing milkshakes, ate coxinhas and churrasco. I kept watch for police as my friend flew his drone around Ibirapuera Park to much public dismay, reassured fleeing bystanders, and in general did a whole lot of chasing other people around São Paulo. I had excellent ramen in Liberdade. Somehow managed to give a coherent talk about machine learning. Or maybe not. English is not their first language.

Oh My Milkshakes has the best milkshake I have ever tasted. It's really, really good.

In July, I fired my dentist. In hindsight, that was probably not a good idea. He was getting on my nerves with the appointment reminders, but he was a consummate professional, through and through. Dentists are good people, and even if they are a little fanatical about oral hygiene, they have your best interests at heart. You should see one regularly.

Later that month, I landed in Los Angeles. This was a big mistake. After breaking down in an antique automobile on I-15, I nearly perished from dehydration in the great suburban wasteland that lies east of LA, running to make an appointment with my father’s third ex-wife. Do not try to drive a car twice your age through the desert in summertime.

In August, after surviving another bout of food poisoning, I flew from LA to Bergen, Norway, by way of Dallas and London. The plan was to meet my girlfriend in Hamburg ten days later. Plenty of time to take the train from Bergen to Oslo to Stockholm to Malmö to København to Hamburg. Or so I thought.

I spent a day dodging lorries in a dodgy part of London on Santander Cycles, and eating mediocre ramen in Piccadilly Circus. After making my Heathrow connection with seconds to spare, I arrived in Bergen, still breathless, with my newly printed Eurail pass proudly in hand to show the customs officer. He sighed, and stamped my passport.

The Bergen Line connects Bergen and Oslo, carrying millions of people each year.

The journey from Bergen to Oslo was completely otherworldly, filled with gaping fjords, scenic vistas, glacial streams, and mountain passes. I sat next to a biologist studying seasonal migration patterns and teaching a summer seminar in Voss. I watched Norwegian villages and stations glide past my window. Norway is a strikingly gorgeous country, and I highly recommend the Bergen Railway if your travels should take you anywhere in the vicinity.

Stayed with a rock band living in Oslo. They kindly loaned me their bicycle, which I put to extensive use exploring the neighboring countryside. There, I wandered into a surreal village that looked like it was from the 17th century. The townsfolk, dressed in period clothing, made pointed gestures at my bicycle while speaking Norwegian. This was confusing, as I do not speak Norwegian and seemed to be only person from the 21st century that morning. I discovered it was a historical reenactment, with live actors and livestock. Nearby, is the Royal farm of Norway, stocked with heavily armed soldiers in open pastures. I took the ferry back to Oslo and ate some delicious Punjabi food for dinner.

I arrived in Stockholm late. The host was asleep, or forgot about my arrival. After accidentally waking up her neighbor and probably the whole apartment building, we arranged to meet a nice old lady, who apologized to everyone in Swedish, made some tea and introduced me to her city. Stockholm is a big park with a city attached. You can bike alongside people horsebacking through the woods. Have lunch in the middle of green fields. Toured an old warship dredged up from the bottom of the sea.

King of Sweden orders the largest warship ever built. Three masts. 64 guns. Topheavy, narrow-looking abomination. Two years to build the damn thing. Launches to much public fanfare. No ballast. Light breeze starts to blow in the harbor, and she sinks, killing thirty sailors. King starts an investigation. Everyone points up the chain of command, back to the King who rubber stamped the whole thing. Investigation quickly ends. Remind you of any software projects?

Kaknästornet is a tower that broadcasts television and radio signals all over Sweden.

Malmö is an little city on the coast of southern Sweden. The area where I stayed had long rows of block-style housing with WWII-era bomb shelters underneath. They have a tall building called the Turning Torso, visible from any point in the city and which my host pointed out enthusiastically. Disappointingly, it did not turn. Very windy and chilly here in August. I rode to Øresund Bridge and returned for hot chocolate, then boarded the commuter train to København.

In København, I stayed with the marketing director for the ESS in Sweden (apparently, big science projects need marketing people to help secure grants and so forth). He gave me directions to a pretty good ramen place nearby, and told me to visit a high-energy particle accelerator in Hamburg called DESY.

There exists a train from København to Hamburg which holds the distinction (unbeknownst to me at the time) of being a boat train. So you can imagine my excitement to be suddenly sitting on such a train, aboard a boat. They even gave us the opportunity to disembark from said train onto said boat, which I availed to great satisfaction.

My girlfriend met me in Hamburg. Much to her chagrin, my primary reason for visiting there was to see the Miniatur Wunderland. To her further chagrin, the second reason was now DESY. This time, I went alone. I slipped into a small tour, lead by someone who looked important. Later I discovered the tour was in German, but by then it was too late to turn back. I nodded thoughtfully and continued to do so for the remaining three hours as we walked from one secure facility to the next. Crisis averted.

Beneath a quiet suburb of Hamburg, DESY accelerates particles to 0.9999999C.

We then proceeded to visit a number of central European cities. Bonn for a week. Berlin for a few days. Potsdam for a day. Zürich for a day. Bern for a day. Lausanne for two. Zermatt and St. Moritz for a day. Salzburg for a week. Two days in Frankfurt. It was a whirlwind of trains, running to catch trains, finding our train was delayed, scrambling taxis to catch another train, etc. This required weeks of planning, but happily we made all of our scheduled connections, through careful chronological concentration.

Bonn is a quaint city with a surprising amount of trivia. Once home to Beethoven and former capital of Germany, it now houses the T-Mobile headquarters, Haribo, a large solar conglomerate, and has the best scrambled eggs I have personally tasted. Met a geology student who wanted to visit America. I encouraged him to do so. Later, I tried Club-Mate for the first time, and went to some good talks at FrOSCon.

From what I gathered, Berlin is a very culturally diverse and hip place to be. Personally, I was more interested in the historical bits, but even if you’re not there’s a lot to take in. There is Shanghainese food, neat European bistros, foreign exchange students hopping gleefully with stories of their latest party, vast swaths of beautifully manicured parks, loads of history. You could spend weeks here. Unfortunately, I only had a few days.

It was in Berlin where I began to appreciate the cultural differences between Germany and America. In the US we are outwardly warm, sometimes overtly so. Germans are outwardly cold, but inwardly warm and caring. These differences may cause some misunderstanding. You may think a German is being curt, or rude. Do not fret. Germans are very kind, once you get to know one.

There is a monument in the middle of Berlin, which you should visit, called Neue Wache. On a boulevard full of pompous looking buildings, it looks like any other. But if you take a moment to step inside, it will catch you off guard. Behind iron bars and beneath a hole in the roof sits a mother cradling her dead son. There are thousands of war memorials with huge guns. Go here instead.

The Berlin Victory Column, as seen from outer space, looks out over the Tiergarten.

Potsdam is nice to visit if you like palaces and gardens. I took the S-Bahn to Potsdam, rented a bike there, and rode for hours through the park. Later, I returned to Berlin and visited the Olympiastadion, a very impressive piece of architecture, accessible from the same line. But even if you’re not into stuffy old castles, you should visit Potsdam, because it is a very peaceful place indeed.

After a brief episode of frantically running around Berlin, chasing down a taxi, swerving to avoid oncoming traffic, and scrambling around the station to find our platform, we leapt aboard the City Night Line to Zürich, luggage and all, as they closed their doors and whistled to beat the band. The conductor tapped on his watch crossly. It was all very Wes Anderson-esque, except far less fun.

We awoke to the Zürich sunshine in August, and quickly realized the sun can be very hot under prolonged exposure. Storing our luggage in the train station, we got Muesli for breakfast, walked around the river baths and had fish at Nordsee, a German fast food chain. You probably wouldn’t associate fresh fish with fast food, but it is surprisingly palatable. Then it was time to go to Bern.

Bern has some very good restaurants which I highly recommend visiting if you have the time and inclination. We toured the Einstein House, strolled through the Rose Garden, peered into a bear bit, dipped our feet into the icy blue river, and did a good bit of touristy sounding things. Switzerland is a marvelous place if you have never been there before.

By the bank of the Aare river. It looks calm, but swimming is actually very dangerous.

Geneva is the home of CERN, which is mainly why we went there. You need to book a guided tour two weeks in advance, which gets reserved within a few minutes of new availability each day, so be prepared to mash F5 15 days before your arrival. The group self-selects for scientifically literate people, and our guide knew particle physics, so the tour is technically very educational.

From Geneva, we took a train to Visp, and up to Zermatt via a special railway that clings to the mountain like a snake in a tree, between cliffside dwellings perched on dubious ledges. We avoided making any sudden movements, lest they decide to take up flight. Cancelled a similar funicular excursion just a few days earlier, due to a large earthquake that had recently struck central Italy.

The town of Zermatt is only accessible by train, which is a good thing, because we had no other way of getting up there. However this means it is inaccessible when there are slides or avalanches. Fortunately, there were no slides or avalanches, and we arrived without incident. Stayed at Best Western (a great place in Switzerland) and watched the sun set over the Matterhorn.

The Matterhorn in the summer afternoon. It is one of the deadliest peaks in the world.

The next morning, we left bright and early on the Glacier Express, the slowest high speed train in the world. Scenic waypoints are narrated in dulcet sing-song, like a person reading a children’s fairy tale, in English, German, Italian, French and possibly Romansh. We passed through steep mountain passes, sweeping valleys, and over trestles spanning deep gorges.

Arriving in St. Moritz later that afternoon, we hauled our bags a kilometer uphill into a hotel above a pizza shop then walked around the lake. We watched people maneuver small sailboats, and tuxedo-clad chaps jump horses over fences in the late summer sunshine. It was all spectacularly European. St. Moritz is very extravagant, dripping with fur, fast cars and grand old hotels.

Then we took a train to Salzburg, by way of Zürich. Salzburg was really interesting, and much more salt-of-the-earth. I grew up watching The Sound of Music every Christmas, so there was an immediate connection. To my dismay, Austrians do not typically break out in A Capella musical overtures, and frowned on my own attempts to engage them in doing so, vis-à-vis My Favorite Things.

Salzburg is a town on a river, as many European towns are prone to be. More importantly, it has an abundance of salt. Salt was once the currency of the day, hence the English, salary. This means that Salzburg was a very wealthy place when salt was the legal tender, and accumulated a great deal of castles and aristocratic types through the taxation and export of salt along its river.

One of these was the Archbishop of Salzburg, Markus Sittikus, who by all accounts was not a very pleasant person, but hey, times were different. He built a large estate, known as Hellbrunn Palace, containing many cruel hydraulic innovations. To this day, guides use his collection to torment passing tourists.

Paragliders taking off from the top of Zwölferhorn Mountain in St. Gilgen, Austria.

After spending several pleasant days exploring the lakes and mountains of Salzburg, riding gondolas, watching people fling themselves off mountaintops on kites, catching Pokémon, concertgoing, and eating far too much Mozartkugel, we waved goodbye to Austria and made great haste for Frankfurt.

I commandeered a Car2Go on the Frankfurt autobahn, a road with no need for speed limits, beside those required by the laws of physics. This was fortunate, as we were short on time. Unfortunately, the laws of physics have bounded Car2Go’s maximum horizontal velocity to the approximate speed of a golf cart, so it ended up taking a while. But once you finally park the thing, you’re right beside the airport check-in.

Then I spent a week in San Jose at the beginning of September, and saw some great talks at Cassandra Summit. Cassandra is a pretty cool technology if you have truly enormous amounts of data to record. Maybe if this blog post is ever finished I will have a legitimate need for it.

Rented a car on Friday and drove south for the weekend. If you have a special weekend enterprise and are looking for automotive transportation, take the Enterprise Weekend Special. They’ll pick you up anywhere but the airport and you’ll pay more for gas than the car itself.

Monterey is the ancestral home of John Steinbeck, and the setting for many of his novels. John is a great writer. But he doesn’t do it justice. The coastline of northern California is indescribably beautiful. You can run on warm beaches crashing with surf in the Caramel Highlands and walk through quiet stands of Coast Redwood in Big Sur. Go there, you will not regret it.

Chrysaora quinquecirrha, or Sea Nettle, are swimming inside the Monterey Aquarium.

Flew to Boston and back the next weekend for JavaOne. Taught some kids how to break ciphers with an friend from India, who lives in Utah. Kids are wicked smart. Went for a nice bike ride to Sausalito with some speakers from Switzerland, a contractor for the US Patent Office, a developer from Ireland. Took the ferry back. Had a fair bowl of ramen.

Each year, Oracle swoops into San Fransisco like Christmas in September and paints everything red and white. Flies racing yachts in by crane, blocks off city streets. The last couple years, they built a carnival complete with ferris wheel and free food, fireworks, Elton John, then bussed ten thousand people out to Treasure Island for an all-night bacchanalian celebration. This year, they were conservative and filled a baseball stadium instead.

From SF, I flew to LA and back the following week, for SVCC. It rained, so I rented a car and drove into the northern Diablo Mountains. I drove into the clouds, and arrived at Lick Observatory on Mt. Hamilton to a haunting silence. The air was still and there was no sign of humanity. I found a custodian, who showed me the telescopes. We had to be very quiet, because most astronomers are asleep during the day.

Supposedly the Mt. Hamilton Road has 365 turns, but this is a coastline problem.

Returning from the clouds, I had lunch at Sweet Tomatoes, which is my favorite restaurant in California. They have unlimited soup and salad, but not the usual buffet fare. It’s shipped in fresh from the San Joaquin, on produce trucks every day. The supply chain is excellent.

Then I lived in Brasil for a month. I met my girlfriend in São Paulo, where we stayed for some time. Her mother cast spells in the kitchen, conjuring comida caseira: each morning there was coffee, tea, and eggs. Each afternoon there was pão carteira with mortadela and cheese. Each night there was a feast of file de frango, batatas fritas, feijao com arroz, pavé.

We flew to Florianopolis for Python Brasil, where I visited a local faculdade to talk about unsupervised learning. An audience member offered me a drink of hot mate, which was high quality stuff. Floripa is a very warm and friendly place. You can walk for miles on white sand beaches, eat green pasta and perfect pizza. It was here where I discovered a most remarkable drink, limonada suiça.

Brazilians have a long tradition of making juice. Their fruit crushing and blending technology has progressed far beyond the outside world. And they have refined the science of mixing fruit beverages, with profound combinatorial breakthroughs. I will share with you one of their secret recipes. Take fresh lemonade. Carbonated water. Add ice, condensed milk and sugar. Mix to a froth. Serve immediately, with a bowl of hot spaghetti and warm bread.

From Floripa, we headed to the city of Foz do Iguaçu for Latinoware. Foz do Iguaçu is a fascinating place. Located at the confluence of three rivers, it has an important hydroelectric dam, an enormous Buddha, and a vibrant Islamic community. Oh, and the largest waterfall in the world.

The Iguaçu Falls spill 1,756 m³/s, which in Imperial, is a gargantuan amount of water.

The Cataratas do Iguaçu, or Iguazu Falls, is an enormous chasm in the Iguaçu river, like a giant’s sunken footprint in the jungle. From the air, it looks like something from Avatar. From the ground, it’s breathtaking. There’s two sides, both heavily touristed. If you have to choose between them, go to the Argentinian side. Our guide, who spoke fluent Spanish, Portuguese and English, was a tour de force. She could lead armies.

Visited the largest mosque in Foz and tried a pastry at an Islamic bakery. Tried to eat all the pastries at the bakery. I debated crossing the bridge to Ciudad del Este, but my girlfriend was convinced the Paraguayans would kidnap and sell me into slavery, which is probably not unlikely, so I decided that I was quite happy eating Islamic pastries in Brasil. On to Itajubá.

Itajubá is a small city in Minas Gerais, about four hours from São Paulo by bus. Its primary attraction is the local university, one of the best in Brasil for engineering, or so I am told. I was introduced by an energetic young professor, gave a short speech to the students, then we all went out and had salgadinhos. The students were all lovely and treated me like an esteemed scholar. It was a little embarrassing, but very thoughtful and well organized.

Itajuba is home to UNIFEI, the second largest university in the state of Minas Gerais.

Later that day, I learned how to deliver a baby. At the invitation of my new friend the professor, we visited his laboratory. Inside, was a woman in labor. As you can imagine, this was all very difficult for me to process. But there was work to be done, so we rolled up our sleeves and prepped her for C-section.

I soon learned the lab was studying the usability of medical equipment for neonatal ICUs. To do so, they needed a test subject. This subject was me. Or rather, the version of me with a degree in gynecology. The patient was surprisingly realistic, with fake bodily fluids, breathing, vitals. It was uncanny.

From São Paulo, we just barely made our Heathrow connection to Dublin. Transferring from an international flight to anywhere in the British Isles requires crossing no less than three different security checkpoints. By some accident of chance, I have never missed a flight in my life, but this was the closest I have ever been. The stewardess scolded us sternly.

My family picked us up in Dublin, where we all had a nice supper together. The next day, I brought the folks to Python Ireland, which is an excellent conference that I highly suggest you attend. Thanks to the organizers, I was able to wheel Grannie to the front row, where she learned about machine learning with TensorFlow. It was great fun, and a very gratifying experience.

Back row: Mark (Uncle), Margaret (Mother), Breandan (yours truly). Front row: Grannie.

From Dublin, we flew to Brussels for Devoxx Belgium early Monday morning. After landing in the pouring rain, taking a train in the wrong direction, taking a trolley in the wrong direction, and running around like a madman, I arrived for my talk still dripping wet. It went okay, I probably should have spent less time delivering babies and prepared more. However I did learn a lot from other speakers, saw some old friends and made new ones there.

Spent a weekend in Amsterdam, where I got a haircut and had a great steaming cup of hot-chocolate, made with real chocolate chips. Then we took a train back to Brussels (Charleroi, to my frustration) and flew to Vilnius for BuildStuff Lituania. There are a few small conferences in the world that are extremely high quality. BuildStuff is one of them, and apparently the organizers donate their proceeds to charity. Go there, if you have the chance.

Why else would you want to go to Vilnius? For one, they have great restaurants where you can eat well without breaking the bank. They have a pretty nice city, with good transportation. And Lithuanians are a very friendly people. It’s not Switzerland, but the Europa section is very modern and has good restaurants. My favorite one is in the Old Town, a place called Gusto blyninė. If it looks ridiculous, you’ll know you’re at the right place.

Gediminas Tower has overlooked Vilnius for over 600 years. It is in need of repair.

From Vilnius I flew back to Dublin to take Grannie home, who in the meantime had acquired 200 kilograms of cast iron plates for our woodstove. How did Grannie do this? I haven’t the faintest clue. Anyhow, I got the bright idea of putting them all in my carry-on luggage. Surprisingly, this worked. Now I just needed to get them aboard the airplane, with Grannie and her wheelchair.

I decided it was a problem for another day, so we drove up to Belfast to see the Titanic Museum. Instead we ended up at the Titanic Exhibition Center, which was hosting a snooker championship, so we watched tuxedo-clad gentlemen sink snooker balls on an oversize pool table. Then we went to the Titanic Museum across the street, which was thoroughly engrossing. After wheeling Grannie halfway around Belfast, we returned to Dublin for our flight.

We arrived at the airport in the early morning. Somehow made it through security and all the way to US pre-clearance with our 200 kilograms of cast iron. The Irish authorities were furious. Grannie convinced them not to arrest us all then and there (thank you, Grannie!), and even managed to get our stove parts back, but we had to check everything with special permission from the airline, US Customs, pilots, and a bunch of stern looking people at the gate.

A wood stove should never be placed in your overhead bin, whether in whole or part.

I made it back to Boston, dropped of Grannie, rebuilt our woodstove, got the fire going, and headed to Vancouver on a series of red-eye flights through Newark and Minneapolis. As you might expect, this was one 48-hour long nightmare. In Vancouver, I took the SkyTrain downtown, planned to use Car2Go, but due to earlier shenanigans in Frankfurt, Car2Go believed I was in Germany and needed to update my license. I realized this only after purchasing a boatload of groceries. It would take 24 hours to reactivate.

With groceries and luggage in hand, I ran around downtown Vancouver trying to hail a taxi. Little did I know, taxis there do not stop for grocery-toting, flailing pedestrians, nor do they accept credit cards. And I do not carry cash, or use SIM cards, because I am an idiot. Several hours later, I found a bank to withdraw cash, and boarded a bus to North Vancouver, dropped off the groceries, called Car2Go, and collapsed into a deep slumber.

When I awoke it was nearly midnight. Luckily, Car2Go had resolved the license issue while I was asleep, allowing me to drive their golf carts again. So I ran half a kilometer to the nearest Car2Go and floored it all the way to the airport to meet my girlfriend, who was just then arriving from Chernobyl. On past occasions I have put Cars2Go to very effective use, however this was simply not the case in Canada.

For reasons I cannot begin to explain, we were locked inside a malfunctioning Car2Go at a gas station. We were stuck in park at a busy intersection. We were fined for alleged parking violations. We had to call for help dozens of times, and spent hours troubleshooting their broken fleet of golf carts. Car-sharing is clearly a win in some places, but does not work well in Canada. End rant.

I went to Vancouver for ConFoo, since I enjoyed the one in February so much. It was their first time in Vancouver, and a lovely conference, filled with lovely people. Afterwards, they bussed a few of us up to Whistler and we all went skiing for two days. Met some swell folks between runs, and we talked about science, religion and the post-truth society.

From Vancouver, we flew to Toronto and spent a few days visiting friends there. I had amazing borscht at Gayleys Cafe. I read a great book, called the The Three Body Problem. Then we took a train to Ottawa. Spent a few days visiting more friends, nearly dying of frostbite. Took the train to Montreal. Then I came home for the holidays and spent Christmas day in a hospital because Grannie came down with pneumonia. She’s one tough cookie. We’re hoping for good news.

When all is said and done, I went to 18 countries, 64 cities, 16 conferences, and visited dozens of universities and meetups in 2016. If I had to do it all over again, I wouldn’t do everything the same way. I spent most of the time exhausted and jetlagged. But I also had some very happy memories, particularly those that did not involve nearly dying. And I got to spend time with some extraordinary people along the way. So I’ll mark it with a B+.

If you made it this far, I feel compelled to leave you with some sort of message. The best tales have lessons, answers, endings. But my answers are not the ones you are seeking, and even if they were, they wouldn’t do you much good anyhow. There is a great big world outdoors, and if you have the desire to see it, then I would strongly encourage you to do so (you will not regret it). But don’t try to bring 200kg of cast iron along with you.

This story was made possible by the patience of my family. If you enjoyed reading it, please consider giving a hug to your Grannie, or someone you love.

Democratizing Education in China

2016-04-13T00:00:00+00:00

In former president Hu Jintao’s address to the 17th Party Congress in 2007, he urges delegates to, “enhance our nation’s cultural soft power,”¹ citing China’s, “5,000 Years of Culture”, an immense source of national pride for billions of Chinese people around the world. In recent comments, current Party Secretary Wang Qishan appeals to this legacy stating, “The Party’s legitimacy lies in history, its popular support and the mandate of the people.”² But the strength of a great civilization is not in numbers, or how far its borders reach, but according to Ghandi, in how it treats the weakest members of society.

During President Xi Jinping’s 2015 visit to the United States, he called attention to this demographic, acknowledging, “over 70 million Chinese citizens living on guaranteed minimum income, and over 85 million people with disabilities.”³ The Chinese Dream, a recurring theme of Xi’s presidency, emphasizes economic prosperity and military strength, but despite these promises, services for the disabled are severely lacking throughout mainland China.⁴^,⁵ According to some statistics, 28% of disabled children lack access to basic education and over 40% of China’s disabled are illiterate.⁴

These figures are not merely provisional lapses of welfare in a developing economy, but signs of an endemic problem in mainland China. Not only are disabled citizens overlooked by government services, but many are actively discriminated by primary schools, barred from receiving higher education through mandatory physical examinations, and disqualified from certain vocations based on physical “defects”.⁴ Where special education is available, students are segregated from their peers and often forced to travel long distances for equal access.⁴ These are insurmountable obstacles for millions of rural citizens.

Even if these obstacles were lifted overnight, China faces steep social and economic barriers to equal rights. But there is real hope for technological innovation in paving the road to equal opportunity. With the emergence of new digital platforms like massive online open courses (MOOCs) and the widespread accessibility of assistive technologies for digital communication, we can disrupt the crippling economic disincentive that traps millions of disabled citizens in poverty and give them tools to become contributing and valued members of society. And with the growth of telepresence software and flexible work environments, as China transitions into a service economy, mobilizing these individuals in a technology-enabled workforce will become more practical than ever before.

The key to making rapid progress in disability services is by investing in technology, not politics. Today, we can give sight to the blind, subtitles to the deaf, voices to the language-impaired, and adaptive input systems to the physically handicapped, all with software that can run on a wristwatch. We can transform early childhood education through interactive speech recognition, and radically improve higher education through online learning initiatives. By investing in usability, democratizing access to a basic education, and driving down the barriers economic prosperity, we can help make the dream of a stronger, healthier China a real possibility for individuals from all walks of life.

“…必须提高国家文化软实力…” Full text of Hu Jintao’s report at 17th Party Congress. (2007). http://np.chineseembassy.org/eng/ChinaABC/t377776.htm ↩
“中国共产党的合法性源自于历史, 是人心向背决定的, 是人民的选择.” The Party and the World Dialogue. (2015). https://news.sina.com.cn/c/2015-09-10/232332296609.shtml ↩
“中国城乡有7000多万低保人口, 还有8500多万残疾人.” From a speech on China-U.S. relations. (2015). https://news.sina.com.cn/c/2015-09-10/232332296609.shtml ↩
“Barriers to Education for Persons with Disabilities in China.” Human Rights Watch. (2013). https://www.hrw.org/report/2013/07/15/long-they-let-us-stay-class/barriers-education-persons-disabilities-china ↩ ↩² ↩³ ↩⁴
“Paper commissioned for the EFA Global Monitoring Report 2006, Literacy for Life.” Ross. (2006). https://unesdoc.unesco.org/images/0014/001461/146108e.pdf ↩

Democratizing Learning Technology

2015-10-11T00:00:00+00:00

Growing up in rural New Hampshire, the world is a small place. Our nearest neighbors lived miles away, and you could drive for hours before seeing anything like a city. My grandparents moved out of New York in the 1960s, and raised me in a single-family home thirty years later. They would stoke the fire at night and tell me stories of places where buildings rose a hundred stories tall and trains ran underground. We would read newspapers by candlelight, and day by day, the world grew a bit larger.

It is a little unreal to think of how much the world has changed these last few years. Today, we have self-driving cars and information superhighways, smartphones and social networks, but in many ways we are further from our neighbors than ever before. As a computer science major, I understand the vital importance of technology in today’s connected society. As a student and software engineer traveling to developing countries like China, I have watched that same technology leave many behind. I know we can do better.

Growing up with a mentally disabled person helps you appreciate the little things. Like reading. We may take it for granted, but reading is a gift that opens doors into another world, shut to those who cannot read. Yet by some estimates, over a quarter of the world’s adult population is unable to read a book. This needs to change, and quickly. We have the technology to completely eradicate adult illiteracy for children born after 2010, even without access to schools or internet. And now more than ever, we need literate citizens of the world.

In the software industry, far too much talent is squandered on frivolous things like selling ads and sharing selfies. And far too much effort is spent exploiting those who do not understand technology, rather than educating the ones who need it most. In the industrial revolution, child labor grew to an all-time high, despite promises of giving young people more time to pursue an education. Likewise, today’s technology can be a tool to empower and enlighten, but is often used to persuade and preoccupy those who are most susceptible to its influence.

After college, I was lucky to find fulfilling work that allowed me to repay my student loans and provide for the family who raised me. But I am often reminded of those more gifted than I, who are unable to do so for circumstances beyond their control. Who must take unfulfilling jobs to feed their families. Who lack the necessary means to attend school. Who have no one to call a family. I know that if those people could have the same opportunities I have enjoyed – if they could meet the same people I have met, the world would be a much better place.

Our history is full of men and women who were at the right place and the right time. Who were given the right combination of resources and opportunity to effect lasting change. But far more, are those who were never given the opportunity to put their talents to use – held back because they could not read or marginalized by a society that could not spare the time to teach these children. It is said the true measure of a civilization is how it treats the least of its citizens. I believe today, that truth is more important than ever before.

Today, we are uniquely equipped with the technology to give millions of children a world-class education – technology that can hold conversations, teach foreign languages and do simultaneous translation, with devices that fit on a child’s wrist. Today, we are standing at a turning point in human history. In recent comments, Stephen Hawking, a physicist and distinguished beneficiary of assistive technology, predicts two possible futures. One, where society is elevated by technology - where knowledge and opportunity is abundant and freely given. But there is a second future, where only the privileged few - those endowed with the talents and resources to outrun the accelerating pace of technology can enjoy the prosperity and wellbeing it confers. We cannot allow this future to exist.

We have the technology to radically transform education in developing countries, where millions of children are learning English and foreign languages. We have the ability to teach languages in a completely different way, with machine learning and speech recognition. And we have the responsibility to use these tools to help cultivate the minds of young people around the world, regardless of their intellectual abilities. Providing a quality education to the underprivileged and disabled is one of the most pressing issues of our time, and one that I am committed to solving.

A Short History of Shorthand

2015-02-01T00:00:00+00:00

From early recorded history, stenography, or shorthand has been used to translate oral information onto paper, which was later transcribed into a legible medium for publication. This process was often a painstaking one, requiring specialized knowledge to read and translate. In the process of transcribing spoken language, conventions took root and new words emerged, giving rise to literature. Today, many of the words we have at our disposal are products of stenography - yet despite its historical importance, stenography is a forgotten art.

Night Mooring at Maple Bridge – Zhang Ji, A.D. 730-780
The moon is setting, a crow caws in the frosty night air
Beneath the river maples, fishing lanterns flicker and sway
And beyond these city walls, from the temple on Cold Mountain
The midnight bell is tolling, for me in restless slumber.

One of the early reasons for stenography was limited resources - primarily time, although tortoises¹ and teachers were not in steady supply. Early authors were “bandwidth limited”, forcing them to condense their vocabulary in order to efficiently refer to common phrases. Today we enjoy many of their typographical innovations, such as ampersands, acronyms, and more recently, hashtags. While these conventions may have once shared frugal origins, we are quite happy to maintain them out of familiarity - for our readers’ sake, but as we soon realized, increasingly for our own benefit as well.

It turns out – perhaps unsurprisingly – as we started using more abbreviations, they began to take up permanent quarters in our own vernacular. Whereas Laser² and Radar³ may have been deliberate interventions, many acronyms are more benign, eg. ASAP, A.D., B&B, ETA, FAQ, IQ, Q&A, RSVP. Some more contemporary slang can be easily deciphered based on its context and frequency, eg. AFAIK, FWIW, IMHO, IIRC. You may recall a moment of brief happiness when recognizing a new shorthand and then using it to communicate with another human being. This is not abnormal.

$\begin{align} \int_0^{10} 6x^4 dx &= \left.\frac{6}{5}x^5\right|_0^{10} \\ &=\frac{6}{5}(10)^5 \\ &=1.2\times10^5 \end{align}$

With the advent of terabyte HDDs and modern CPUs, one might imagine the structure of text alone would become less important - if shorthand were simply a means of abbreviation, then we should have little need for it today. But all along, the goal was never really conservation of ink and parchment, but the conservation of ideas. Far too many good ideas have been lost in a sea of poor notation⁴. Notation allows us to quickly employ declarative and procedural memories with the same ease which they come to mind. By standardizing their usage, we are able to communicate more effectively.

$\begin{align*} x \in V & \Rightarrow x \in \Lambda \\ M, N \in \Lambda & \Rightarrow (M N) \in \Lambda \\ M \in \Lambda,\, x \in V & \Rightarrow (\lambda x M) \in \Lambda \end{align*}$

Notation can also change how we think. Alonzo Church’s λ-calculus is one example of notation that changed not only just nomenclature, but usage as well. In the 1930s, Church developed a notation for describing computable functions called a lambda expression. Today we think of λ-expressions as a kind of shorthand for defining anonymous functions. More importantly, λ-calculus treats functions and data as the same and provides an alternative model of computation through substitution. While equivalent to the procedural model, it can lead to vastly different (and often far simpler) solutions to the same problem.

Translating a λ-expression in Java. Objects in editor are closure than they appear.

Today there are many writing systems which share a number of interesting similarities with traditional stenography. For example, Emmet is a shorthand for typing HTML. IntelliJ IDEA has a similar system of macros and keyboard shortcuts for automating repetitive programming tasks. Some television subtitles are now generated with a supervised speech recognizer⁵. And many handheld Character amnesia is a growing problem in Asia.devices have predictive keyboards, which use Markov chains to suggest the next letter or word in a sentence. All of these are modern examples of stenography, yet they are radically different in form and function.

This uncoupling between the creation and representation of text raises a number of intriguing questions. With the sudden ease of which we can create new abstractions and programming shorthands through macros, polymorphism and reflection, what is the appropriate level of notation required for a particular task? With the availability of code generation tools that write code, type systems for verification, and powerful IDEs to understand how it fits together, how do we differentiate between source and target? And who, or what, is the audience? These are not new questions⁶, but their significance today has only become more apparent.

Some editors like MPS treat text as an intermediate format, rather than the "source". This enables some degree of flexibility in the composition and presentation of code.

Let us consider a few examples. Conciseness is one desirable property in language design, although achieving it while maintaining precision is often quite challenging. Even in Java, one of the most carefully guarded⁷ programming languages today, new language features have created the potential for ambiguous code. Method references are a shorthand in Java 1.8 whose simplicity apparently outweighed the potential ambiguity.

public class Foo {
    private interface F<T, R> {
        R apply(T arg);
    }

    public String bar() {
        return "foo";
    }

    public String bar(String foo) {
        return foo;
    }

    public static String bar(Foo foo) {
        return "bar";
    }

    public static void main(String[] args) {
        F<String, String> f1 = new Foo()::bar;
        System.out.println(f1.apply("OK")); //Prints "OK"

        F<Foo, String> f2 = Foo::bar;
        //Compile error: reference to bar is ambiguous
    }
}

Does Foo::bar refer the instance method bar() or the static bar(Foo foo)?

In Kotlin, there are no static members, making this somewhat less of an issue. However unlike Java, which is capable of distinguishing method references based on the type signature, Kotlin does not support ambiguous method references (ie. function references) to overloaded functions. It should be noted that while it is not possible to resolve ambiguous method references in Java without altering the syntax, disambiguating overloaded functions in Kotlin ought to be relatively straightforward.

fun foo() = "bar"

fun foo(bar : String) = bar

fun main(args : Array<String>) {
    var f1: (String) -> String = {s -> foo(s)}
    println(f2.invoke("OK")) //Prints "OK"

    var f2: (String) -> String = ::foo
    //Compile error: Overload resolution ambiguity
}

Does ::foo refer to the niladic or monadic foo? The λ parameter gives us a clue.

We can see how removing certain elements of notation may occasionally lead to ambiguity. Likewise, by adding notation we can also gain additional expressiveness and precision. For example, annotations in Java 1.5⁸. Java has added a small number of language features and keywords since its inception, including assert and enum⁹. But perhaps one of the best examples of this can be found in a typesetting language called $\TeX$.

\overbrace {  (r_1 + r_2  + \cdots + r_w)^v =   }^{\text  {1}}
\underbrace{  \sum_{k_1 + k_2 + \cdots + k_w=v} }_{\text  {2}}
\overbrace {  v \choose k_1, k_2, \ldots, k_w   }^{\text  {3}}
\underbrace{  \prod_{1\le t\le w}r_{t}^{k_{t}}  }_{\text  {4}}

$$ \overbrace { (r_1 + r_2 + \cdots + r_w)^v = }^{\text {1}} \underbrace{ \sum_{k_1 + k_2 + \cdots + k_w=v} }_{\text {2}} \overbrace { v \choose k_1, k_2, \ldots, k_w }^{\text {3}} \underbrace{ \prod_{1\le t\le w}r_{t}^{k_{t}} }_{\text {4}} $$

$\TeX$ was conceived in the late 1970s by Donald Knuth¹⁰, prior to the era of IDEs and modern text editors. It consists of a set of commands that exactly specify the visual arrangement of text on a page, and has been used for this purpose in academic circles for nearly four decades. Despite its popularity, $\TeX$has remained largely unchanged over that time, owing in part to a flexible system of macros¹¹. Macros (short for macro instructions), are a substitution rule for replacing text, which in $\TeX$’s case, occurs during compilation.

\def\smz#1#2{  #1_1 + #1_2 + \cdots + #1_#2  }
\def\sqt#1#2{  #1_1,  #1_2,  \ldots,  #1_#2  }
\def\pdx#1#2{  \prod_{ 1 \le #1\le\ #2 }     }

\overbrace {         (  \smz{r}{w}  )^v =  }^{\text       {1}}
\underbrace{    \sum_{  \smz{k}{w}=v  }    }_{\text       {2}}
\overbrace {  v \choose \sqt{k}{w}         }^{\text       {3}}
\underbrace{  \pdx{t}{w} r_{t}^{ k_{t} }   }_{\text       {4}}

$$ \def\smz#1#2{ #1_1 + #1_2 + \cdots + #1_#2 } \def\sqt#1#2{ #1_1, #1_2, \ldots, #1_#2 } \def\pdx#1#2{ \prod_{ 1 \le #1\le\ #2 } } \overbrace { ( \smz{r}{w} )^v = }^{\text {1}} \underbrace{ \sum_{ \smz{k}{w}=v } }_{\text {2}} \overbrace { v \choose \sqt{k}{w} }^{\text {3}} \underbrace{ \pdx{t}{w} r_{t}^{ k_{t} } }_{\text {4}} $$

In automatic programming, macros are not an uncommon feature - the C preprocessor performs lexical substitution prior to compilation. IDEs and text editors offer increasingly sophisticated keyboard macros under various names (eg. templates, macros, autocorrection). And many build tools incorporate document generators and string expansions that substitute and rewrite portions of source code. In a way, this gives programmers the ability to take notation into their own hands. It is unclear whether this is a good idea.

The key, it seems, to make automatic programming work is readability. In a perfect world, we might never have to write a single line of code twice. But perhaps what we should really be working towards, is not necessarily the conservation of keystrokes, but rather the conservation of ideas through effective notation. For all the time we spend writing code, albeit plenty, is dwarfed by the amount of time others spend trying to understand it (and failing to do so, spin off their own version with the same reckless abandon we poured into the first).

Did you know? Frank Liang spent five years¹² studying a better hyphenation algorithm for $\TeX$¹³. Assuming Liang’s Algorithm has saved two minutes per year for each of a million readers (in academic journals, textbooks and CVs), Liang has singlehandedly saved over a century.

While the process of rapidly writing code is well-attended, the limiting factor in developer productivity is not the bandwidth of our fingertips on keys, but rather the attention span it takes to process their collective output. This is evident on a large scale in commercial code reviews, but also on an individual level, in the time it takes to effectively learn a new language or framework. It is not unreasonable to imagine how an upfront investment in readability could result in a hundredfold savings in time spread over a large userbase.

With this in mind, we should think of programming as an exercise in good notation. Notation matters - but not because it is easier for us to write. It is often convenient to choose a notation that our team is most familiar with, or with the most votes votes on HN. But if we succeed, we will not be the only ones maintaining it in ten years. So experiment. And when you have a good idea, spare no effort to make it plain.

Footnotes

Oracle bone script. (2015, January 17). Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Oracle_bone_script. ↩
Gould, R. Gordon (1959). “The LASER, Light Amplification by Stimulated Emission of Radiation”. In Franken, P.A. and Sands, R.H. (Eds.). The Ann Arbor Conference on Optical Pumping, the University of Michigan, 15 June through 18 June 1959. p. 128. ↩
Radar. (2015, January 30). Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Radar. ↩
Although even average ones have stayed afloat with good notation. ↩
Imai, Toru. (2012). “Speech Recognition for Real-time Closed Captioning,” Broadcast Technology No. 48, pp.1-9. ↩
Wesch, M. (2007, January 31). Web 2.0. The machine is us/ing us [Video file]. Retrieved from https://www.youtube.com/watch?v=NLlGopyXT_g. ↩
Goetz, B. (2007, January 31). Stewardship: The sobering parts [Video file]. Retrieved from https://www.youtube.com/watch?v=2y5Pv4yN0b0. ↩
Java annotation (History). (2015, January 20). Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/wiki/Java_annotation#History. ↩
Java language keywords. Oracle Corporation. https://docs.oracle.com/javase/tutorial/java/nutsandbolts/_keywords.html ↩
Author and computer scientist at Stanford University, more famously known for writing The Art of Computer Programming. ↩
Knuth, D. E. (1986). Definitions (also called Macros). The TeXbook. https://web.mit.edu/jgross/www/LaTeX/texbook.pdf. ↩
Liang, F. M. (2010). Interview. http://tug.org/interviews/liang.pdf. ↩
Liang, F. M. (1983). Word Hy-phen-a-tion by Com-put-er (Doctoral dissertation, Stanford University). https://www.tug.org/docs/liang/liang-thesis.pdf ↩

IntelliJ IDEA on Linux

2014-08-18T00:00:00+00:00

The following script installs a JetBrains IDE on any Debian based Linux distro. To download it, simply run wget z.ndan.co/jetbrains-install.sh:

#!/bin/sh

# We need root to install
if [ $(id -u) != "0" ]; then
    echo "Elevating to root..."
    exec sudo "$0" "$@"
fi

# Prompt for edition
while true; do
   read -p "Please select from one of the following choices:
   [1] IntelliJ IDEA Community Edition
   [2] IntelliJ IDEA Ultimate Edition
   [3] PyCharm Community Edition
   [4] PyCharm Professional Edition
   [5] CLion
   [6] WebStorm
   [7] RubyMine
   [8] PhpStorm
   [9] DataGrip
  > " CODE
   case "${CODE}" in
       1 ) CODE=IIC; IDE=idea; break;;
       2 ) CODE=IIU; IDE=idea; break;;
       3 ) CODE=PCC; IDE=pycharm; break;;
       4 ) CODE=PCP; IDE=pycharm; break;;
       5 ) CODE=CL;  IDE=clion; break;;
       6 ) CODE=WS;  IDE=webstorm; break;;
       7 ) CODE=RM;  IDE=rubymine; break;;
       8 ) CODE=PS;  IDE=phpstorm; break;;
       9 ) CODE=DG;  IDE=datagrip; break;;
   esac
done

echo "Installing $IDE..."

# Prepend base URL for download
URL="https://data.services.jetbrains.com/products/download?platform=linux&code=$CODE"

# Get location header for file URL
HEADERS=$(wget -qS --max-redirect 0 --spider "$URL" 2>&1)
LOCATION=$(echo "$HEADERS" | tac | grep -m 1 "Location: ")
FILE_URL=$(echo "$LOCATION" | sed 's/.*Location: //')
VERSION=$(echo "$FILE_URL" | sed -En 's/.*\/(.*).tar.gz/\1/p')
echo "File to be downloaded: $FILE_URL"
echo "Latest stable version: $VERSION"

# Set install directory
INSTALL_DIR="/opt/$VERSION"

# Check if latest version has been installed
if [ -d "$INSTALL_DIR" ]; then
   echo "Found an existing install directory: $INSTALL_DIR"
   echo "$VERSION may have previously been installed."
   while true; do
       read -p "Would you like to reinstall? (Y/N) > " REPLY
       case $REPLY in
           [yY] ) echo "Reinstalling $VERSION..."; break;;
           [nN] ) echo "Aborted install."; exit 1; break;;
       esac
   done
fi

# Set download directory
DEST_DIR=$(mktemp)

# Download binary
echo "Downloading $VERSION from $FILE_URL to $DEST_DIR"
wget -cO ${DEST_DIR} ${FILE_URL} --read-timeout=5 --tries=0
echo "Download complete."

# Overwrite installation directory if it exists
if [ -d "$INSTALL_DIR" ]; then
   echo "Removing existing installation in $INSTALL_DIR"
   rm -rf ${INSTALL_DIR}
fi

# Untar file
if mkdir ${INSTALL_DIR}; then
   echo "Extracting $DEST_DIR to $INSTALL_DIR"
   tar -xzf ${DEST_DIR} -C ${INSTALL_DIR} --strip-components=1
fi

# Grab executable folder
BIN=${INSTALL_DIR}/bin

# Add permissions to install directory
echo "Adding permissions to $INSTALL_DIR"
chmod -R +rwx ${INSTALL_DIR}

# Enable to add desktop shortcut
# DESK=/usr/share/applications/${IDE}.desktop
# echo "[Desktop Entry]\nEncoding=UTF-8\nName=${IDE}\nComment=${IDE}\nExec=${BIN}/${IDE}.sh\nIcon=${BIN}/${IDE}.png\nTerminal=false\nStartupNotify=true\nType=Application" -e > ${DESK}

# Create symlink entry
TARGET=${BIN}/${IDE}.sh
echo "Placing symbolic link to $TARGET in /usr/local/bin/"
ln -sf ${TARGET} /usr/local/bin/${IDE}

# Prompt to launch newly installed IDE
while true; do
   read -p "Installation complete. To launch $IDE, run: $IDE
   Would you like to launch $IDE right now? (Y/N) > " REPLY
   case $REPLY in
       [yY] ) eval ${IDE}; break;;
       [nN] ) echo "Done."; break;;
   esac
done

Next, run chmod 755 jetbrains-install.sh && ./jetbrains-install.sh and follow the instructions. Once the chosen IDE has launched, one issue you may encounter is poor font rendering, which can occur on certain Linuxes when running Swing applications, resulting in jagged or broken fonts in the UI and Editor. Here, the best course of action is to use a different font. You can switch your default font in Settings, under the ‘Appearance’ menu.

If you would like to see further support for IntelliJ IDEA, PyCharm, WebStorm, PhpStorm, Clion, RubyMine, DataGrip, et al. on Linux, one way you can contribute is by reporting issues that you encounter on Linux. There is a small but growing community of Linux users, and by reporting issues, you will improve the IDE for everyone. Thanks for reading, and if you have feedback on the install script, please reach out to me!

Inspections, Instructions & Intensions

2014-04-20T00:00:00+00:00

IntelliJ IDEA offers a formidable array of inspections (632 and counting), each thoughtfully crafted to help you deliver clean, bug-free code. One which recently caught my attention, Suspicious Name Combination, is an inspection that detects “suspicious” argument-parameter assignments by analyzing lexical tokens in the method signature. This frequently occurs in dyads such as setSize(width, height), or triads like assertEquals(message, actual, expected) where the order of shared-type parameters can be easily confused and will often slip through static type-checks unnoticed. For example, IntelliJ will alert you when passing a variable named btnHeight into a parameter named width.

If you think about it, this is a Non. Trivial. Inspection. Writing good inspections is a fairly delicate process to begin with due to the tradeoff between sensitivity and specificity. If an inspection is too specific, it will lack adequate coverage - if it is too general, then you’ll end up supporting a bunch of exceptions to handle the edge cases. In many ways, this is the same dilemma faced by diagnostic tests for cancer and other medical conditions. Taking the analogy a bit further, if software is the patient, bugs are the cancer and inspections are the diagnostics. So how exactly does IntelliJ Suspicion work? It’s either very ad-hoc or very clever. Let’s go to the source for a closer look. The first thing we notice is that identifiers are grouped by similarity into so-called name groups.

 protected final List<String> myNameGroups = new ArrayList<String>();
 private final Map<String, String> myWordToGroupMap = new HashMap<String, String>();

 public SuspiciousNameCombinationInspectionBase() {
    addNameGroup("x,width,left,right");
    addNameGroup("y,height,top,bottom");
  }

 protected void addNameGroup(@NonNls final String group) {
    myNameGroups.add(group);
    List<String> words = StringUtil.split(group, ",");
    for(String word: words) {
      myWordToGroupMap.put(word.trim().toLowerCase(), group);
    }
  }

It looks like they’re building a data structure to detect set membership or something. Let’s go a little further. There are three areas of suspicion: at the assignment, call site, and return statement (nice). We’ll focus on the call site, since that’s the most general case.

@Override public void visitCallExpression(PsiCallExpression expression) {
      final PsiMethod psiMethod = expression.resolveMethod();
      final PsiExpressionList argList = expression.getArgumentList();
      if (psiMethod != null && argList != null) {
        final PsiExpression[] args = argList.getExpressions();
        final PsiParameter[] parameters = psiMethod.getParameterList().getParameters();
        for(int i=0; i<parameters.length; i++) {
          if (i >= args.length) break;
          if (args [i] instanceof PsiReferenceExpression) {
            // PsiParameter.getName() can be expensive for compiled class files, so check reference name before
            // fetching parameter name
            final String refName = ((PsiReferenceExpression)args[i]).getReferenceName();
            if (findNameGroup(refName) != null) {
              checkCombination(args [i], parameters [i].getName(), refName, "suspicious.name.parameter");
            }
          }
        }
      }
    }

Forget about what a PSI is for now (it’s something like XPath for ASTs with built-in reflection, you can almost get a feel for it here), this method basically iterates through an argument list. If the argument in question belongs to an existing name group, then it matches that argument name refName to its respective parameter’s name parameters[i].getName(), from the method signature and performs some sort of check.

private void checkCombination(final PsiElement location,
                                  @Nullable final String name,
                                  @Nullable final String referenceName,
                                  final String key) {
      String nameGroup1 = findNameGroup(name);
      String nameGroup2 = findNameGroup(referenceName);
      if (nameGroup1 != null && nameGroup2 != null && !nameGroup1.equals(nameGroup2)) {
        myProblemsHolder.registerProblem(location, JavaErrorMessages.message(key, referenceName, name));
      }
    }

Now if the parameter’s name group findNameGroup(name) doesn’t match the argument’s name group findNameGroup(referenceName) then we have a problem. There is a semantic contradiction. So far we’re just peeling back the implementation layers. The heart of our algorithm lies in findNameGroup(String name). Let’s take a look.

@Nullable private String findNameGroup(@Nullable final String name) {
      if (name == null) {
        return null;
      }
      String[] words = NameUtil.splitNameIntoWords(name);
      String result = null;
      for(String word: words) {
        String group = myWordToGroupMap.get(word.toLowerCase());
        if (group != null) {
          if (result == null) {
            result = group;
          }
          else if (!result.equals(group)) {
            result = null;
            break;
          }
        }
      }
      return result;
    }

Tokenize and scan the name for salient keywords, with a sentinel value: an inconsistent group. Here’s the catch. Not immediately obvious here, but it’s a precaution against false positives! Someone was worried about ambiguous names like bottomRight being flagged for suspicion. If and only if name belongs to exactly one name group then return that group, otherwise return null.

All things considered, this is a very modest heuristic, accomplishing just what it sets out to do. If it were any more ambitious it might attempt to solve a longest common substring against neighboring parameter name groups, and it might also fail twice as often. The truth is difficult to escape: there is no sure substitute for writing clean code. IntelliJ will catch a limited subset of specific name substitutions, but will not prevent a misplaced argument in an arbitrary function with consecutive, order-dependent, shared-type parameters. Until telepathic connectivity is fully supported, best favor monadic functions, use the builder pattern, and name variables intelligibly.

On Static Imports in Java

2014-04-11T00:00:00+00:00

Recently someone asked me, “Breandan, why do I need to write import static org.junit.Assert.*; in order to use JUnit assertions?” I did not immediately have a good answer, besides the tautology that static imports were used to access static members, and made things generally more readable. What I did not realize, is that there is a very explicit reason why fluent interfaces such as jOOQ, Mockito, TestNG and JUnit4 have all chosen to adopt this language feature, and that reason is The Constant Interface Antipattern.

Prior to JUnit4, test classes would inherit the static members of junit.framework.Assert by extending junit.framework.TestCase. TestCase was the parent test class, and by extending it, we had access to a standard set of assertions. Now you might say, “Well Breandan, what does this have to do with The Constant Interface Antipattern? This is a perfectly canonical use of inheritance.” It turns out that by forcing test classes to inherit from TestCase, users are restricted from building more natural test hierarchies and must reimplmenent common functionality or resort to interfaces and utility classes to access a static member. So there must be a better way.

Enter static imports. There is a selective use case here. We must be careful not to pollute the namespace with ambiguous member functions, yet would still like to have some measure of brevity for DSLs and commonly used functions. We would prefer composition over inheritance and allowing the user the flexibility to write standalone test cases. Static imports are appealing for number of reasons: tests use assertions, but assertions ‘belong’ no more to test cases than exceptions belong to a handler. Assertions are frequently used, but seldom all at once (test cases should be short and concise), thereby encouraging users to think carefully about what types of assertions are being used. And assertEquals(...) is simple and recognizable, reducing boilerplate.

You might be curious how to start writing your own tests in IntelliJ with just a few keystrokes. You can simply press [CTRL+SHIFT+T] within the class you would like to cover, select JUnit4 within the wizard (import the library if prompted), and immeadiately begin writing your unit test (don’t forget to annotate with @Test). Once you are ready to assert a result, type assert[CTRL+ALT+SPACE] and navigate to the desired assertion, then press [ALT+ENTER] and statically import the assertion method for maximum typing efficiency.

The End of Illiteracy

2014-02-09T00:00:00+00:00

Reading. It’s something we must all do, and millions of hours are spent each day, passing on that ability to our children. Reciting nursery rhymes, reading children’s books, sounding out each and every syllable for the untrained eye and ear. It’s not something we often think about unless we have children of our own, yet is an inseparable part of our daily life. It is how we learn to write, share, and relive, some have said a path to immortality itself - by inheriting and sharing just a bit of our inner world with the one outside us. Yet some children have no one to teach them, and many more can barely read at a functional level, lacking the education, resources and time. Even those of us lucky enough to receive a college education, most can only read one language. All of that is about to change.

In the next few years, there will be a breakthrough in speech recognition technology. There is a quiet revolution already taking place in the speech recognition community, and one that will change the way we learn to read and communicate forever. It is an engineering solution years in the making, on its way to tremendous fruition. With recent breakthroughs in deep learning using restricted Boltzman machines, the availability of enormous data sets, and a few more steps of Moore’s Law, we will have real-time, native-level speech recognition for the English language. What does this all mean? It means that without a powerful supercomputer, on my smartphone or wristwatch even, and no internet connection whatsoever, I will be able to hold a conversation with a deaf person, using just my voice.

Some say that with the advent of speech recognition and machine translation, we will no longer need to learn new languages and they will all eventually converge into one, I’m not so sure. What I do know, is that we will never be able to upload a language directly into grey matter. Natural language is acquired, reading is not. It requires effort, and constant practice - there is no Broca’s area for reading, at least not yet. Here is where machine learning comes in - we have all the necessary tools to eradicate illiteracy in the next century. And I do not mean reduce illiteracy to ten or one percent, I mean eradicate it, like smallpox - where every single person with the functional ability to read, can read Tolstoy and Hemingway. Where every person above the age of fifteen can learn to become an astronaut or test the speed of light, write stories and teach new ideas to one another. And all this is possible (here is where you will object) to an orphan without access to school or even a single adult.

How is this possible, you may ask? With (1) spaced repetition, (2) speech synthesis, (3) speech verification, and (4) ubiquitous closed captioning, in precisely increasing order of difficulty. The first is easy, but not as easy as you might think. At its heart, is a simple scheduling algorithm, which can be found in plenty of existing flashcard systems, including Anki and Mnemosyne. In order to do it well, we need a few simple machine learning techniques to customize the results. The second is more difficult in general, but can be faked for pre-recorded learning materials, such as audio books. The third is a boundary value problem, but entirely possible at the moment, and several orders of magnitude simpler than speech recognition. If the second is done well, we can achieve the objective of the third without needing perfect accuracy. Finally, automatic speech recognition is the capstone - the crowning achievement, but if we are ever even remotely successful at speech recognition, speech verification is securely within our reach. We only need three.

Grammar	Example
`mmp → mat \| mat * mmp mvp → (mmp) * vec`	\((\mathbf{M}\mathbf{M})\mathbf{v}, (\mathbf{M}\mathbf{M}\mathbf{M})\mathbf{v}, \ldots\)
`mvp → mat * vec \| mat * (mvp)`	\(\mathbf{M}(\mathbf{M}\mathbf{v}), \mathbf{M}(\mathbf{M}(\mathbf{M}\mathbf{v})), \ldots\)
`fun → mat * vec / ‖ mat * vec ‖ rec → fun \| mat * rec / ‖ mat * rec ‖`	\(\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}, \frac{\mathbf{M}\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}}{\\|\mathbf{M}\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}\\|}, \frac{\mathbf{M}\frac{\mathbf{M}\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}}{\\|\mathbf{M}\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}\\|}}{\\|\mathbf{M}\frac{\mathbf{M}\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}}{\\|\mathbf{M}\frac{\mathbf{M}\mathbf{v}}{\\|\mathbf{M}\mathbf{v}\\|}\\|}\\|}, \ldots\)

Breandan's Blog

Computation graphs and graph computation

New decade, new delusions

Everything old is new again

What are graphs?

Induction introduction!

Regular languages

Arithmetic

λ-calculus

Cellular automata

Graphs, inductively

Weisfeiler-Lehman

Graph Diameter

Graph Neural Networks

Graph languages

Graphs, computationally

Examples

Linear chains

Nondeterminstic finite automata

Dataflow graphs

Graphs, efficiently

Programs as graphs

Program synthesis

Roadmap

References

Trust in Automation

Believe in Bias

Work in Progress

Further Reading

Footnotes

Tales of a Traveling Twentysomething

Democratizing Education in China

Democratizing Learning Technology

A Short History of Shorthand

Further Reading

Footnotes

IntelliJ IDEA on Linux

Inspections, Instructions & Intensions

On Static Imports in Java

The End of Illiteracy