Evolutionary programming

I’ve realized that working on Effes is a different beast than working in my day job, primarily due to time and a lack of continuity. It hasn’t been quite as challenging as I thought it would be, but it does result in a slightly different development style.

At work, I come in and get to chew on a problem for hours at a time (modulo the usual office interruptions). If I don’t finish, I’ll pick things up the next day pretty much where I left off, with everything still more or less fresh in my head. But with Effes, I usually work on things an hour here, a half-hour there, maybe a stretch of a few hours on a weekend. Sometimes I won’t have time to work on it for weeks at a time.

So I’ve taken an evolution-like approach to Effes. Good traits are brought in atomically and incrementally; bad traits are culled, but ones that are merely superfluous can stick around for a while. Sometimes I work on pieces that I think will mesh, but I work on them without a grand spec of how to combine them. Instead, I work on each one individually, and then try to stitch them together.

It’s not that I don’t do cleanup (I do!), but the goals are different: there’s less emphasis on a tight, succinct code base, and more on making progress. Less on the ethos, and more on the id.

Two things I haven’t deemphasized are code clarity and maintainability. Like with the day job, those characteristics are vital, since I may not come back to a piece of code for months. On the other hand, if something works fine and is clear, but could maybe be collapsed a bit or consolidated with another class for aesthetic reasons — I have less incentive to do that, since that half hour task might well be the only half hour I get to spend on Effes for two weeks.

These are, of course, skills not unique to my hobby program. I have a bit of a “architectural perfectionist” streak in me, and working on Effes has helped me hone my ability to get things done efficiently, without worrying too much about how pretty they are.

A type by any other name

I think it’s actually going to be easier than I feared.
~ Me, writing Famous Last Words

Uh, yeah. So it turns out it won’t actually be that easy. Instead of just plugging things in straightforwardly, it turns out that I’m going to need to take a tangent to kinda overhaul the classes that represent Effes types.

Here’s the problem. Let’s say I do something like this:

b : Something[Boolean]
case b of:
  Something[True]: doSomethingWith b
  c: doSomethingElseWith c

… then I want two things to happen. Firstly, in doSomethingWith b, I want the type system to know that b : Something[True]. And secondly, in doSomethingElseWith c, I want the type system to know that c : Something[False] | Nothing.

Neither one of those work today, because of an implementation choice I made a while back. To simplify recursive types like Cons[T](head: T, tail: Cons[T] | Empty), I separated each type’s name from its argument. One bit of information says “I’m a Cons with a generic param T” , and a separate class holds the information “given a Cons[E], I can tell you that its arguments are (E, Cons[E] | Empty).”

This means that there’s no place to store the information that in some particular case, the arguments are (T, Empty) (i.e., that this cons represents the last item of a list). That’s exactly the kind of information that the pattern matching can and should provide.

So, before I go further with the pattern matching, I’m going to have to redo my type registration quite a bit: I’m going to need to remove my “arguments for each type” registry altogether and put that information in the types themselves.

I'm too lazy to type

The title is a double joke. It’s a pun with “lazy” and [data] types, but it’s also funny cause it’s true! This post was supposed to be about lazy evaluations in my pattern matching code, but I don’t feel like writing much about it.

And actually, there’s not a whole lot to write.

• the “type subtraction” has to be lazy, because one of the operands is a potentially-infinitely-recursive type. Trivial example: IntSequence(head: Int, tail: IntSequence).
• for the sake of keeping expansions down, it’s better to defer forcing the evaluations as long as possible — but that was covered in my last post
• step three, profit I guess?

Anyway, point is there’s not much to say, so I’m just going to leave it at that.

My next step is t actually plug all this stuff into my compiler. I did a quick look the other day to remind myself what the compiler code actually looks like in that area, and I think it’s actually going to be easier than I feared. There’s already a decent abstraction of “the pattern that’s being matched,” with logic that takes that pattern and assigns variables to it and such, so with luck it’ll be pretty easy to swap in the new abstractions I’ve been using.

In Soviet Russia, recursion invokes you!

This is one of those “I know what worked, but I don’t quite know what I learned” posts.

To implement my type-safe pattern matching, I have to essentially recurse down into two structures: the type of the thing being matched, and the individual pattern that’s trying to match it. That is, given:

t: List[Maybe[Boolean]]     //          call this the haystack type
case t of
    Cons(One(True), _): doWhatever(...) // and this the needle type

… I need to recurse both into the List[Maybe[Boolean]] and into the Cons(One(True), _). The question is, which of those looks like recursion, and which looks like a dispatch? The problem is interesting because both structures are polymorphic: the haystack can be simple type (Cons(head: T, tail: List[T])) or a union type (Cons(...) | Empty), while the needle type can either be a concrete type (One(True)) or a wildcard (_).

Essentially, some piece of code has to look something like this:

Result recursePolymorphically(Arg arg) {
  if (arg instanceof OneThing) { ... }
  else if (arg instanceof AnotherThing { ... }
}

and the question is whether I:

• recurse into the haystack polymorphically, and instanceof-ify the needle
• recurse into the needle polymorphically, and instanceof-ify the haystack

A quick observation that drove my thinking on this: the haystack type is potentially infinite (the tail of a Cons is a disjunction that includes a Cons, the tail of which is a disjunction that includes a Cons, and so on) while the needle is always finite. Thus, the base case must depend on the needle.

I tried the first of those first, since I like the idea of the base case being driven off the method’s arguments rather than an attribute of this; with the needle as an argument, the base case is when that argument represents a wildcard or a concrete, no-arg type.

The problem is that this didn’t work well with laziness (the subject of my next post), since it meant forcing the haystack more than I wanted. This in turn caused types to explode out much faster than they needed to. Instead of ending up with something like:

t': Cons(Nothing, ...)
  | Empty

I might end up with something like:

t': Cons(Nothing, Cons(Nothing, ...) | Empty)
  | Cons(Nothing, Cons(Maybe[True], ...) | Empty)
  | Cons(Nothing, Cons(Maybe[False], ...) | Empty)
  | Empty

This made testing more difficult, as I had to start balancing thoroughness and verbosity in the test code. It also means that in the case of a compilation error, the user would see a much more confusing error message. Would you rather see “saw True but expected Cons(Nothing, …)” or “saw True but expected Cons(Nothing, Cons(Nothing, …) | Empty) | Cons(Nothing, Cons(Maybe[True], …) | Empty) | Cons(Nothing, Cons(Maybe[False], …) | Empty) | Empty”? I just wrote that, and I can barely even make sense of it!

As it happens, the testing was important because this approach lent itself to more bugs, though I haven’t done the introspection to figure out why.

The explosion problem pretty much went away when I switched the recursion and the dispatch. Now, the polymorphic call to the needle’s Any form can take its unforced argument (a thunk representing some piece of the haystack’s type) and shove it into the result, still in its lazy form. The result becomes something like matched={<thunk>}, which keeps things from having to expand further.

Which gets me back to the question at the top: what did I learn? I still don’t know. I tried one technique, identified its flaws, and tried the other — but I didn’t learn how to pick techniques better. My algorithm is still a linear search.

Maybe what I learned is that the base case should be driven off polymorphism, not off method arguments. Or maybe it’s that lazy evaluation and polymorphism don’t mix well. Or maybe that sometimes, you just have to use brute force to figure new stuff out.

Pattern matching using recursion

(part 1 of 4ish) I’ve made a fair amount of progress in the past few weeks, and have mostly implemented the pattern matching I mentioned in my last post. All that remains now (famous last words…) is to hook up all this pattern matching stuff, which I created as a separate unit, to the compiler.

I ran into a few hurdles along the way, which I’ll split into a few posts. But first, to recap. The objective is to take something like this:

t : List[Maybe[Bool]]
case t of:
    Cons(Nothing, Cons(_, Cons(One(True), _))): ...

… and figure that after that first case matcher, t is any list except that whose first element is Nothing and whose third element is One(True).

This has a recursive feel to it, since at each argument you can drill down (Cons -> Cons -> Cons -> One -> True, for instance). I did end up using recursion, but for a while I was fumbling around without a plan, and getting nowhere. In the end, I had to take a step back and think like a CS 101 student: what’s my base case, what’s the recursive step, what kind of thing is being returned, and how is it combined?

• base case: a simple type (not a disjunction) with no args
• recursive steps:
  
  • disjunctive case
  • simple type with args
• return value: a struct that contains a set of matched types and a set of unmatched types.
  For instance, if the possibility is True | False | FileNotFound, and the match case is True, then return value is (matched={True}, unmatched={False, FileNotFound}).
• combining step:
  
  • for disjunctions, recurse down on each component, and combine the results (matched=<matched values from each component>, unmatched similar)
  • for simple types, recurse down on each argument. If any argument returns back no matches, the result is no match. Otherwise, do a cartesian product of all of the matched and unmatched arguments. For each row that corresponds only to matched arguments, create a match; for all other rows, create an un-match.

That last point is hard to explain succinctly, but an example will illustrate it. Let’s say you have:

VehicleType = Car | Truck
Color       = Red | Green | Blue
Vehicle(type: Car | Truck, color: Color)
t : Vehicle

You match t against Vehicle(Car, Red). Recursing down, you find that the first argument is (matched={Car}, unmatched={Truck}) while the second argument is (matched={Red}, unmatched={Green, Blue}). The cartesian product of these arguments (with matches marked with asterisks) is:

*Car,   *Red
*Car,    Green
*Car,    Blue
 Truck, *Red
 Truck,  Green
 Truck,  Blue

Of these six options, only the first row has all matched arguments, so it’s the match; the other rows are unmatched:

matched = {
  Vehicle(Car, Red)
}
unmatched = {
  Vehicle(Car, Green | Blue),
  Vehicle(Truck, Red | Green | Blue)
}

This gave me the overall approach, but I still had a couple problems. The first was dealing with laziness (which is needed to handle infinitely recursive types, like List[T]), and the second was in figuring out how to structure the recursion. I’ll talk about those in the next couple posts, in reverse order.

Getting clever with pattern matching

If I haven’t blogged much lately, it’s because I haven’t worked on Effes much lately. Some of it is because I’ve been busy, and some of it is because my current task is, well, a bit difficult. It’s been hard to find the time and energy to focus on it for more than a half-hour here or there, which is really what I need to do.

The problem I’m working on is pattern matching:

truthy : True | False
myValue = case truthy of
    True: 1
    _: 0

Okay, so that example is pretty easy. The problem is that I really want to disallow what Haskell calls partial functions: functions that might not apply to all inputs that the type system allows. Consider:

possiblyTruthy : True | False | Unknown
myValue = case possiblyTruthy of
    True: 1
    False: 0
    -- no match for "Unknown"

Haskell will happily compile the equivalent of this code, and unhappily throw a runtime exception if myValue is Unknown. For a language that prides itself on its awesome type system, that’s not super helpful!

The easy option is to require an “else” (_: foo) on all case expressions, but that’s annoying (or even dangerous) when you know (or think) that you’ve already specified all the possibilities. I want to do better: I want the compiler to know whether you’ve specified the possibilities. Specifically, I’d like it to require:

• that all possibilities are specified
• that nothing impossible is specified

To do this, I need a way of “subtracting” types.

t : True | False
myValue = case t of
    True: 1  -- t is now (True | False) - True,
             -- so t : False
    False: 0 -- t is now (False) - False
             -- so t : ()
    -- no need to specify a catch-all _: case

For simple expressions like this, the subtraction is easy. But it gets tricker when you allow more complex patterns, ones that let you peer into a value’s components. Consider:

Boolean = True | False
List[T] = Cons(head: T, tail: List[T]) | Empty
bools : List[Boolean]
v = case bools of:
    Cons(True, _): 0
    Cons(False, Cons(_, Cons(False, _))): 1
    Cons(False, Cons(_, Cons(False, _))): 2
    Cons(True, Empty): 3 -- error!
    _: 4

After the first pattern, bools is:

List[Boolean] - Cons(True, _)
=   List[True | False]
  - Cons(True, _)
=   Cons(True | False, List[Boolean]) | Empty
  - Cons(True,  _)

Let’s “factor out” the True | False from the first Cons. I’ll also use one-letter type names as shorthand, since this gets tricky: B for Boolean, T for True, etc.

=   C(T | F, L[B]) | E
  - C(T, _)
=   C(T, L[B]) | C(F, L[B]) | E -- factor out the T | F
  - C(T, _)
=                C(F, L[B]) | E

Okay, that wasn’t so hard. But then, the pattern I matched was pretty simple (“any list that starts with True). As the patterns get more complex, so does the logic; I might need to recurse down an arbitrary number of times on an given argument. I also need to do this lazily: List[T] is potentially infinite, so I can’t just factor everything out and subtract out the simple terms.

One way is to do a lazy breadth-first expansion: produce a sequence of types, each with one more layer expanded, and just keep going down that list until I either find the exact type I need, or find that it can’t possibly exist. That would work, but my spidey sense doesn’t like it. It feels like I should be able to hone in better on the expansions I actually want. That will probably also give me better error messages, if a user misses a possibility or types out something impossible (like the Cons(True, Empty) above, which is impossible since we’ve already covered all lists that start with True). I don’t think it’s super difficult; but it’s not trivial.

Sophisticated primitives

I mentioned built-in types (aka primitives) in my last post. It turns out, pattern matching lets Effes be a bit more expressive than the standard “here’s an int, go do int things with it” operations. For instance, imagine a world where divide-by-zero exceptions can’t happen! (Big disclosure: I don’t think I’ve ever actually triggered one, so they’re not actually that big a deal to me. Still, I like the idea of getting rid of them at compile time.)

Integers in Effes work something like this:

type Int = IntZero | InvValue

type IntValue @builtin:
  + (other: Int) -> Int: @builtin
  - (other: Int) -> Int: @builtin
  * (other: Int) -> Int: @builtin
  / (other: IntValue) -> Int: @builtin

type IntZero: @builtin
    ...

As you can see, there are actually two int primitives, one for zero and one for everything else. Int is just an alias for the disjunction of those two types, and most of the basic math operations take two Ints (this and other). Division is the exception: the denominator must be an IntValue specifically. That means it can’t be an IntZero — and thus that divide-by-zero errors are caught at compile time.

Here’s how you’d use it:

hitsPerMinute = case minutes of
  IntZero: Error -- or whatever
  IntValue: hits / minutes

In this snippet, minutes comes in as a standard Int. We can’t divide by Int, so we throw minutes into a case expression. If minutes is an IntZero, the result is explicitly some sort of error; if it’s IntValue, we can divide by it.

I’m still not sure if I want to do any such trickery for other primitives. I think I won’t, because other primitives don’t have operations that are undefined (ie, throw an exception) for certain inputs. Floating points, for instance, let you divide by zero, add infinity, or do anything else and always get a value back. It may be NaN, but it’s still a value.

It’s actually a bit interesting to me that other languages don’t have this sort of behavior; all you really need to make it work is pattern matching. My guess is that it’s just not a very compelling problem (as I mentioned earlier, I don’t think I’ve ever actually gotten tripped up by it), so it’s not worth the work to catch it. Effes’ type scheme lets me catch it with minimal compiler trickery, which is probably about as much as it’s worth.

Embracing efficient exceptions in Effes

Generics are wrapping up, and I’ve just implemented built-in types at last, so I’m starting to think ahead to next tasks. One of them is exception handling, and I have an idea that combines throwables and standard return types in a way that should settle the “checked vs unchecked” exceptions battle for good. Take that, Java!

Checked exceptions in Java are useful for defining edge cases at API boundaries. For instance, all sorts of things can go wrong in a SQL query, and it makes sense for the entry point to the SQL API to declare, “hey, this method can throw a SqlException, and you should know that.”

But sometimes the best you can do with that exception is to propagate it. This results in a whole bunch of methods declaring throws SqlException (or throws IOException, or, if the programmer is a bit lazy, the infamous throws Exception). Eventually you get to a method like Runnable::run that can’t throw any checked exceptions, so you just handle the exception generically, probably by wrapping it in a RuntimeException and throwing that. Yo dawg, I heard you like exceptions.

So, the problem is that one piece of code wants to treat SqlException as a checked exception, while another wants to treat it as an unchecked exception. Java doesn’t let you do that.

In Effes, there’ll be two ways to handle an exception: by throwing it, or by returning it. This is where the dynamic nature of disjunctive types comes into play.

All exceptions will be unchecked in Effes, meaning that you can throw them willy-nilly:

executeQuery (query: String) -> QueryResult:
  throw SqlException("dummy SQL engine") -- unchecked

But a method can also include an exception as one of its return type disjunctions:

runQuery (query: String) -> QueryResult | SqlException:
  return SqlException("dummy SQL engine") -- "checked"

The latter method essentially turns the exception into a checked exception, because the resulting variable is a disjunction that has to be picked apart with a case statement:

query = runQuery "SELECT * FROM bobby_tables"
summary = case query of:
  SqlException(msg): throw query
  SqlResults: summarize query

Note that in this snippet, we converted SqlException from a “checked” exception (in that it was a disjunction in the result type) to unchecked, just by throwing it.

Moreover, if a method declares an exception as part of its return type, then it’ll never throw it. Trying to throw it from the method directly results in a compile-time exception, and if it’s thrown from down the stack, it’ll be returned immediately. It’s essentially shorthand for:

runQuery (query: String) -> QueryResult | SqlException:
  try:
    <whatever>
  catch (SqlException e)
    return e

This lets us easily convert unchecked SqlExceptions thrown from <whatever> to the equivalent checked exceptions — thus providing that API border we wanted.

Generics are kinda done

So, fun story: I haven’t updated this blog in forever and a day. (Fun corollary: “forever” is 292 days in my world.)

Basically, other stuff came up (new job, life stuff, blah blah) and Effes got put on the backburner for a while. I started revisiting it about a month ago, a couple hours a week, and I’m now more or less officially back on the project.

Generics are… done? Well, not done, but far enough along that I feel comfortable moving on to other things. The syntax is a bit clunky because I don’t have type inference yet (so, maybeInt = Maybe[Int](5) instead of maybeInt = Maybe(5)), and methods can’t declare generic parameters. I’m convinced that what I have will be a good basis for those, though.

The whole exercise was more challenging than I expected. My code ended up being confused as to when a type was reified, and in the end, I went with the approach that a type is always considered reified, but can be “re-reified” at will. That is, Foo[T] is considered reified to the generic type T on Foo, but that can be re-reified to map T on Foo to, say, Int to produce Foo[Int].

So, I’m going to leave generics not-quite-finished and move on to other projects. My next one is built-in types. Despite the examples above, you can’t actually declare or instantiate integers yet — or strings, floats or any other primitive type. (The only thing close to a primitive you can use today is a Boolean, and that’s because it’s just a disjunction of two normal, no-arg types, True and False.) The lack of primitive types makes for some un-interesting tests (maybeTrue = Maybe[True](True)… Zzzzz), which is as good a motivator as any to get things done.