Of Optional and nulls

Here at last is that rant about Optional<T> I've promised for so long. Let me preface it by saying that I am not about to propose an ideal way of handling nulls in Java; I don't think Java's null handling will ever be great. That said, there are better and worse ways of doing it, and I think Optional<T> isn't the best way. What's worse, it's edging out a better way.

For the unfamiliar, Optional<T> is a Guava class that aims to eliminate NullPointerExceptions. It has two forms: Optional.absent() and Optional.of(T item). Rather than a method passing back a nullable Foo, it returns an Optional<Foo>. You then call isPresent(), followed by get() iff the item is present.

Optional<Foo> myFooOpt = tryGetFoo();
if (myFooOpt.isPresent()) { // like a != null check
    Foo myFoo = myFooOpt.get();
    // work with the foo
} else {
    throw NoFooFoundException(); // or whatever
}

The idea is that since you have to call get() to get at the Foo, you'll probably remember to check isPresent first — and thus, no NPEs. It seems reasonable enough, but there are two big problems with it. First, it's verbose; and second, it's not backwards compatible.

The verbosity comes down to a lack of pattern matching in Java. Optional<T> is inspired by functional programming languages that have pattern matching — think of it (very roughly) as an instanceof check combined with an un-constructor. Here's how you'd use Haskell's equivalent of Optional<T>:

case tryGetFoo of
    Just foo -> handleFoo foo
    _ -> handleNoFoo

See how much cleaner that is? Optional<T>-type constructs really benefit from a terse way to get at the wrapped object. Pattern matching lets you do this two ways: by combining the isPresent() and the get(), and by therefore eliminating the need for that temporary, throwaway reference to myFooOpt.

Java is trying to move away from verbose boilerplate; one could argue that the driving force behind both Java 7 and 8 is conciseness, not new features. So why is the Java world embracing the overly-verbose Optional<T>?

The backwards compatibility problem is more clear-cut: existing libraries can't be retrofitted with Optional<T> without huge changes to how overload and method resolution is handled. For instance, Map.get returns V — you can't just change it to return Optional<V> without breaking a lot of code.

Before Optional<T> got cool, one idea people had was to use annotations to do static analysis on the code. Mark a field as @Null, and you know it can be nullable; try to use it without checking for nullity, and you'll get a warning. Nullity can be propagated through result types and arguments, and it all checks out at compile time.

The best part is that you can retrofit it to existing classes. Map.get will never return an Optional<V>, but it could return a @Null V.

There were a few different attempts at these checks, each leading to different sets of annotations. If I had it my way, we'd see one of these — preferably a concise one — get Oracle's official blessing and widespread usage.

A type checker has to be conservative, and that means that you'd have to assume that legacy code always returns nullable references. On the other hand, for new code you'd want an un-annotated method to be assumed to be @NotNull (to cut down on verbosity). This mismatch could be solved in three ways.

• Classes compiled annotated with a new @NullChecked annotation would also have their methods assumed to be @NotNull.
• All newly compiled code would assume @NullChecked
• The type checker could take additional inputs in the form of files that list methods which should be treated as @NotNull regardless of their bytecode.

The third one of those would mean that you could mark methods as not-nullable without touching their bytecode at all. This could be useful for some serialization issues, but more importantly, it would let people locally update projects without waiting on their maintainers.

With that migration path in place, compilers could start treating unsafe dereferencing as errors rather than warnings. And maybe, just maybe, Java can recognize it as important enough as to warrant syntactic sugar: T? as shorthand for @Null T. Kotlin employs a similar trick, and while I haven't actually used it, it sure looks nice.

There are other tricks you can do with annotations that expose a lot of power (including how it interacts with subtyping, etc), at the cost of more complexity. I'm not sure Java needs all those — but even without any of them it's still at least as powerful as Optional<T> — with the added benefit of backwards compatibility.

I'm not sure why annotation-based static analysis never caught on. Maybe the pushes were too fragmented, and developers weren't willing to hack in ugly ways to solve backwards compatibility (like my "additional inputs" file)? Maybe the edge cases are just too many and complicated? A quick google search didn't give me any answers.

CoffeeScript should handle callbacks better

I want to add a quick addendum to yesterday's post about best practices. I mentioned the staircase problem caused by Node's reliance on callbacks: if one action is a prerequisite for other actions in a method (for instance, you query a database and then act on those results), the rest of that method ends up indented.

Node has another big problem, which is that its target language, JavaScript, is awful. Luckily, one of Node's third-party modules, CoffeeScript, provides a decent language that compiles down into JavaScript. We get all the Node goodness without the CoffeeScript badness!

Since CoffeeScript is a new language that can evolve quickly, and since one of its main use cases is Node, and since Node relies heavily on callbacks... why not add some sugar to make callbacks a bit nicer? I propose a way to bind callback arguments to left-hand variables. This is actually pretty similar to what Haskell does with its do notation, and for pretty similar reasons.

Let's take a simple, imperative-with-exceptions snippet of code:

try
  res1 = func1 arg1
  [res2a, res2b] = func2 res1
  if res2a is "foo"
    doFoo()
  else
    res3 = func3 res2b
    doBar res3
catch err
  handle err

That's pretty simple. Watch how gross it turns when we use callbacks instead of just returning back the results:

func1 arg1, (err, res1) ->
  if err?
    handle err
  else
    func2 res1, (err, res2a, res2b) ->
      if err?
        handle err
      else if res2a is "foo"
        doFoo()
      else
        func3 res2b (err, res3) ->
          if err?
            handle err
          else
            doBar res3

My suggestion is to create some sugar for that. It would look something like this:

do and throw err...
  (err, res1) <- func1 arg1
  (err, res2a, res2b) <- func2 res1
  if res2a is "foo"
    doFoo()
  else do...
    (err, res3) <- func3 res2b
    doBar res3
  catch err
    handle err

Notice how similar this is to the original, easy-to-read, imperative style. The general idea is simple: the new do... syntax introduces a block of code in which callback variables can be bound on the left-hand side. Every time that happens, it starts a new callback nested in the previous one. If you provide the and throw varname syntax, then it treats left-hand bound variables of this name as errors, and if one ends up being non-null, its callback will run the code in the catch block and nothing else.

I won't pretend this is a small bit of sugar; it probably has some interesting edge cases, and the concept might be a bit weird to grok for someone who's new to it. But it's an elegant solution to a real problem that's pretty significant for a major part of CoffeeScript's target audience.

Tutorials need a section on best practices

I took a bit of a break from Effes over the Fourth of July weekend to relax with friends, watch some TV and teach myself node.js by jumping into a small project. The process reminded me of a problem I've hit a few times in various self-learning exercises: docs for languages and frameworks tend to focus almost exclusively on syntax, APIs and other hard facts, but they don't talk much about best practices.

With Node, my question was how to get around the "staircase problem," where indentations march the code off the right side of the screen. Node is non-blocking and uses callbacks for just about anything that goes to the outside world. For instance, its mysql module puts its result set into a callback. That has some neat benefits, it leads to indentation hell. Here's an example in CoffeeScript:

m = mysqlPool()
m.query getFizzes, (err, fizzes) ->
  if err?
    console.log "Error: #{err}
  else
    foos = [fizz.foo for fizz in fizzes]
    for foo in foos
      do (foo) ->
        m.query selectBars, [foo], (err, bars) ->
          if err?
            console.log "Error 2: #{err}
          else
            console.log("Found a bar!", b) for b in bars

That barely even fit in this blog layout; I had to change bar to b in the last line. And while it's true that blog layouts aren't where most coding happens, the staircase problem is annoying and makes code hard to read.

Now, this is not the biggest problem in the world, but it does make code a bit hard to read if the sequence of actions gets much longer (it's only two above). I solved it in one kinda-hairy section (4-long sequence with a conditional, oh man!) with a bit of refactoring, but the resulting code was actually a bit harder to follow because it broke up the logic's natural flow.

There are other ways I could have solved the problem, and there's at least one third-party module that may help, but here's the real point: the Node docs don't help me out. They don't even acknowledge the problem. If the best thing to do is to use async with CoffeeScript, Node should tell me so.

Maybe it's unfair to pick on Node's docs, since there's barely anything as far as a tutorial. You've got a "hello world" example on the front page, a link to the built-in library's APIs, and that's about it. But it's not just Node.

For instance, the intro guide for Ruby on Rails promises to teach the reader "the basic principles of MVC," but it never mentions MVC again, and doesn't give much guidance for when to put logic in a controller vs a model. Github doesn't talk about the benefits of forking vs branching — the first three hits for "git fork or branch" on google are all on Stack Overflow. Guice's tutorial doesn't tell us where to put our injector. And so on.

I'll give the RoR guide some slack, because MVC has been around for a while; maybe they just assumed that most of their readers would already know it. But in general, the more newfangled a technology is, the more its project should tell newbies not just how to use it, but how to use it well.

Weak typing is weak

I spent a couple hours the other day digging through layers of some unfamiliar code to track down a bug that ended up being a really simple, one-line fix. That's nothing spectacular, but this bug had to do with weak typing, which gives me a good opportunity to rant against that. Here was the fix, with some variable names changed to protect the innocent:

--- foo = bar[key] or null +++ foo = if bar[key]? then bar[key] else null

The idea was to get bar[key] if there is such an entry, or else default to null. I recognize that pattern because I've seen it before, and I've written similar code. It's a common pattern; the CoffeeScript page even uses it in an example. (In case you're not familiar with CoffeeScript/JavaScript, the original line works because a or b in CoffeeScript means "a if a is a true value, otherwise b." It's used as a quick get-or-default pattern.)

The problem arises if bar[key] is a defined, useful value that happens to also be a false value — like an integer 0, an empty string, or (to state the obvious), the boolean false. In that case, since the first expression is a false value, the whole expression evaluates to the second expression, null; the original line translated a 0 into a null.

In other words, the problem is that the first expression in a or b doesn't have to be a boolean, it just has to be interpretable as a boolean — and it just so happens that any type in JavaScript/CoffeeScript can be interpreted as a boolean, meaning that it's really easy to use the a-or-b-as-defaulter pattern, which happens to be wrong in many cases. And come to think of it, why are we using the same construct for boolean logic and getter-with-default, which are two completely different concepts?

Here's my issue with that: it's always valid code, and sometimes it's a valid pattern, but the language doesn't tell you when. And that, in a nutshell, is my beef with weak typing.

Now, there are lots of ways to write bugs in code, and it's reasonable to ask why I'm singling out this one class of bugs. I would argue that whereas other bugs are due to misrepresenting a problem, or to applying the wrong solution to a problem, this bug is due to applying the wrong dialect  of the correct solution. Ugh. Debugging is enough of a pain without the compiler encouraging us to write the programming equivalent of a tongue-twister. Weak typing makes it easy to write bugs, and, once you've written them, the type system does absolutely nothing to help you find them.

When it comes to programming, I want the computer to mean what I say. The quid pro quo is that I need to say what I mean. If what I mean is "bar[key] if it exists, otherwise null," then the language should require that I just say that. It's well and good for a language to make that pattern succinct, but blurring the lines between boolean logic and plain-old-values is funky reasoning that can lead to funky bugs.