Hixie's Natural Log

2022-08-10 23:28 UTC Flutter: Static analysis of sample code snippets in API docs

One of the things I am particularly proud of with Flutter is the quality of our API documentation. With Flutter's web support, we're even able to literally inline full sample applications into the API docs and have them literally editable and executable inline. For example, the docs for the AppBar widget have a diagram followed by some samples.

Here's a neat trick, I can even embed these samples into my blog:

These samples actually are just code in our repo, which has the advantage of meaning we run static analysis on them, and even have unit tests to make sure they actually work. (Side note: this means contributing samples is really easy and really impactful if you're looking for a way to get started with open source. It requires no more skill than just writing simple Flutter apps and tests, and people love sample code, it's hugely helpful. If you're interested, see our CONTRIBUTING.md)

Anyway, sometimes a full application is overkill for sample code and instead we inline the sample code using ```dart markdown. For example, the documentation for ListTile has a bunch of samples, but nonetheless starts with a smaller-scale snippet to convey the relationship between Material and ListTile.

This leads to a difficulty, though. How can we ensure that these snippets are also valid code? This is not an academic question; it's very hard to write code correctly without a compiler and sample code is no exception. What if a typo sneaks in? What if we later change an API in some way that makes the sample code no longer valid?

We've had a variety of answers to this over the years, but as of today the answer is that we actually run the Dart static analyzer against _all_ our sample code, even the little snippets in ```dart blocks!

Our continuous integration (and precommit tests) read every Dart file, extracting any blocks of code in documentation. Each block is then examined using some heuristics to determine if it looks like an expression, a group of statements, a class or function, etc, and is embedded into a file in the temporary directory with suitable framing to make the code compile (if it's correct). We then call out to the Dart analyzer, and report the results.

To make it easier for us to understand the results, the tool that does this keeps track of the source of every line of code it puts in these temporary files, and then tweaks the analyzer's output so that instead of pointing to the temporary file, it points to the right line and column in the original source location (i.e. the comment). (It's kind of fun to see error messages point right at a comment and correctly find an error.)

The code to do all this is pretty hacky. To make sure the code doesn't get compiled wrong (e.g. embedding a class declaration into a function because we think it's a statement), there's a whole bunch of regular expressions and other heuristics. If the sample code starts with `class` then we assume it's a top-level declaration, and stick it in a file after a bunch of imports. If the last line ends with a semicolon or if any line starts with a keyword like `if`, then we stick it into a function declaration.

Some of the more elaborate code snippets chain together, so to make that work we support a load-bearing magical comment (hopefully those words strike fear in your heart) that indicates to the tool that it should embed the earlier example into this one. We also treat // ... as a magical comment: if the snippet contains such a comment, we tell the analyzer to ignore the non_abstract_class_inherits_abstract_member error, so that you don't have to implement every last member of an abstract class in a code snippet. We also have a special Flutter-specific magical comment that tells the tool to embed the snippet into a State subclass, so that you can write snippets with build() methods that call setState et al.

My favourite part of this is that to make it easier to just throw ignores into the mix without worrying too much about whether they're redundant, the tool injects // ignore_for_file: duplicate_ignore into every generated file.

As might be expected, turning all this on found a bunch of errors. Some of these were trivial (e.g. an extra stray ) in an expression), some were amusing (e.g. the sample code for smallestButton() called the function rightmostButton() instead), and some were quite serious (e.g. it turns out the sample code for some of the localizations logic didn't compile at all, either because it was always wrong, or because it was written long ago and the API changed in an incompatible way without us updating the API docs).