2022-08-10 23:28 UTC Flutter: Static analysis of sample code snippets in API docs
One of the things I am particularly proud of with Flutter is the
quality of our API documentation. With Flutter's web support, we're
even able to literally inline full sample applications into the API
docs and have them literally editable and executable inline. For
example, the
docs for the AppBar widget have a diagram followed by
some samples.
Here's a neat trick, I can even embed these samples into my blog:
These samples actually are just code in our repo, which has the advantage of meaning we run static analysis on them, and even have unit tests to make sure they actually work. (Side note: this means contributing samples is really easy and really impactful if you're looking for a way to get started with open source. It requires no more skill than just writing simple Flutter apps and tests, and people love sample code, it's hugely helpful. If you're interested, see our CONTRIBUTING.md)
Anyway, sometimes a full application is overkill for sample code
and instead we inline the sample code using ```dart
markdown. For example, the documentation
for ListTile has a bunch of samples, but nonetheless
starts with a smaller-scale snippet to convey the relationship between
Material and ListTile.
This leads to a difficulty, though. How can we ensure that these snippets are also valid code? This is not an academic question; it's very hard to write code correctly without a compiler and sample code is no exception. What if a typo sneaks in? What if we later change an API in some way that makes the sample code no longer valid?
We've had a variety of answers to this over the years, but as of
today the answer is that we actually run the Dart static analyzer
against _all_ our sample code, even the little snippets in
```dart blocks!
Our continuous integration (and precommit tests) read every Dart file, extracting any blocks of code in documentation. Each block is then examined using some heuristics to determine if it looks like an expression, a group of statements, a class or function, etc, and is embedded into a file in the temporary directory with suitable framing to make the code compile (if it's correct). We then call out to the Dart analyzer, and report the results.
To make it easier for us to understand the results, the tool that does this keeps track of the source of every line of code it puts in these temporary files, and then tweaks the analyzer's output so that instead of pointing to the temporary file, it points to the right line and column in the original source location (i.e. the comment). (It's kind of fun to see error messages point right at a comment and correctly find an error.)
The code to do all this is pretty hacky. To make sure the code doesn't get compiled wrong (e.g. embedding a class declaration into a function because we think it's a statement), there's a whole bunch of regular expressions and other heuristics. If the sample code starts with `class` then we assume it's a top-level declaration, and stick it in a file after a bunch of imports. If the last line ends with a semicolon or if any line starts with a keyword like `if`, then we stick it into a function declaration.
Some of the more elaborate code snippets chain together, so to make
that work we support a load-bearing magical comment (hopefully those
words strike fear in your heart) that indicates to the tool that it
should embed the earlier example into this one. We also treat
// ... as a magical comment: if the snippet contains
such a comment, we tell the analyzer to ignore the
non_abstract_class_inherits_abstract_member error, so
that you don't have to implement every last member of an abstract
class in a code snippet. We also have a special Flutter-specific
magical comment that tells the tool to embed the snippet into a
State subclass, so that you can write snippets with
build() methods that call setState et
al.
My favourite part of this is that to make it easier to just throw
ignores into the mix without worrying too much about whether they're
redundant, the tool injects
// ignore_for_file: duplicate_ignore into every
generated file.
As might be expected, turning all this
on found a bunch of errors. Some of these were trivial (e.g. an
extra stray ) in an expression), some were amusing (e.g.
the sample code for smallestButton() called the function
rightmostButton() instead), and some were quite serious
(e.g. it turns out the sample code for some of the localizations logic
didn't compile at all, either because it was always wrong, or because
it was written long ago and the API changed in an incompatible way
without us updating the API docs).