We’ve got Vitest up and running and have written a few simple tests to check that everything’s working. Today I’m going to look at Vitest’s code coverage story. Code coverage tools provide a useful set of metrics for determining how much of your source code is actually being tested by your test suite.
As ever in the Vite universe, before getting started I have to make a choice.
Choices
Vitest supports two code coverage engines, v8 and Istanbul.
v8 uses the built in code coverage measurement in the Chrome v8 engine. It uses a combination of metrics that v8 already collects together with additional block level instrumentation in the generated byte code.
Istanbul is a long standing JavaScript test coverage tool. It works by instrumenting your JavaScript source code, inserting counter increments for each statement.
So which is best? The wisdom of the internet says “it depends”. There’s no clear cut consensus so I guess I’ll need to try them both.
v8
The big advantage with v8 is there’s no need for upfront instrumentation and transpilation. Everything happens during the JIT compile within v8. Instrumentation is at the block level. This all means that raw speed is much higher than Istanbul. The lead maintainer of Istanbul suggests 10% performance penalty for v8 vs 300% for Istanbul.
There’s no control over what gets instrumented. Everything that is executed, including the many npm modules your code sits on, are instrumented. Execution data for everything is collected and post-processed, with most of it then thrown away when it doesn’t relate to your code. That may negate the raw speed benefits of v8 if, like me, you have very little user code sitting on a mountain of third party modules.
There are some reports that
- v8 can get mixed up between user code and generated code
- Mark non-executable code like TypeScript type definitions as uncovered
- Doesn’t track coverage of an if-statement that evaluates to false unless it has an explicit else clause.
- v8 output is converted to istanbul compatible format and then fed into the istanbul reporting tools. This can lead to a loss of precision where v8 and Istanbul formats don’t align.
Enough caveats, let’s get it installed.
% npm install -D @vitest/coverage-v8
added 26 packages, and audited 375 packages in 6s
found 0 vulnerabilities
Then run it after configuring vite.config.ts to only include files in the src directory and to exclude files in src/test.
% npm run test -- --run --reporter verbose --coverage
Duration 921ms (transform 252ms, setup 403ms, collect 377ms, tests 31ms, environment 850ms, prepare 194ms)
% Coverage report from v8
----------------------------------|---------|----------|---------|---------|-------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
----------------------------------|---------|----------|---------|---------|-------------------
All files | 58.08 | 58.69 | 65.21 | 58.08 |
App.tsx | 0 | 0 | 0 | 0 | 1-28
VirtualList.tsx | 93.38 | 66.66 | 75 | 93.38 | 39-40,50-52,89-91
VirtualScroller.jsx | 0 | 0 | 0 | 0 | 1-94
main.tsx | 0 | 0 | 0 | 0 | 1-12
useAnimationTimeout.ts | 81.13 | 66.66 | 66.66 | 81.13 | 16,34-39,46-48
useEventListener.ts | 100 | 70 | 66.66 | 100 | 31-35
useFixedSizeItemOffsetMapping.ts | 100 | 100 | 100 | 100 |
useIsScrolling.ts | 100 | 25 | 100 | 100 | 16-20
...iableSizeItemOffsetMapping.ts | 0 | 0 | 0 | 0 | 1-54
useVirtualScroll.ts | 88.46 | 50 | 100 | 88.46 | 13-15
vite-env.d.ts | 0 | 0 | 0 | 0 | 1
----------------------------------|---------|----------|---------|---------|-------------------
Istanbul
Istanbul instrumentation requires a transpilation pass implemented using a Babel plugin. Fortunately, everything is handled for you by Vitest. The big advantage with Istanbul is that instrumentation happens at the level of individual lines of code. You should get very precise results. Only the code you’re interested in gets instrumented.
Instrumenting source code depends on a deep understanding of the Javascript language to instrument correctly without altering meaning. In the past, updates to Istanbul have lagged behind changes to the Javascript spec.
Installing Istanbul is as simple as v8.
% npm install -D @vitest/coverage-istanbul
added 31 packages, and audited 406 packages in 6s
found 0 vulnerabilities
You do need to update vite.config.ts to specify Istanbul, as v8 is the default. After that, you run it in exactly the same way.
% npm run test -- --run --reporter verbose --coverage
Duration 992ms (transform 578ms, setup 412ms, collect 697ms, tests 31ms, environment 806ms, prepare 151ms)
% Coverage report from istanbul
----------------------------------|---------|----------|---------|---------|-------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
----------------------------------|---------|----------|---------|---------|-------------------
All files | 54.7 | 45 | 55.81 | 56.25 |
App.tsx | 0 | 0 | 0 | 0 | 12
VirtualList.tsx | 86.84 | 84.61 | 80 | 85.71 | 39,50-51,89-90
VirtualScroller.jsx | 0 | 0 | 0 | 0 | 4-74
main.tsx | 0 | 100 | 100 | 0 | 7
useAnimationTimeout.ts | 71.42 | 50 | 85.71 | 71.42 | 16,35-38,46-47
useEventListener.ts | 87.5 | 60 | 85.71 | 91.3 | 31,35
useFixedSizeItemOffsetMapping.ts | 100 | 100 | 100 | 100 |
useIsScrolling.ts | 69.23 | 42.85 | 20 | 100 | 8-20
...iableSizeItemOffsetMapping.ts | 0 | 0 | 0 | 0 | 5-51
useVirtualScroll.ts | 87.5 | 50 | 100 | 87.5 | 14
----------------------------------|---------|----------|---------|---------|-------------------
Comparison
Istanbul is slower but not by much. It’s significantly slower in the transform and collect phases. There must be some parallel execution going on as it has little impact on overall duration.
There’s a lot of variation in the coverage percentages, with v8 generally reporting higher percentages. When you look at the uncovered line numbers for the files that matter, such as VirtualList.tsx and useAnimationTimeout.ts, they’re very close.
By default, Vitest also outputs a detailed html report. Let’s look at VirtualList.tsx in more detail.
v8
Istanbul
The html reports include a per line execution count in green, highlight unexecuted lines in pink, highlight the unexecuted part of a conditional in yellow and annotate if statements with an I or an E when the if or else part of the statement is unexecuted.
You can see immediately why the percentages are different. v8 appears to work out which lines aren’t covered and then mark every other line in the file as executed, even if they’re blank or comments. Istanbul’s source code level instrumentation only considers lines that include executable statements as significant and ignores everything else.
v8 can’t tell the difference between an if statement, a ternary condition or the condition in a for loop. It highlights them all as conditional expressions. Istanbul is more precise.
If you look carefully at the summary output from each coverage run, you’ll see that v8 includes an additional file, vite-env.d.ts. This is a vite configuration file which includes a single non-executable line. As such, Istanbul ignores the whole file. With v8, I would have to manually exclude it.
There’s a significant difference in the uncovered lines reported for useIsScrolling.ts. Let’s dig into that next.
v8
Istanbul
v8 can tell that one side of the conditional expressions hasn’t been executed but the reporting isn’t precise enough to say which side. Istanbul has the detail needed.
v8 completely misses two other issues which Istanbul has highlighted. The function has a default argument of window but our test never calls useIsScrolling without an argument. We pass lambdas to useEventListener and useAnimationTimeout. However, as our tests don’t send any events, they never get executed.
For me, Istanbul is the clear winner. Unless speed becomes an issue, I can’t see any reason to use v8.
Vitest UI
As well as its command line output, Vitest also has its own UI with support for code coverage. Let’s check that out.
% npm install -D @vitest/ui
added 6 packages, and audited 412 packages in 10s
found 0 vulnerabilities
You activate the UI by adding the --ui switch when running vitest on the command line. You get the normal command line output as well as launching the web based UI.
At first glance there’s not a lot beyond the command line output. You get a list of source files that include tests that you can drill into, allowing you to filter test output by file.
The UI has an integration with the coverage tool. Unfortunately, there’s nothing particularly integrated going on. The coverage button opens up an iframe with the static coverage report dropped in.
Maybe the UI will do something more interesting if I have a failing unit test?
I get the same information as the command line but instead of an excerpt of the relevant lines of code there’s a button to open the code in my editor. That could be interesting …
Could not open VirtualList.test.tsx in the editor.
The editor process exited with an error: spawn code ENOENT.
On dear. A quick Google suggests that the problem might be that I haven’t set up Visual Studio Code to launch from the command line. I followed the instructions and tried again.
Hurray, it works!
This could be a decent development experience with my usual split screen setup of browser on the left and editor on the right.
Visual Studio Code Integration
While looking at development experience, I should give the Visual Studio Code plugin for Vitest a try. It claims to support integrated test running, debugging and editing.
The plugin was simple to install. The link from Visual Studio Code Marketplace opens up the Visual Studio Code plugin interface and a single click downloads and installs. You do need to manually configure the plugin to tell it the command line needed to run vitest. In my case, that’s npm run test --.
You get a new test tool in the left toolbar. Clicking on it brings up a list of tests in the current project. useEventListener.ts is included because it contains an in-source test but it doesn’t run. In-source tests are currently not supported. Support for displaying code coverage is also on the wish list.
On the positive side, the integration between test and editor works well. Clicking on a failing test takes me straight to the corresponding code with a tooltip showing the assertion failure. The integrated debugger also worked first time. I was able to set a breakpoint and rerun the failing test under the debugger.
You can do a one off run of an individual test or the whole suite. You can also run the test suite in watch mode so that it automatically reruns relevant tests when you save a file. Any failures in the file being edited are shown in context with tooltips.
Conclusion
I think Vitest’s code coverage tools are going to be helpful for me. They make it very easy to see where additional test cases are needed.
Next time, I’ll start filling in the gaps and see how much I can improve my coverage.