Testing#

nupp test
nupp test --json
nupp test --verbose
nupp test sometest        # extra arguments reach the test command

nupp test builds the configured target, then runs the command the manifest names. Nupp does not supply a test framework — it supplies the step that builds first, so a test never runs against a stale compiler.

Coverage#

nupp coverage
nupp coverage --out reports/coverage
nupp coverage checktest

nupp coverage builds a separate instrumented artifact under build/coverage, runs the configured test command, and writes a static report to build/reports/coverage/index.html by default. Normal builds and their cache never contain coverage probes. The output directory also holds coverage.json, summary.json, and lcov.info for CI or editor integrations.

Agents and other tools can read the full existing report without rerunning tests:

nupp coverage --report-json
nupp coverage --report-json --out reports/coverage

It writes the complete coverage.json document to stdout: per-file metrics, missed locations, and counted coverage sites. Source text and generated Lua remain in the HTML report.

The HTML report has a collapsible source tree, root and per-directory totals, sortable file metrics, and syntax-highlighted Nupp and generated-Lua views. Green means executed, red means executable but missed, amber means a partial branch, and gray is non-executable source such as a type-only line.

The included tests/run.lua runner reads NUPP_COVERAGE_BUILD and writes the coverage shard automatically. Custom runners must load that build directory ahead of their ordinary output and flush the generated global __nuppCoverage.hits to the file named by NUPP_COVERAGE_FILE; otherwise nupp coverage reports incomplete data rather than treating it as zero coverage. Coverage probes intentionally add runtime work, so use ordinary nupp test or the profiler for timing.

Configuring it#

return {
   include = { "src" },

   build = {
      outDir = "build",
      default = "app",
      targets = { app = { kind = "modules", entries = { "app.main" } } },
   },

   test = {
      build = "app",
      argv = { "luajit", "tests/run.lua" },
      env = { NUPP_TEST_MODE = "ci" },
   },
}
 Key    Required  Means
 ─────  ────────  ────────────────────────────────────────────
 argv   yes       The command, as an argv array
 build  no        The target to build first
 env    no        Environment variables, as string to string

test requires a build table to exist in the manifest. The command runs with the project root as its working directory, and anything you pass after nupp test is appended to argv.

Arguments#

nupp test does not parse its arguments — they belong to the test command. Two consequences:

  • -h and --help are honoured only as the first argument. Use -- before a test argument literally named --help.
  • --json is passed along rather than interpreted, so the test command decides what it means.
  • --verbose asks this repository's runner to print output captured from every test. Without it, output is shown only for failures.

JSON output#

nupp test --schema prints the schema the runner in tests/run.lua writes for. The shape is a summary plus a record per test:

{
  "ok": true,
  "total": 724,
  "passed": 724,
  "skipped": 0,
  "failed": 0,
  "durationMs": 41230.5,
  "tests": [
    {"suite": "checktest", "name": "narrowsOnIs", "status": "passed",
     "durationMs": 12.4}
  ]
}

A failing record carries the message and the file and line the error came from, plus its captured output.stdout and output.stderr. Lines are 1-based, as everywhere else; a Lua error carries no column, so none is invented.

This repository's suite#

The compiler's own tests use a small runner: tests/run.lua loads every tests/*test.lua and compiles every tests/*test.nupp. Both kinds return a table of test functions. Nupp suites run with the project's runtime loader, so they can require project modules. The runner prints . for a pass, S for a skip, and E for a failure while it runs. Its summary reports every outcome and elapsed time. With --json, progress is written to stderr and the one JSON document remains clean on stdout. Output from passing tests is captured; it is printed for failures or with --verbose.

./bin/nupp test              # everything
./bin/nupp test checktest    # one suite

Writing one is defining a function on the returned table:

local test = require("assert")
local M = {}

function M.narrowsOnIs()
   local got = checkOf("local s: string | number = 'x' if s is string then end")
   test.equal(got, "")
end

return M

Lifecycle hooks#

A suite may define JUnit-style lifecycle functions. They are not ordinary test cases; a failing hook becomes an explicit synthetic failure:

local M = {}

function M.beforeAll()
   -- once, before this suite's cases
end

function M.beforeEach()
   -- before every case
end

function M.afterEach()
   -- after every case, including a failed setup or case
end

function M.afterAll()
   -- once, even when beforeAll fails
end

function M.opensAConnection()
   -- test body
end

return M

beforeAll failure prevents the suite's cases from running and is reported as beforeAll; afterAll still runs. A failing afterEach is reported with the case failure, if there was one, so cleanup failures do not hide the original problem. The same four names work in Nupp; a hook in a .nupp file needs a : nil return annotation, since that is a strict file and its exports are typed.

The same shape works in Nupp; save this as tests/mathstest.nupp:

test.equal, test.notEqual, test.matches, and test.raises include their expected and actual values in failures. test.skip("reason") records a skipped test. The ordinary assert is also upgraded by the runner to say which falsy value it received, so existing tests get better failures without being rewritten.

Verifying the compiler itself#

./bin/nupp fixpoint

builds a stage-1 compiler, has stage 1 build stage 2, and compares the declared artifacts byte for byte. The working compiler is updated only after a match. This is the standing check that a change to the compiler does not quietly change its output. See distribution for the packaged variant, nupp fixpoint --binary.