Skip to main content

ADR 017 — Test strategy and quality levels

Status​

Accepted

Context​

Brightspacosaurus transforms Markdown into several outputs and crosses multiple boundaries: unified parsing, HTML generation, QTI, PDF conversion, Common Cartridge packaging, Docusaurus preview and Brightspace import. A single test level cannot provide useful confidence for all of these behaviors.

New functionality also tends to affect shared rendering paths. Without a deliberate test strategy, changes can pass a narrow unit test while breaking a generated package, a preview, an accessibility fallback or a real Brightspace import.

Criteria​

  • Tests must run at the boundary where a behavior is decided.
  • Every new behavior must have automated regression coverage.
  • The suite should preserve at least 80% overall line coverage as a quality goal, with branch and function coverage reported when coverage is measured.
  • Generated output must be tested for deterministic and structural invariants.
  • Brightspace-specific behavior needs an explicit manual import scenario because the real LMS is an external system.

Considered options​

Option A — Unit tests only​

Fast and focused, but insufficient for Markdown-to-HTML, CLI, PDF, package and Brightspace behavior.

Option B — End-to-end tests only​

Close to user behavior, but slow, brittle and difficult to diagnose. Many failures would not identify the responsible transformation.

Option C — Layered test pyramid (chosen)​

Combine unit, property-based, integration, automated end-to-end and manual system tests. Each layer covers a different boundary and gives feedback at an appropriate cost. This follows the test pyramid, whose essence Vocke (2018) summarizes as: "Write tests with different granularity" and "The more high-level you get the fewer tests you should have".

Decision​

Use these test levels:

  1. Unit tests for parsers, validators, link conversion, metadata and pure transformations.
  2. Property-based tests for deterministic output, path safety, ordering and format invariants.
  3. Integration tests for Markdown-to-HTML, Markdown-to-PDF, Markdown-to-QTI, CLI commands and package/manifest generation.
  4. Automated end-to-end tests for complete local workflows such as demo-course source to prepared output to IMSCC inspection. Browser automation may be used for Docusaurus preview when that project is available.
  5. Manual system tests for Brightspace import, LMS topic URLs, script policies, responsive behavior and assistive-technology checks. These cannot be fully replaced by local tests.

The durable demo-course is the manual system-test fixture: its Markdown source, deno task demo, fixture assertions and README checklist are versioned. Generated .imscc files are disposable artifacts, not the source of the test.

Every feature change updates the relevant tests and documentation. Architecture, rendering contracts and test-boundary decisions are recorded in the Software Guidebook and relevant design/specification artifacts; contributor expectations are summarized in CONTRIBUTING.md.

Consequences​

Positive:

  • Fast feedback for local logic and clear diagnosis of failures.
  • Regression protection across generated formats and packaging boundaries.
  • Explicit evidence for behaviors that require a real Brightspace course.
  • Coverage is treated as a signal of test completeness rather than a substitute for boundary-appropriate tests.

Negative:

  • The full suite is slower because PDF and integration tests invoke external tools.
  • Manual Brightspace tests remain necessary for LMS-specific behavior.
  • Some end-to-end tests require optional tools such as pandoc or a browser runtime.

References​