Mutation testing metrics (Scala)

Zero-dependency library to calculate mutation testing metrics in Scala.

See mutant states and metrics in the Stryker handbook for more details about mutation testing metrics.

Cross-compiled for Scala 2.12, 2.13 and 3 on JVM and Scala.js. If you want to use this library but require another target or platform (such as Scala Native) feel free to create an issue!

Usage example

Add the dependency to your project :

libraryDependencies += "io.stryker-mutator" %% "mutation-testing-metrics" % version

Or for Scala.js:

libraryDependencies += "io.stryker-mutator" %%% "mutation-testing-metrics" % version

The mutation-testing-elements and mutation-testing-report-schema projects are also published, see NPM_PROJECTS_PUBLISHING for more information.

First create the mutation test report:

import mutationtesting._
import io.circe.JsonObject

val report = MutationTestResult[JsonObject](thresholds = Thresholds(high = 80, low = 10),
  files = Map(
    "src/stryker4s/Stryker4s.scala" -> FileResult(
      source = "case class Stryker4s(foo: String)",
      mutants = Seq(
        MutantResult("1", "BinaryOperator", "-", Location(Position(1, 2), Position(2, 3)), status = MutantStatus.Killed)
      )
    )
  )
)

The MutationTestResult case classes generate a JSON compliant with the mutation-testing JSON schema. It has a type parameter [C] for the type of the used configuration, which can be any JSON object.

Then calculate and use metrics from that report:

val metrics: MetricsResult = Metrics.calculateMetrics(report)

That report will have all the metrics you need:

val mutationScore = metrics.mutationScore
// mutationScore: Double = 70.12987012987013
val killed = metrics.killed
// killed: Int = 162
val survived = metrics.survived
// survived: Int = 69

mutation-testing-metrics-circe

Circe transcodings are provided by the mutation-testing-metrics-circe library to work with JSON if you don't want the extra dependency on circe-generic. It has two dependencies: circe-core and circe-parser.

libraryDependencies += "io.stryker-mutator" %% "mutation-testing-metrics-circe" % version

Encoding & decoding

Import the codec:

import io.circe.syntax._
import mutationtesting.circe._

val encoded: io.circe.Json = report.asJson

val decoded: Either[io.circe.Error, MutationTestResult[JsonObject]] = decode[MutationTestResult[JsonObject]](encoded.toString)

API reference

MetricsResult

A MetricsResult has the following properties, as described in the handbook:

metrics.killed
// res1: Int = 162
metrics.survived
// res2: Int = 69
metrics.timeout
// res3: Int = 0
metrics.noCoverage
// res4: Int = 0
metrics.compileErrors
// res5: Int = 0
metrics.runtimeErrors
// res6: Int = 0
metrics.ignored
// res7: Int = 0
metrics.totalDetected
// res8: Int = 162
metrics.totalUndetected
// res9: Int = 69
metrics.totalCovered
// res10: Int = 231
metrics.totalValid
// res11: Int = 231
metrics.totalInvalid
// res12: Int = 0
metrics.totalMutants
// res13: Int = 231
metrics.mutationScore
// res14: Double = 70.12987012987013
metrics.mutationScoreBasedOnCoveredCode
// res15: Double = 70.12987012987013

• MetricsResult is a sealed trait with three implementations:
  • MetricsResultRoot: The root of a MetricsResult, contains zero or more MetricsResult's
  • MetricsDirectory: Representation of a directory. Has a directory name and zero or more MetricsResult's
  • MetricsFile: Representation of a file with mutated code. Has a filename and zero or more MetricMutant's
• MetricMutant: Contains a MutantStatus

Contributing

To use this project, you will need a JDK and sbt. The recommended way to install sbt is with with Coursier. Alternatively, on macOS/Linux sbt-extras, or on Windows using the official .msi are also options.

This is a normal sbt project, you can compile code with sbt compile and run tests with sbt test. Running sbt +test will compile and test all targets. For more information on cross-compilation in sbt, see https://www.scala-sbt.org/1.x/docs/Cross-Build.html.

This readme uses mdoc. To edit it, please edit the readme in the docs/ folder and call sbt docs/mdoc to compile the readme in the root of the project.