SuMo

SuMo is a mutation testing tool for Solidity Smart Contracts.

SuMo was designed to run mutation testing on Solidity projects in a NodeJS environment. It can run test using Hardhat, Brownie and Forge, hybrid test suites, and custom test scripts.

Table of Contents

• Installation
• Configuration
• CLI Usage
• Mutation Operators
• Publications

Installation

To install sumo run npm install @morenabarboni/sumo

Configuration ⚙️

Before using SuMo you must specify your desired configuration in a sumo-config.js in the root directory of your project. The sumo-config.js is automatically generated when SuMo is installed.

Here's a simple example of sumo-config.js:

module.exports = {  
      buildDir: "auto",                               //build directory of the SUT (auto detect)
      contractsDir: "auto",                           //contract directory of the SUT (auto detect)
      testDir: "auto",                                //test directory of the SUT (auto detect)
      skipContracts: ["interfaces", "mock", "test"],  // Relative paths from contractsDir
      skipTests: [],                                  // Relative paths from testsDir
      testingFramework: "auto",                      //testing framework (auto detect)
      minimalOperators: false,                       // use minimal mutation rules
      randomSampling: false,                         //use random mutant sampling
      randomMutants: 100,                            //if random sampling is enabled, generate 100 mutants max
      testingTimeOutInSec: 500                       //testing time-out for a mutant
}

1) SUT directories

SuMo will try to automatically find your project directories based on standard naming conventions (e.g., /contracts and /test). These can be overriden in the sumo-config.js file. Contracts and test files/folders to be ignored by SuMo can be specified as well.

| Field | Description | Default Value | | ------ | ------ | :----: | | contractsDir| relative path to the directory of the contracts to be mutated | auto | | testDir| relative path to the directory of the tests to be evaluated | auto | | buildDir| relative path to the directory of the compilation artifacts | auto |
| skipContracts| blacklist of relative paths to contract files (or folders) | ["interfaces", "mock", "test"] | | skipTests| blacklist of relative paths to test files (or folders) | [] |

2) Testing Frameworks Configuration 🔗

These fields allow to customize the testing frameworks used by SuMo.

By default, testingFramework is set to auto: SuMo will automatically select the testing framework(s) to be used based on the configuration files (e.g., foundry.toml) present in your workspace. If multiple configuration files are found, SuMo will try to run a hybrid testing process.

| Field | Description | Available Options | Default Value | | ------ | ------ | ------ | :----: | | testingFramework| the testing framework to be used for compiling and testing the smart contracts | auto, brownie, forge, hardhat, custom | auto |

Brownie

When choosing brownie:

• SuMo will rely on a local/global brownie installation;
• The smart contracts will be compiled with a minimal compile command (e.g., brownie compile );
• The smart contracts will be tested with a minimal test command and (optionally) by a list of test files to be executed (e.g., brownie test ...testFiles --exitfirst) .

Forge

When choosing forge :

• SuMo will rely on the global installation of foundry;
• The smart contracts will be compiled with a minimal compile command (e.g., forge build);
• The smart contracts will be tested with a minimal test command and (optionally) by a list of test files to be executed (e.g., forge test ...testFiles --fail-fast).
• Make sure that your forge installation is up-to-date to enable --fail-fast.

Custom

If you set testingFramework to custom, SuMo will invoke the compile and test script defined in your package.json. This allows you to customize both scripts and have more control over the testing process. For example, you can define the scripts as follows:

//package.json
 scripts: {
    compile: "hardhat compile",
    test "hardhat test --bail && forge test --fail-fast"
 }

//sumo-config.js
 project: {
    buildDir: "artifacts",
 }

Additionally, you must also explicitly define a buildDir (matching your compile command) in your sumo-config.js.

⚠️ Limitations of Custom Test Scripts:
     * buildDir: must be explicitly specified it in the sumo-config.js
     * skipTests: will be ignored. You have to specify them in your custom script.

3) Mutation Testing Process Configuration

These fields allow you to further customize the mutation testing process:

| Field | Description | Default Value | | ------ | ------ | :----: | | minimalOperators| use minimal mutation rules | false | | randomSampling| use Random Mutant Sampling | false |
| randomMutants| the maximum number of mutants to be tested (only if randomSampling is enabled) | 100 |
| testingTimeOutInSec| seconds after which a mutant is marked as timed-out during testing | 500 |

CLI Usage

Selecting the Mutation Operators

Before starting the mutation process you can choose which mutation operators to use:

| Command | Description | Usage | Example | |---------------|------------------------------------|--------------------------|-------------------------------------| | list | Shows the enabled mutation operators. | npx/yarn sumo list | $ npx sumo list | | enable | Enables one or more mutation operators. If no operator IDs are specified, all of them are enabled. | npx/yarn sumo enable [...ID] | $ npx sumo enable $ npx sumo enable AOR BOR | | disable | Disables one or more mutation operators. If no operator IDs are specified, all of them are disabled. | npx/yarn sumo disable [...ID] | $ npx sumo disable $ npx sumo disable FVR |

Viewing the available mutations

| Command | Description | Usage | Example | |---------------|------------------------------------|--------------------------|-------------------------------------| | lookup | Generates the mutations and creates reports without starting mutation testing. | npx/yarn sumo lookup | $ npx sumo lookup | | mutate | Generates the mutations and saves a copy of each .sol mutant to to ./sumo/mutants. | npx/yarn sumo mutate | $ npx sumo mutate |

Running Mutation Testing

| Command | Description | Usage | Example | |---------------|------------------------------------|--------------------------|-------------------------------------| | pretest | Runs the test suite on the original smart contracts to check if all tests pass and can be successfully evaluated. Pretest is automatically run when sumo test is executed. | npx/yarn sumo pretest | $ npx sumo pretest | | test | Starts the mutation testing process. You can also choose a single mutant / an interval of mutants to be tested by sepcifying <startHash> and (optionally) <endHash>.| npx/yarn sumo test <startHash> <endHash> | $ npx sumo test $ npx sumo test mbc5e8f56 mbg5t86o6| | restore | Restores the SUT files to a clean version. This should be executed if you suddenly interrupt the mutation process. Note that the restore command overwrites your codebase with the files stored in the sumo/baseline folder. If you need to restore the project files, make sure to do so before performing other operations as the baseline is automatically refreshed on subsequent preflight or test runs.| $ npx/yarn sumo restore | $ npx sumo restore|

Viewing the results

SuMo automatically creates a sumo\results folder in the root directory of the project with the following reports:

• mutations.json: List of mutations in json format, synchronoysly updated during testing.
• index.html: A simple web display of the results (you can view this using VSCode extensions like Live Server). From here, you can also download a csv with the results.
• \mutants: Folder with mutated .sol source files (only if generated with sumo mutate)

Mutation Operators 👾

SuMo includes the following Traditional and Solidity-specific operators. Note that not all mutation operators are enabled by default.

Traditional Mutation Operators

| Operator | Name | Mutation Example | Enabled by Default | Minimal Available | | ------ | ------ | ------ | ------ | :----: | | ACM| Argument Change of overloaded Method call | overloadedFunc(a,b); → overloadedFunc(a,b,c); | Y | N | | AOR | Assignment Operator Replacement | += → = | Y | N | | BCRD | Break and Continue Replacement and Deletion | break → continue → break | Y | N | | BLR | Boolean Literal Replacement | true → false | Y | N | | BOR | Binary Operator Replacement | + → - < → >= | Y | Y | | CBD | Catch Block Deletion | catch{} → | Y | N | | CSC | Conditional Statement Change | if(condition) → if(false) else{} → | Y | N | | ER | Enum Replacemet | enum.member1 → enum.member2 | Y | Y | | ECS | Explicit Conversion to Smaller type | uint256 → uint8 | Y | N | | FCD | Function Call Deletion | foo() → | Y | N | | HLR | Hexadecimal Literal Replacement | hex\"01\" → hex\"random\"| Y | N | | ILR | Integer Literal Replacement | 1 → 0 | Y | N | | LCS | Loop Statement Change | while(condition) → while(false) | Y | N | | OLFD | Overloaded Function Deletion | function overloadedF(){} → | Y | N | | ORFD | Overridden Function Deletion | function f() override {} → | Y | N | | SKR | Super Keyword Replacement | x = getData() → x = super.getData() | Y | N | | SLR | String Literal Replacement | "string" → "" | Y | N | | UORD | Unary Operator Replacement and Deletion | ++ → -- ! → | Y | Y |

Solidity Mutation Operators

| Operator | Name | Mutation Example |Enabled by Default | Minimal version available | | ------ | ------ | ------ | ------ | :----: | | AVR | Address Value Replacement | 0x67ED2e5dD3d0... → address.this()| Y | Y | | CCD | Contract Constructor Deletion | constructor(){} → | Y | N | | DLR | Data Location Keyword Replacement | memory → storage | N | N | | DOD | Delete Operator Deletion | delete → | Y | N | | ETR | Ether Transfer function Replacement | delegatecall() → call() | Y | Y | | EED | Event Emission Deletion | emit Deposit(...) → /*emit Deposit(...)*/ | Y | N | | EHD | Exception Handling Deletion | require(...) → /*require(...)*/ | Y | N | | FVR | Function Visibility Replacement | function f() public → function f() private | N | Y | | GVR | Global Variable Replacement | msg.value() → tx.gasprice() | Y | Y | | MCR | Mathematical and Cryptographic function Replacement | addmod → mulmod keccak256 → sha256 | Y | Y | | MOD | Modifier Deletion | function f() onlyOwner → function f() | Y | Y | | MOI | Modifier Insertion | function f() → function f() onlyOwner | N | Y | | OMD | Overridden Modifier Deletion | modifier m() override {} → | Y | N | | PKD | Payable Keyword Deletion | function f() payable → function f() | Y | N | | RSD | Return Statement Deletion | return amount; → //return amount; | Y | N | | RVS | Return Values Swap | return (1, "msg", 100); → return (100, "msg", 1); | Y | Y | | SCD | Selfdestruct Call Deletion | selfdestruct(); → //selfdestruct(); | Y | N | | SFR | SafeMath Function Replacement | SafeMath.add → SafeMath.sub | Y | Y | | SCEC | Switch Call Expression Casting | Contract c = Contract(0x86C9...); → Contract c = Contract(0x67ED...); | Y | N | | TOR | Transaction Origin Replacement | msg.sender → tx.origin | Y | N | | VUR | Variable Unit Replacement | wei → ether minutes → hours | Y | Y | | VVR | Variable Visibility Replacement | uint private data; → uint public data; | N | Y |

Minimal Mutation Rules

Some mutation operators foresee a minimal version:

• The extended operators generate a more comprehensive set of mutants. These guarantee a more in-depth test adequacy assessment, but they can generate more than one replacement per target (e.g., + is mutated in both - and *), which can lead to longer execution times.
• The minimal operators define simplified rules that only inject one replacement per target (e.g., + is mutated in -), limiting the generation of subsumed mutants and speeding up the testing process.

By default, SuMo employs the extended operators. However, you can enable the minimal rules in the sumo-config.js file.

Publications

To cite SuMo, please use the following:

@article{BARBONI2022111445,
  title = {SuMo: A mutation testing approach and tool for the Ethereum blockchain},
  journal = {Journal of Systems and Software},
  volume = {193},
  pages = {111445},
  year = {2022},
  issn = {0164-1212},
  doi = {https://doi.org/10.1016/j.jss.2022.111445},
  author = {Morena Barboni and Andrea Morichetta and Andrea Polini}
}