clj-native

clj-native helps a Clojure or Squint (ClojureScript) library bind a C library. The library declares each C function one time. It can then call the function on each of the three backends below.

Install

npm: ffi-wasm. Clojars: net.willcohen/native, with the namespaces net.willcohen.native.*. The badges show the current version. Do not use a version range, because the API can change.

Use

Make one library value with dispatch/library. Call each C function with dispatch/call!. Register one handler spec for each library with workload-pool/register-handler!. The docstrings of these functions give their options.

Backends

The host and the build of the C library select the backend.

| | native .so, .dylib, .dll | emscripten .wasm | |----------------|-----------------------------------|--------------------------------------| | JVM | Panama, through dt-ffi (:ffi) | GraalWasm, through ccall (:graal) | | JavaScript | not possible | worker pool, through ccall |

A JavaScript host cannot load a native library. The worker pool uses worker-router, which runs on Node worker_threads and in browser Web Workers.

The two WASM backends load the same emscripten build and call C through its ccall. dispatch/library computes the ccall types of each function one time. The FFI backend binds each C symbol directly and ignores these types.

On the JVM, platform-state/try-init! runs the FFI bootstrap and records :ffi. If the FFI bootstrap throws, try-init! runs the GraalVM bootstrap and records :graal. After platform-state/force-graal!, the next init runs only the GraalVM bootstrap.

The GraalVM backend has a different memory model and a polyglot lock. The graal-wasm namespace docstring gives the lock rules. It also gives the exports that the emscripten module must have. The bootstrap-graal-module! docstring gives the loader options.

Build helpers

These parts make the files that the backends load:

• net.willcohen.native.build, in src/bb, makes the native library and the emscripten .wasm with its loader. It runs on babashka only.
• gen-handler makes the handler module for the JavaScript worker pool.
• flake.nix supplies the toolchain. Refer to Nix flake.

The build.clj at the repository root is only the tools.build script for the clj-native jar.

Heap helpers

Because the JVM and JavaScript get access to the emscripten heap through different APIs, clj-native has one implementation of each helper for each host.

| Helper | JVM | JavaScript (npm) | |---|---|---| | ccall | graal-wasm/ccall | handler-heap/ccallMethod | | heap read and write | graal-wasm/read-heap-array, heap-write-bytes! | heapHelpers heap*_get, heap*_set | | malloc, free, UTF-8 | graal-wasm/malloc, free-on-heap, utf8->string | heapHelpers malloc, free, utf8_to_string | | blocking HTTP for C | net.willcohen.native.http | http-bridge, with fetch-worker on Node | | callback from C to the host | callbacks (Panama upcall), graal-wasm/put-js-globals! | a method of the consumer handler | | C string array to strings | graal-wasm/string-array-pointer->strs | none | | C struct to map | graal-wasm/read-struct | none |

heapHelpers(getModule) returns an object of methods. Spread it into the object that the methods(ffi) of the overrides module returns, before the consumer's own keys. A consumer key with the same name replaces the helper.

On JavaScript, a consumer writes its string-array and struct reads in its own overrides module, which knows the struct layouts of its library. On the JVM, read-struct takes the layout as an argument.

Packages

The npm package ffi-wasm ships its own modules in dist/. ffi-wasm.mjs is an esbuild bundle of the page side. handler.mjs holds the worker side. The modules that the two share are files of their own, and each importer imports them by a relative path, so a page or a Node process loads one copy of each. ./handler, ./handler-*, ./http-bridge, ./fetch-worker and ./test-runner point at their own files, and each other subpath points at ffi-wasm.mjs. dist/ holds no other package: a page maps ffi-wasm, worker-router, worker-router/worker-bootstrap, comlink, squint-cljs/ and resource-tracker in its importmap, one copy of each. A page that imports a subpath also maps it to the file of that subpath.

A module worker does not use the importmap of the page. handler.mjs and the shared modules import only each other and node: builtins, and init-pool! gives each handler the URL of handler.mjs. Worker code imports ffi-wasm/handler or a worker-side subpath, not ffi-wasm. A bundler of worker code marks node:* as external. The gen-handler-source docstring gives the contract of the overrides module.

Besides dist/, the package holds macros.cljc, because squint expands the macros at compile time, and macros.mjs, which the squint output of a consumer imports by path.

The jar net.willcohen/native holds the source of these 13 namespaces, and a clj-kondo config export. It holds no .mjs file. Each namespace docstring is the reference.

| Namespace | Contents | |---|---| | dispatch | library and call!, the call path for each C function | | macros | Helpers for a consumer's wrapper macro | | workload-pool | The handler registry above pool, and the JVM executor slots | | pool | The worker-router wrapper, on CLJS | | platform-state | Backend selection and init | | platform | JVM platform detection, and the FFI bootstrap | | graal-wasm | The JVM WASM backend, on GraalVM polyglot | | ffi-mem | JVM native-memory reads and writes, for the FFI backend | | callbacks | JVM dt-ffi upcall registration | | http | The blocking host HTTP transport for a WASM or native library | | gen-handler | The build-time generator of the handler module | | build | Build helpers for a consumer's bb tasks, in src/bb | | test-runner | A cljs.test runner that always exits the Node process |

Build, test and deploy

bb build:js          # compile the npm .cljc modules to .mjs
bb jar               # build the JVM jar (net.willcohen/native)
bb test              # run test:clj, test:cljs and test:bb
bb deploy:npm        # publish ffi-wasm to npm
bb deploy:clojars    # publish net.willcohen/native to Clojars

Each test directory holds the suites of one runtime:

• test/clj/: JVM suites (clojure.test), for bb test:clj.
• test/cljc/: suites for both runtimes. test:clj and test:cljs run the same body.
• test/cljs/: CLJS suites (cljs.test), for bb test:cljs. squint compiles them, and Node runs them.
• test/bb/: tests of the babashka build helpers, for bb test:bb.

Nix flake

Add the flake as an input to a build or a dev shell:

inputs.clj-native.url = "github:willcohen/clj-native";

devShells.default is the clj-native dev shell. It has the Clojure toolchain, zig, and Node for the tests.

lib.<system>.mkCrossShells returns the devShells set (default) for a library that binds a C library. Its arguments are jdk, extraBuildInputs and extraDevInputs. lib.<system>.actualSystem is the real system in a NixOS container, where system can name the host.

Linux and Windows libraries with zig

The dev shell has zig, readelf and llvm-readobj. zig builds the Linux and Windows libraries on a Linux or macOS host. net.willcohen.native.build/zig-toolchain! writes the compiler wrappers for one of five resource directories. It returns the env, the configure --host and the CMake args.

| Directory | zig target | Loads on | |---|---|---| | linux-amd64, linux-aarch64 | <arch>-linux-gnu.2.28 | glibc 2.28 and later | | linux-amd64-musl, linux-aarch64-musl | <arch>-linux-musl | musl | | windows-amd64 | x86_64-windows-gnu | Windows 10 and later |

On musl Linux, platform/extract-and-bind-library! tries <os>-<arch>-musl/ first, and then <os>-<arch>/.

zig links compiler-rt into each library. For C++ code, zig also links libc++, libc++abi and libunwind into the library, in place of libstdc++ and libgcc. These LLVM runtimes are under Apache-2.0 WITH LLVM-exception.

zig links libc dynamically. From glibc, a linux-<arch> library holds only the libc_nonshared wrappers, for example stat and atexit. The glibc link exception covers these wrappers. A windows-amd64 library holds mingw-w64 runtime code, under the ZPL 2.1 and permissive licenses. Ship their notices with the library.

With the env and the CMake args of zig-toolchain!, a build finds no library, header or pkg-config file on the build machine. Give each dependency to CMake by its path, for example -DZLIB_LIBRARY.

Library checks

After the build, check each library before it ships. Each check prints OK, or throws with a list of the errors.

check-linux-lib! reads the library with readelf. It throws when the library is not a shared object for the arch of its directory. It also throws when the library has a run path, or needs libstdc++ or libgcc. A glibc library can load only glibc libraries, up to glibc 2.28. A musl library can load only libc.so.

check-windows-lib! reads the DLL with llvm-readobj. It throws when the file is not an AMD64 DLL. It also throws when the DLL imports a DLL other than KERNEL32, SHELL32 or the UCRT (api-ms-win-crt-*), for example a mingw runtime DLL.

check-darwin-lib! reads the dylib with otool. It throws for a universal binary, because each resource directory takes a thin dylib. It also throws when the dylib loads a library outside /usr/lib and /System/Library, or has a run path. A run path names a directory on the build machine.

License

Apache-2.0 WITH LLVM-exception. Refer to LICENSE.