Eve AWS Lambda MicroVM sandboxes

This version targets Eve 0.75.1.

Extracted from vercel/eve#208, authored by Andrew Barba, and adapted into a standalone package for Eve 0.75.1. This package is licensed under Apache-2.0; see LICENSE. See NOTICE for upstream attribution and a summary of the packaging changes.

The AwsLambdaMicrovmSandbox provider runs each durable eve sandbox in an ARM64 AWS Lambda MicroVM.

eve creates and tags MicroVM images, launches MicroVMs, and stores image artifacts, leases, template descriptors, and full-filesystem checkpoints under one prefix in your S3 bucket. eve does not create the bucket, IAM roles, VPCs, or network connectors.

Install

This package targets Eve 0.75.1 exactly:

pnpm add [email protected] eve-aws-lambda-microvms

Configure the backend

Create the bucket and roles first, then author the sandbox:

import { defineSandbox } from "eve/sandbox";
import { AwsLambdaMicrovmSandbox } from "eve-aws-lambda-microvms";

export const environment = AwsLambdaMicrovmSandbox.environment({
    applicationId: "analytics-agent",
    region: "us-east-1",
    artifactBucket: "company-eve-sandboxes",
    artifactKmsKeyId: process.env.EVE_AWS_ARTIFACT_KMS_KEY_ARN,
    buildRoleArn: process.env.EVE_AWS_BUILD_ROLE_ARN!,
    executionRoleArn: process.env.EVE_AWS_EXECUTION_ROLE_ARN,
});
export default defineSandbox(async () => {
  const sandbox = await environment.open();
  await sandbox.run({ command: "dnf install -y git jq" });
  return sandbox;
});

applicationId is a stable resource namespace, not a display label. Keep it identical at build and runtime. The package replaces Eve 0.75.1's path-derived key scope with this application scope so templates and sessions remain stable when build and deployment roots differ. The bucket must be in region. The default prefix is eve/lambda-microvms/<application-id-hash>; set artifactPrefix when the bucket policy requires a fixed path.

artifactKmsKeyId is optional. When supplied, eve sends explicit aws:kms and key-ID headers on JSON, image-artifact, and multipart checkpoint writes. AWS accepts a key ID, key ARN, alias name, or alias ARN; cross-account keys require an ARN. Grant callers kms:Encrypt, kms:Decrypt, and kms:GenerateDataKey as needed for that key. When omitted, eve sends no SSE headers and preserves the bucket's default encryption behavior.

The important defaults are 2 GiB baseline memory, an eight-hour maximum lifetime, suspension after five minutes without endpoint traffic, suspended retention for 30 minutes, automatic resume, no shell access, and no guest execution role. For compatibility, omitted networkingMode (or explicit "legacy") retains 0.1.0's managed Internet connector defaults. Production callers should use the explicit fail-closed "customer-managed" mode below. Supplying an execution role enables CloudWatch runtime logging by default. Set runtimeLogging: false to disable it.

Customer-managed runtime networking may be paired with the exact AWS-managed INTERNET_EGRESS connector for image builds when the build requires public package repositories. This intentionally gives the build unrestricted Internet egress; the runtime connector remains customer-managed and fail-closed.

eve dev and eve start provision authored bootstrap and workspace templates. Eve's Vercel build hook also prewarms them during eve build. When Eve supplies no template key, this package lazily provisions an empty application template during the first session create. A legacy caller therefore needs image-build permissions unless the same default template was already provisioned.

Separate image reconciliation from runtime

Production deployments can make image creation an explicit environment step. Call reconcileAwsLambdaMicrovmImage() under a dedicated bake identity, persist its returned JSON, and pass that exact identity back as verifiedImage at runtime:

import {
  AwsLambdaMicrovmSandbox,
  reconcileAwsLambdaMicrovmImage,
} from "eve-aws-lambda-microvms";

const verifiedImage = await reconcileAwsLambdaMicrovmImage({
  applicationId: "analytics-agent-production",
  region: "us-east-1",
  artifactBucket: "company-eve-sandboxes-production",
  buildRoleArn: process.env.EVE_AWS_BUILD_ROLE_ARN!,
  baseImage: {
    arn: process.env.EVE_AWS_AL2023_IMAGE_ARN!,
    version: process.env.EVE_AWS_AL2023_IMAGE_VERSION!,
  },
});

const environment = AwsLambdaMicrovmSandbox.environment({
  applicationId: "analytics-agent-production",
  region: "us-east-1",
  artifactBucket: "company-eve-sandboxes-production",
  verifiedImage,
});

The returned schema-versioned identity records the controller artifact digest, controller protocol, exact base image ARN/version, public CA digest, build connector and lane identities, memory, image ARN/version, Region, and complete configuration digest. Reconciliation is deterministic and idempotent: the same inputs reuse the same successful image version. Selecting a previously recorded identity is an explicit rollback.

The verified runtime path rebuilds only the deterministic bytes in memory to check their digest, checks the complete configuration identity, and reads the exact AWS image version to require an active successful result. It does not upload image artifacts, discover a newer base image, call CreateMicrovmImage, or require buildRoleArn. Keep the bake and runtime identities in separate IAM roles and accounts/environments as appropriate. The identity contains public CA digests and public configuration only; never put a CA private key in any option or artifact.

Persistence and lifecycle

Before eve terminates a session, it freezes workload processes and publishes the complete writable overlay to S3. The archive preserves numeric ownership, modes, links, ACLs, xattrs, device entries, and overlay whiteouts. It excludes /proc, /sys, /dev, /run, and controller state.

AWS terminates every MicroVM by its configured maximum duration, at the end of suspended retention, or after an operational failure. On the next turn, eve launches the exact image version recorded in the checkpoint and restores all writable paths, including changes under /etc, /usr/local, /root, /var, /tmp, and /workspace. Files survive replacement; processes do not. If AWS has recalled or removed the recorded image version, eve leaves the checkpoint intact and fails instead of restoring only /workspace onto a different image.

Session ownership and the committed checkpoint metadata share one conditional S3 authority write. Every takeover advances a fencing generation; stale holders cannot publish a checkpoint over a successor, including when their S3 write was already in flight. Checkpoint object names are unique to each upload so a stale upload cannot replace a successor's bytes. Eve-returned session metadata contains no AWS credentials, activation tokens, or presigned URLs.

Bounded launch authority

launchTimeoutMs defaults to 240,000 ms and accepts positive integer values up to that limit. Set it below your owning workflow attempt timeout. The budget starts after bucket validation and any lazy template provisioning, and includes session lease acquisition, template/session reads, activation creation, RunMicrovm, controller readiness, and restore. Build-time image provisioning has its own lifecycle and is not included in this budget.

The launch lease's persisted expiry never exceeds its absolute deadline. Deadline or cancellation aborts the SDK request and rejects the caller even if an injected provider ignores cancellation. The final ownership check remains bounded. Promotion happens synchronously when returning the ready handle, and the first full-TTL renewal is scheduled afterward, so a stalled renewal response cannot block launch. The ready session then uses the normal renewable ten-minute lease; the short launch budget does not limit its session lifetime. The concrete backend's create method also accepts abortSignal; the pinned Eve 0.49 API does not supply a workflow cancellation signal automatically.

Retries replay the same durable client token and exact request payload. An unconfirmed request survives lease expiry and release. After a successful checkpoint/retirement, the next launch receives a new token and activation. A retry cannot silently change a pending launch's image or runtime request configuration. A late result is rejected; cleanup terminates it only after acquiring session authority and confirming the launch identity. If a successor is using the same idempotent result, cleanup leaves its VM alone. An AWS ResourceNotFoundException or HTTP 404 confirms successful retirement; other termination failures retain the launch record for retry.

The session lease.json now uses authority format version 2 and retains its fencing generation and checkpoint metadata after release. Existing version-1 leases and separate manifests are imported on first acquisition. Older runtimes reject the new format: drain old runtime owners before rollout, and do not roll back to a version that cannot read version 2 for migrated sessions. Session deletion clears the authoritative checkpoint reference and deletes its checkpoint, but retains a small tombstone to prevent stale state resurrection. Never independently delete authority records while a session might have an outstanding attempt.

For customer-managed networking, the pending authority record stores the activation envelope and exact launch payload, including controller and placeholder tokens, to make AWS idempotency safe across retries. This is sensitive runtime state, distinct from the credential-free metadata returned to Eve. Restrict the session prefix to the trusted runtime, keep it inaccessible to guests, use the bucket's encryption policy or artifactKmsKeyId, and apply appropriate noncurrent-version retention. Successful checkpoint/retirement clears the pending request from the current authority record; S3 version history follows your bucket retention policy.

Set onLifecycleEvent to receive structured phase timings and allowlisted AWS request ID, SDK attempt count, and total retry delay. Deadline, cancellation, and late-result outcomes are included. The callback never receives activation envelopes, request bodies, controller tokens, or raw provider errors, and observer failures do not affect session authority.

IAM boundaries

Use separate identities for the caller running eve build, the deployed runtime caller, the Lambda build role, and the optional guest execution role. Replace the account, Region, bucket, prefix, role, and image values below with your own. Tighten Resource entries further where your IAM setup supports it.

Both service roles trust Lambda. Add confused-deputy conditions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "lambda.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:TagSession"],
      "Condition": {
        "StringEquals": { "aws:SourceAccount": "123456789012" },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:lambda:us-east-1:123456789012:microvm-image:*"
        }
      }
    }
  ]
}

The build role reads the uploaded image artifact and writes build logs. It does not need permission to manage images:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::company-eve-sandboxes/eve/lambda-microvms/*"
    },
    {
      "Effect": "Allow",
      "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "arn:aws:logs:us-east-1:123456789012:*"
    }
  ]
}

The build caller manages images, passes the service roles, and reads and writes the artifact prefix:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "lambda:CreateMicrovmImage",
        "lambda:GetMicrovmImageVersion",
        "lambda:ListMicrovmImages",
        "lambda:ListMicrovmImageVersions",
        "lambda:ListManagedMicrovmImages",
        "lambda:ListManagedMicrovmImageVersions",
        "lambda:RunMicrovm",
        "lambda:GetMicrovm",
        "lambda:SuspendMicrovm",
        "lambda:ResumeMicrovm",
        "lambda:TerminateMicrovm",
        "lambda:CreateMicrovmAuthToken"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": [
        "arn:aws:iam::123456789012:role/eve-microvm-build",
        "arn:aws:iam::123456789012:role/eve-microvm-guest"
      ],
      "Condition": {
        "StringEquals": { "iam:PassedToService": "lambda.amazonaws.com" }
      }
    },
    {
      "Effect": "Allow",
      "Action": "s3:GetBucketLocation",
      "Resource": "arn:aws:s3:::company-eve-sandboxes"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::company-eve-sandboxes/eve/lambda-microvms/*"
    }
  ]
}

The runtime caller needs lifecycle, token, tag, S3 checkpoint, and execution-role pass permissions, but not image creation or build-role pass permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "lambda:RunMicrovm",
        "lambda:GetMicrovm",
        "lambda:SuspendMicrovm",
        "lambda:ResumeMicrovm",
        "lambda:TerminateMicrovm",
        "lambda:CreateMicrovmAuthToken"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::123456789012:role/eve-microvm-guest",
      "Condition": {
        "StringEquals": { "iam:PassedToService": "lambda.amazonaws.com" }
      }
    },
    {
      "Effect": "Allow",
      "Action": "s3:GetBucketLocation",
      "Resource": "arn:aws:s3:::company-eve-sandboxes"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::company-eve-sandboxes/eve/lambda-microvms/*"
    }
  ]
}

The runtime caller needs read access to image and template objects and read/write access to session manifests, leases, temporary multipart objects, and checkpoints. Keep build and runtime prefixes separate with bucket-policy conditions when your organization requires stricter deployment separation.

The optional execution role is guest-visible through IMDS. Its trust policy should use the same service principal and aws:SourceAccount, with a MicroVM source ARN:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "lambda.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:TagSession"],
      "Condition": {
        "StringEquals": { "aws:SourceAccount": "123456789012" },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:lambda:us-east-1:123456789012:microvm:*"
        }
      }
    }
  ]
}

Grant that role only the AWS APIs workload code is intentionally allowed to call, plus CloudWatch Logs permissions when runtime logging is enabled. Never grant it access to the artifact prefix or checkpoint bucket. Omitting executionRoleArn is the safest default and prevents guest AWS credentials.

See AWS's security and permissions documentation for the current action and ARN reference.

Networking

eve always attaches AWS's ALL_INGRESS connector and creates auth tokens scoped only to controller port 8080. Tokens last at most 60 minutes, are refreshed before expiry, and are never persisted. shellAccess: true additionally attaches SHELL_INGRESS; shell tokens still come from AWS's separate shell-token API.

For production, select customer-managed networking explicitly and provide the non-secret policy lane bound to each connector. Use createAwsLambdaMicrovmSandbox() to inject the trusted-host activation provider while retaining the package's default AWS API, controller, and S3 implementations:

import {
  createAwsLambdaMicrovmSandbox,
  type AwsLambdaMicrovmActivationProvider,
} from "eve-aws-lambda-microvms";

// Implemented by the trusted host; it never sends proxy credentials to the guest.
declare const activationProvider: AwsLambdaMicrovmActivationProvider;

createAwsLambdaMicrovmSandbox({
  activationProvider,
  options: {
    // required fields omitted
    networkingMode: "customer-managed",
    egressProxyCaBundlePem: process.env.EVE_EGRESS_PROXY_PUBLIC_CA_PEM,
    buildNetworkLaneId: "package-build-v1",
    buildEgressNetworkConnectorArns: [process.env.EVE_AWS_BUILD_CONNECTOR_ARN!],
    runtimeNetworkLaneId: "agent-runtime-v1",
    runtimeEgressNetworkConnectorArns: [process.env.EVE_AWS_RUNTIME_CONNECTOR_ARN!],
  },
});

egressProxyCaBundlePem accepts only one or more public X.509 CERTIFICATE PEM blocks. eve normalizes the bundle, rejects private-key or other PEM blocks, and bakes the public certificates into the AL2023 system trust store before the controlled guest root is captured. The private CA key must remain in the trusted proxy workload and must never be passed to this package, the MicroVM, the image artifact, checkpoints, or logs. Omitting the bundle preserves the existing trust store; deployments that intercept TLS must provision the public bundle before enabling that path.

This mode requires exactly one connector for image build/template prewarm and exactly one for live sessions. Both must be customer-managed connector ARNs in the configured Region and the build role's AWS account. Missing, blank, empty, multiple, cross-Region, cross-account, and AWS-managed INTERNET_EGRESS values fail before AWS clients are constructed. The connector AWS reports for each launched MicroVM is checked before controller traffic; a mismatch is terminated. Session metadata records the runtime lane ID, connector ARN, and public trust-bundle digest, and stale reattachment is terminated and replaced before exposing the guest. Changing the public bundle creates a new deterministic image/config identity. After the updated template is provisioned, the next replacement launches that image and restores the checksum-verified writable checkpoint onto it instead of silently reusing the old trust root.

Legacy mode continues to default both phases to INTERNET_EGRESS and preserves explicit connector arrays (including []) for existing consumers. Do not use legacy mode as a production no-Internet control: explicit empty-array behavior is an AWS API semantic rather than this package's strict invariant. Connectors are fixed at launch, so sandbox.setNetworkPolicy() throws for this backend. Configure VPC security groups, network ACLs, routing, and DNS on the connector instead.

The controller uploads and restores checkpoints through short-lived S3 presigned URLs. A restricted VPC path must therefore reach the bucket, normally through an S3 gateway endpoint or controlled NAT. See AWS's MicroVM networking guide.

Protocol v2 replacement lifecycle

Customer-managed sessions require an injected activation provider. Each launch receives a strict, secret-free activation envelope smaller than 4 KiB containing an opaque provider credential placeholder with its exact environment-variable placement and generations, a separate opaque localhost controller-session token, an activation identifier, and the immutable controller CA digest. Neither opaque value is a JWT or authorization assertion. JWT-shaped input is rejected; the guest never receives, requests, renews, validates, or presents an egress JWT, and placeholder possession establishes no authority.

Session metadata v2 persists the non-secret activation identifier, placeholder placement and generation, trusted-binding generation, controller CA digest, and public egress-proxy CA digest. Capture commits the verified checkpoint manifest, revokes the old trusted proxy binding, and then terminates the MicroVM. Opening the session again always launches a fresh MicroVM, installs fresh placeholder/binding generations and fresh local controller authentication, and verifies the exact singleton connector, image, protocol, controller CA digest, public proxy CA digest, placement, and generations before restoring the checksum-verified checkpoint. There is no native suspend/resume or stale-placeholder fallback.

Operations and retention

Image build logs use /aws/lambda/microvms/<image-name> unless you supply another CloudWatch target. Runtime logs use the configured runtimeLogging group. eve logs lifecycle phases and failures, but not command text or environment values. Enable CloudTrail management events for Lambda operations and S3 data events on the artifact prefix when you need an audit trail.

The package also emits OpenTelemetry spans through @opentelemetry/api; they are no-ops unless the host application configures an SDK. The eve.aws_lambda_microvm.session.create parent breaks runtime acquisition into lease, activation, aws.lambda_microvms.run_microvm, controller readiness, authentication-token, and checkpoint-restore spans. AWS call spans include request ID, attempt count, total retry delay, HTTP status, and the service-reported MicroVM start time when the SDK returns them. The eve.aws_lambda_microvm.operation.duration histogram and eve.aws_lambda_microvm.operation.count counter use only operation, outcome, Region, and networking mode where available. MicroVM IDs and image versions are trace-only; client tokens, controller credentials, activation payloads, checkpoint keys, presigned URLs, and command or environment values are never recorded. Exception events use a fixed message without the original error name, message, stack, or cause; callers still receive the original error. Metrics also activate when the host registers its provider after this package has been imported or used.

eve does not prune images or durable checkpoints. Configure S3 lifecycle rules appropriate to your retention policy for abandoned multipart uploads, temporary objects, noncurrent object versions, old checkpoint generations, and deleted applications. Do not expire the currently referenced checkpoint or template descriptor.

For failures, start with the image version stateReason, its CloudWatch build stream, and AWS's troubleshooting guide. Also review AWS's snapshot model and best practices.

Atomic AWS end-to-end tests

This repository includes an opt-in live acceptance harness. Each run creates a unique CloudFormation stack containing a private encrypted S3 bucket and a Lambda MicroVM image-build role. The application id, image, MicroVMs, S3 prefix, and image tags are unique to that stack. The runner always attempts the following teardown sequence, whether the tests pass or fail:

1. Find images tagged with the generated stack name.
2. Terminate every MicroVM using those images.
3. Delete the images and wait for them to disappear.
4. Delete the image-build CloudWatch log groups.
5. Delete all S3 objects and abort multipart uploads.
6. Delete the CloudFormation stack and wait for completion.

The live suite validates package-level image provisioning and reuse, command and file APIs, and full-filesystem restore after checkpoint termination. It then runs the deterministic Eve fixture in apps/eve-aws-lambda-microvms-e2e. That fixture sends an attachment through Eve's real staging path, loads a packaged skill, reads its sibling reference, resumes the session, forcibly terminates its MicroVM, and repeats the attachment and skill checks after replacement. It uses Eve's mock model, so no model-provider key is needed.

Select a disposable AWS account and credentials before opting in. The caller must be able to use CloudFormation, create and delete the temporary IAM role and S3 bucket, pass the build role to Lambda, and perform these MicroVM actions:

lambda:CreateMicrovmImage
lambda:CreateMicrovmAuthToken
lambda:DeleteMicrovmImage
lambda:GetMicrovm
lambda:GetMicrovmImageVersion
lambda:ListManagedMicrovmImages
lambda:ListManagedMicrovmImageVersions
lambda:ListMicrovmImages
lambda:ListMicrovmImageVersions
lambda:ListMicrovms
lambda:ListTags
lambda:ResumeMicrovm
lambda:RunMicrovm
lambda:SuspendMicrovm
lambda:TerminateMicrovm

The caller also needs logs:DescribeLogGroups and logs:DeleteLogGroup so the image-build log group does not outlive its test stack.

Inspect the exact CloudFormation template without making AWS calls:

pnpm --filter eve-aws-lambda-microvms test:aws:plan

Run the suite:

AWS_PROFILE=eve-microvm-test \
AWS_REGION=us-east-1 \
EVE_AWS_E2E_ACCOUNT_ID=123456789012 \
EVE_RUN_AWS_MICROVM_E2E=1 \
pnpm --filter eve-aws-lambda-microvms test:aws

The account id is mandatory and must match sts:GetCallerIdentity; a stale or incorrect profile is rejected before CloudFormation is called.

The generated stack name is printed before provisioning. If the process is killed in a way that prevents its finally cleanup from running, use that name to invoke the idempotent recovery path:

AWS_PROFILE=eve-microvm-test \
pnpm --filter eve-aws-lambda-microvms test:aws:cleanup -- \
  --stack eve-microvm-e2e-YYYYMMDDHHMMSS-xxxxxxxx \
  --region us-east-1

The runner catches SIGINT and SIGTERM and finishes teardown before exiting. No process can guarantee teardown after SIGKILL, machine loss, or credential revocation; the recovery command, unique tags, 15-minute MicroVM maximum duration, and CloudFormation ownership are the backstops for those cases. The first run builds a unique image and can take several minutes.

License

eve-aws-lambda-microvms is licensed under the Apache License 2.0. The package includes code derived from vercel/eve; attribution and modification details are recorded in NOTICE.