Kubernetes CronJobs for Resources Management

This package contains Kubernetes CronJobs for managing different resources, including creating snapshots and cleaning up project resources (PVCs, pods, ingresses...) automatically.

PVC Snapshot Cleaner

The pvc-snapshot-cleaner CronJob runs periodically to process PVCs in Terminating state, create snapshots, and remove finalizers.

How It Works

1. Discovers PVCs: Lists all PVCs with:

   • Label: fusion-managed: true
   • Finalizer: snapshot.fusion.io/protect
   • deletionTimestamp set (PVC is being deleted)

2. Creates Snapshots: Calls Cloud Run /projects/kube/webhook/snapshot endpoint with retry logic

3. Removes Finalizers: After successful snapshot, calls /projects/kube/webhook/remove-finalizer endpoint

4. Retry Logic: Implements exponential backoff with state persisted in PVC annotations:

   • Attempt 1: Immediate
   • Attempt 2: 30 seconds delay
   • Attempt 3: 1 minute delay
   • Attempt 4: 2 minutes delay
   • Attempt 5: 5 minutes delay
   • Attempt 6 (final): 5 minutes delay

   Retry state is persisted in PVC annotations, so it survives CronJob restarts.

Building the Docker Image

Images are automatically built and pushed by GitHub Actions workflows. For manual builds:

# From the ai-services root directory
docker build -f Dockerfile.pvc-snapshot-cleaner -t us-central1-docker.pkg.dev/PROJECT_ID/kube/pvc-snapshot-cleaner:latest .

# Push to Artifact Registry
docker push us-central1-docker.pkg.dev/PROJECT_ID/kube/pvc-snapshot-cleaner:latest

Environment Variables

• CLOUD_RUN_SERVICE_URL - URL of the Cloud Run service (e.g., https://your-service.run.app)
• KUBERNETES_WEBHOOK_TOKEN - Token for authenticating with Cloud Run endpoints

Kubernetes Configuration

The CronJob is configured via Terraform in terraform/kubernetes/main.tf. Key settings:

• Schedule: Every minute (*/1 * * * *) - Kubernetes CronJob minimum interval
• Concurrency: Allow parallel executions
• Timeout: 10 minutes per job
• Service Account: snapshot-service-account (with required RBAC permissions)

Local Development

# Set environment variables
export CLOUD_RUN_SERVICE_URL="https://your-service.run.app"
export KUBERNETES_WEBHOOK_TOKEN="your-token"

# Run directly with tsx
npx tsx packages/kube-cron-jobs/src/pvc-snapshot-cleaner.ts

PVC Soft Deletion

The pvc-soft-deletion CronJob runs periodically to mark old PVCs (older than 5 days) for deletion by setting deletionTimestamp, putting them in Terminating state. The actual deletion happens after finalizers are removed (typically by the snapshot cleaner).

Local Development

# Set environment variables (optional - defaults to 5 days)
export PVC_AGE_THRESHOLD_DAYS="5"

# Run directly with tsx
npx tsx packages/kube-cron-jobs/src/pvc-soft-deletion.ts