neeto-email-pipeline-nano

A Rails engine that consolidates all outbound email infrastructure for neeto apps: email prefixing, interception, rate limiting, delivery logging, and SparkPost webhook handling.

Contents

1. Architecture Overview
2. Development with Host Application
   • Engine
     • Installation
     • Configuration
   • Frontend Package
     • Installation
     • Components
3. Features
   • Email Prefixer
   • Email Interception
   • Rate Limiting
   • Email Logging
   • SparkPost Webhooks
4. Testing Locally
5. Instructions for Publishing

Architecture Overview

Every outbound email passes through the following pipeline, whether sent via deliver_later (Sidekiq) or deliver_now (inline). The engine detects the delivery path automatically and captures organization context from Sidekiq middleware or custom mail headers accordingly.

flowchart TD
    A[Mailer.deliver_later] --> B[Sidekiq Queue]
    B --> C["EmailLogMiddleware
    Generates UUID, sets Thread.current context"]
    C --> D["MailThrottleMiddleware
    Checks hourly rate limit per org"]

    D -->|Rate limited| E["Reschedule job with backoff
    Preserves email_log_uuid in job hash"]
    E --> R["EmailLogMiddleware handle_post_yield
    Creates rate_limited placeholder
    (email: nil, rate_limited_at set)"]
    R -->|"Job re-enters queue
    with same UUID"| B

    D -->|Allowed| F[Build Mail Object]

    DN[Mailer.deliver_now] --> DN1["after_action callback
    Sets X-Neeto-Organization-Id,
    X-Mailer-Class, X-Mailer-Action headers"]
    DN1 --> F

    F --> G["EmailPrefixer
    Adds [STAGING]/[DEV] prefix to subject"]
    G --> H["Interceptor
    Drops/forwards based on environment"]
    H -->|Dropped| I["LoggingInterceptor
    Mark as dropped with reason"]
    H -->|Allowed| J["LoggingInterceptor
    Deletes rate_limited placeholder (if any),
    creates per-recipient log records
    with rate_limited_at preserved"]
    J --> K[Mail Delivery]
    K --> L["LoggingObserver
    Mark queued records as handed_over"]
    L --> M{Delivery Method?}
    M -->|SparkPost| N[SparkPost Delivers Email]
    N --> O[SparkPost Webhook]
    O --> P["SparkpostWebhookJob
    Match via email_log_uuid in rcpt_meta"]
    P --> Q2[Update Status: Delivered / Bounced / Spam]
    M -->|Gmail / Outlook / SMTP| S["Final status: handed_over
    No further tracking available"]

    C -->|Exception| Q[Mark as Failed]

    style DN fill:#60a5fa,color:#000
    style DN1 fill:#60a5fa,color:#000
    style E fill:#fbbf24,color:#000
    style R fill:#fbbf24,color:#000
    style I fill:#f87171,color:#000
    style Q2 fill:#34d399,color:#000
    style S fill:#94a3b8,color:#000
    style Q fill:#f87171,color:#000

deliver_now vs deliver_later: The deliver_later path (yellow nodes) goes through Sidekiq middleware for rate limiting and context setup. The deliver_now path (blue nodes) skips Sidekiq entirely — organization context is provided via custom mail headers (X-Neeto-Organization-Id, X-Mailer-Class, X-Mailer-Action) set by an after_action callback. These headers are stripped before the email leaves the system.

Email Lifecycle Statuses

| Status | Meaning | | ---------------- | ------------------------------------------------------------------------------ | | queued | Email entered the system | | rate_limited | Temporarily held due to hourly send limit | | dropped | Blocked before sending (too many recipients, blocked content, bounced address) | | handed_over | Successfully passed to SparkPost for delivery | | delivered | SparkPost confirmed delivery to recipient's mail server | | bounced | Delivery failed (invalid address, full mailbox, policy rejection) | | spam_complaint | Recipient marked the email as spam | | failed | System error before handover (SMTP failure, mail construction error) |

Development with Host Application

Engine

Installation

1. Add the gem to your application's Gemfile:

   source "NEETO_GEM_SERVER_URL" do
     # ..existing gems
   
     gem 'neeto-email-pipeline-engine'
   end

2. Install the gem:

   bundle install

3. Mount the engine in config/routes.rb:

   mount NeetoEmailPipelineEngine::Engine, at: "/neeto_email_pipeline"

4. Copy and run migrations:

   bundle exec rails neeto_email_pipeline_engine:install:migrations
   bundle exec rails db:migrate

Configuration

constants.yml

Add the following to your config/constants.yml:

defaults: &defaults
  email_pipeline:
    email_logging_enabled: true
    retention_days: 30

development:
  <<: *defaults

test:
  <<: *defaults

staging:
  <<: *defaults

production:
  <<: *defaults

secrets.yml (via Vault)

Add SparkPost webhook credentials to config/secrets.yml:

defaults: &defaults
  email_pipeline:
    sparkpost_webhook_username: <%= ENV['...'] %>
    sparkpost_webhook_password: <%= ENV['...'] %>

Scheduled Jobs

Add the purge job to config/scheduled_jobs.yml:

purge_old_email_logs:
  cron: "0 3 * * *"
  class: NeetoEmailPipelineEngine::PurgeOldEmailLogsJob
  description: "Purge email logs older than retention period"

Frontend Package

Installation

1. Add the package:

   yarn add @bigbinary/neeto-email-pipeline-frontend

2. Install peer dependencies (neetoui, neeto-molecules, neeto-commons-frontend, neeto-filters-frontend, etc.) if not already present.

Components

The package exports the EmailLogs component:

import { EmailLogs } from "@bigbinary/neeto-email-pipeline-frontend";

<EmailLogs
  canViewEmailLogs={true}
  breadcrumbs={[
    { text: "Admin panel", link: "/admin" },
    { text: "Email Logs" },
  ]}
  helpUrl="https://neetocalhelp.neetokb.com/p/a-8ed45325"
/>;

Props:

| Prop | Type | Required | Description | | ------------------ | --------- | -------- | ---------------------------------------------------------------- | | canViewEmailLogs | boolean | Yes | Permission gate. If false, nothing renders. | | breadcrumbs | array | Yes | Breadcrumb trail for the Header component. | | helpUrl | string | No | URL for the help article. Renders a help icon next to the title. |

Features

Email Prefixer

Adds environment prefix to email subjects in non-production environments (e.g., [STAGING], [DEVELOPMENT]). Also supports per-organization custom prefixes via the email_prefix field on the Organization model.

Requires: Every mailer must be called with organization_id in params:

UserMailer.with(organization_id: org.id).welcome_email.deliver_later

Email Interception

The Interceptor controls which emails are actually sent based on the environment:

Staging/Development:

• Forwards emails to configured addresses (mail_interceptor.forward_emails_to in secrets)
• Whitelists specific addresses (mail_interceptor.whitelisted_emails in secrets)
• Intercepts emails matching ^cpt.*bigbinary\.com$

Production:

• Filters out emails to @example.com and @example.net

All environments — emails are dropped when:

• More than 10 unique recipients across To, CC, and BCC
• Subject or body contains blocked content patterns (depop, dep0p)
• All recipients have bounced 2+ times (checked via NeetoTower API)

The drop reason is recorded in the email log's status_detail field.

Note: If the NeetoTower API is unavailable (timeout, connection refused, 5xx error), the engine treats the bounce count as 0 and allows the email to be sent. Errors are reported to Honeybadger for visibility.

Rate Limiting

The MailThrottleMiddleware enforces per-organization hourly email send limits:

• Configured via organization.max_hourly_emails.
• Can be globally overridden with GLOBAL_HOURLY_EMAIL_LIMIT_FOR_ALL_ORGS env var
• Rate limiting is enabled by default in production, controlled by RATE_LIMIT_ENGINE_ENABLED env var
• When rate limited, the job is rescheduled with exponential backoff (60s to 30min)
• A notification is sent to the organization (deduplicated per 10-minute window)

Email Logging

When email_logging_enabled is true, every outbound email is tracked with:

• Per-recipient records: One row per recipient (to/cc/bcc) with email, subject, from, delivery method
• Email body capture: HTML body stored for preview in the admin UI
• Full lifecycle tracking: queued → rate_limited → handed_over → delivered/bounced/spam_complaint
• Rate limit awareness: Records carry rate_limited_at timestamp when the email was throttled
• Automatic purge: PurgeOldEmailLogsJob deletes records older than retention_days

deliver_now support: Emails sent via deliver_now are also logged. Since these emails bypass Sidekiq middleware, the engine reads organization context from custom mail headers (X-Neeto-Organization-Id, X-Mailer-Class, X-Mailer-Action) that are set by an after_action callback in the mailer. The headers are stripped before delivery so they never reach the recipient.

SparkPost Webhooks

The engine provides a webhook endpoint for SparkPost delivery events:

Endpoint: POST /neeto_email_pipeline/webhooks/sparkpost/events

Authentication: HTTP Basic Auth using credentials from vault.

Handled events:

• delivery → status: delivered
• bounce, out_of_band, policy_rejection → status: bounced
• spam_complaint → status: spam_complaint

Correlation: The LoggingInterceptor injects email_log_uuid into the X-MSYS-API header metadata. SparkPost echoes this back in webhook events via rcpt_meta.email_log_uuid, which the SparkpostWebhookJob uses to find the exact recipient record.

SparkPost Setup

1. Go to SparkPost → Webhooks → Create Webhook
2. Target URL: https://app.yourproduct.com/neeto_email_pipeline/webhooks/sparkpost/events
3. Authentication: HTTP Basic with username/password from your vault config
4. Events to subscribe:
   • Delivery: delivery
   • Bounce: bounce, out_of_band, policy_rejection
   • Spam: spam_complaint

Testing Locally

Email Interception

To test email forwarding in development, add the following to config/secrets.yml:

development:
  mail_interceptor:
    forward_emails_to: "[email protected]"
    whitelisted_emails: ""

Emails will be forwarded to the configured address instead of the original recipient.

Rate Limiting

Rate limiting is disabled in development by default. To enable:

1. Set RATE_LIMIT_ENGINE_ENABLED=true in your environment

2. Set the org's hourly limit:

   Organization.first.update!(max_hourly_emails: 5)

3. Send more emails than the limit to trigger throttling

Email Logging

1. Enable logging in config/constants.yml:

   development:
     email_pipeline:
       email_logging_enabled: true
       retention_days: 30

2. Restart Sidekiq (the engine registers interceptors/observers at boot)

3. Send an email from the app and check the Email Logs page in admin panel

SparkPost Webhooks (Local)

To test webhooks locally, use a tunneling service:

1. Start a tunnel to your local server (e.g., tunn.dev, ngrok)

2. Use the tunnel URL as the webhook target in SparkPost

3. Add webhook credentials to config/secrets.yml:

   development:
     email_pipeline:
       sparkpost_webhook_username: "your-username"
       sparkpost_webhook_password: "your-password"

4. Use a subdomain that maps to a valid organization (e.g., spinkart.tunn.dev)

To send actual emails from your local, add the following lines to your development.rb

config.action_mailer.delivery_method = :smtp
config.action_mailer.perform_deliveries = true
config.action_mailer.raise_delivery_errors = true

You will also need to add the following env variables:

SPARKPOST_USERNAME=xxx
SPARKPOST_PASSWORD=xxx
SPARKPOST_DOMAIN=xxx # a verified Sparkpost domain. Unverified domains will raise errors.

Instructions for Publishing

Consult the building and releasing packages guide for details on how to publish.