DataOps Data Quality L3 Construct

AWS CDK L3 Construct for deploying AWS Glue Data Quality rulesets in DataOps workflows.

Overview

This construct creates AWS Glue Data Quality rulesets for automated validation and monitoring of data in Glue Catalog tables. It supports both DQDL (Data Quality Definition Language) strings and structured rule objects for defining validation rules.

Features

• Flexible Rule Definition: Support for both raw DQDL strings and structured rule objects
• Comprehensive Rule Types: 15+ built-in rule types for common validation scenarios
• SSM Integration: Automatic publishing of ruleset metadata to SSM Parameter Store
• Project Integration: Seamless integration with DataOps project infrastructure
• Type Safety: Full TypeScript type definitions for all rule types

Installation

npm install @aws-mdaa/dataops-data-quality-l3-construct

Usage

Basic Example

import { DataOpsDataQualityL3Construct } from '@aws-mdaa/dataops-data-quality-l3-construct';
import { Stack } from 'aws-cdk-lib';

const stack = new Stack(app, 'DataQualityStack');

new DataOpsDataQualityL3Construct(stack, 'DataQuality', {
  naming: myNaming,
  domainUnit: myDomainUnit,
  projectName: 'my-dataops-project',
  rulesetConfigs: {
    'customer-quality': {
      name: 'customer-data-validation',
      description: 'Validate customer data quality',
      targetTable: {
        databaseName: 'customer_db',
        tableName: 'customers',
      },
      ruleset: [
        {
          RuleType: 'IsComplete',
          Column: 'customer_id',
        },
        {
          RuleType: 'Uniqueness',
          Column: 'email',
          Operator: '>',
          Threshold: 0.95,
        },
      ],
    },
  },
});

Using DQDL Strings

new DataOpsDataQualityL3Construct(stack, 'DataQuality', {
  naming: myNaming,
  domainUnit: myDomainUnit,
  projectName: 'my-dataops-project',
  rulesetConfigs: {
    'order-quality': {
      name: 'order-validation',
      targetTable: {
        databaseName: 'orders_db',
        tableName: 'orders',
      },
      ruleset: `Rules = [
        IsComplete "order_id",
        ColumnValues "status" in ["pending", "completed", "cancelled"],
        RowCount > 0
      ]`,
    },
  },
});

Supported Rule Types

Completeness Rules

• IsComplete: Column has no null values
• Completeness: Column meets completeness threshold

Uniqueness Rules

• IsUnique: Column has all unique values
• Uniqueness: Column meets uniqueness threshold
• IsPrimaryKey: Column is a valid primary key

Schema Rules

• ColumnExists: Column exists in table
• ColumnDataType: Column has expected data type
• ColumnLength: Column length meets criteria

Value Rules

• ColumnValues: Column values are in allowed set

Count Rules

• RowCount: Table row count meets criteria
• ColumnCount: Table column count meets criteria

Statistical Rules

• Mean: Column mean meets criteria
• StandardDeviation: Column standard deviation meets criteria

Freshness Rules

• DataFreshness: Data is recent based on timestamp column

Custom Rules

• CustomSql: Custom SQL queries for complex validation

API Reference

DataOpsDataQualityL3ConstructProps

| Property | Type | Required | Description | |----------|------|----------|-------------| | rulesetConfigs | { [key: string]: DataQualityRulesetDefinition } | Yes | Map of ruleset configurations | | projectName | string | Yes | DataOps project name for resource coordination | | naming | IMdaaResourceNaming | Yes | Naming configuration | | domainUnit | DomainUnit | Yes | Domain unit configuration |

DataQualityRulesetDefinition

| Property | Type | Required | Description | |----------|------|----------|-------------| | name | string | Yes | Unique ruleset name | | targetTable | DataQualityTargetTable | Yes | Target table configuration | | ruleset | string \| DataQualityRule[] | Yes | DQDL string or rule objects | | description | string | No | Ruleset description |

DataQualityTargetTable

| Property | Type | Required | Description | |----------|------|----------|-------------| | databaseName | string | Yes | Glue database name | | tableName | string | Yes | Glue table name | | catalogId | string | No | AWS account ID for cross-account access |

DataQualityRule

| Property | Type | Required | Description | |----------|------|----------|-------------| | RuleType | string | Yes | Type of validation rule | | Column | string | No | Column name for column-specific rules | | Operator | string | No | Comparison operator | | Threshold | number | No | Threshold value (0-1) | | Value | number | No | Numeric comparison value | | Values | (string \| number)[] | No | Allowed values list | | Where | string | No | SQL WHERE clause | | Sql | string | No | Custom SQL query | | DataType | string | No | Expected data type | | Duration | string | No | Duration for freshness checks |

License

This project is licensed under the Apache-2.0 License.