qvdjs

v0.9.0

Published

a month ago

Library for reading/writing Qlik Sense/QlikView QVD files in Javascript.

Downloads

0High
0Medium
0Low

mountaindude

qvd qlik qlik sense qlikview

qvdjs

Utility library for reading/writing Qlik Sense and QlikView (QVD) files in JavaScript/Node.js

⚠️ Important Disclaimer

This library is based on reverse engineering of the QVD file format, as the format is not publicly documented by Qlik. While extensive effort has been made to understand and implement the format correctly, there may be incorrect assumptions or interpretations of the file structure.

Comprehensive testing has been performed to ensure that QVD files created or modified by qvdjs are valid and can be loaded by Qlik Sense and QlikView without errors. However, users should:

Test thoroughly with their specific use cases
Validate output files in their Qlik environment
Report any issues or inconsistencies discovered

The library works with real-world QVD files and maintains compatibility with Qlik products, but it is an independent, community-driven implementation.

The qvdjs library provides a simple API for reading/writing Qlik View Data (QVD) files in JavaScript.

Using this library, it is possible to parse the binary QVD file format and convert it to a JavaScript object structure and vice versa. The library is written to be used in a Node.js environment exclusively.

qvdjs

Install

qvdjs is a Node.js module available through npm. The recommended way to install and maintain qvdjs as a dependency is through the Node.js Package Manager (NPM). Before installing this library, download and install Node.js.

You can get qvdjs using the following command:

npm install qvdjs --save

Module Format Support: This library is published as a dual ESM/CJS package, providing full compatibility with both modern ES modules and traditional CommonJS environments:

✅ ESM (ES Modules): Native import statements in Node.js and modern bundlers
✅ CommonJS: Traditional require() for compatibility with older Node.js projects
✅ Bundlers: Works with Webpack, Vite, Rollup, esbuild, and other modern build tools

Usage Examples:

// ESM (ES Modules) - Modern Node.js and TypeScript
import {QvdDataFrame} from 'qvdjs';

// CommonJS - Traditional Node.js
const {QvdDataFrame} = require('qvdjs');

The package automatically provides the correct format based on your project's configuration.

Usage

Below is a quick example how to use qvdjs.

import {QvdDataFrame} from 'qvdjs';

const df = await QvdDataFrame.fromQvd('path/to/file.qvd');
console.log(df.head(5));

The above example loads the qvdjs library and parses an example QVD file. A QVD file is typically loaded using the static QvdDataFrame.fromQvd function of the QvdDataFrame class itself. After loading the file's content, numerous methods and properties are available to work with the parsed data.

Lazy Loading

For large QVD files, you can load only a specific number of rows to improve performance and reduce memory usage. The library implements lazy loading - it reads only the necessary portions of the file from disk, not the entire file.

import {QvdDataFrame} from 'qvdjs';

// Load only the first 1000 rows
const df = await QvdDataFrame.fromQvd('path/to/file.qvd', {maxRows: 1000});
console.log(df.shape); // [1000, numberOfColumns]

How it works:

The library reads only the header, symbol table, and the first N rows from the index table
This provides significant memory savings and faster loading times for large files

Important: Symbol Table and High-Cardinality Fields

The QVD format stores data in two parts:

Symbol table: Contains ALL unique values for ALL fields (must be fully loaded)
Index table: Contains row-by-row indices into the symbol table (can be partially loaded with maxRows)

⚠️ Performance Impact of High-Cardinality Fields:

If your QVD file contains fields with many unique values (high cardinality), such as:

Unique IDs (OrderID, TransactionID, UUID)
Timestamps with millisecond precision
Unique text fields

The symbol table can become very large and must be read completely even when using maxRows. This means:

Small symbol table (fields with reusable values): Fast loading regardless of file size
- Example: A 500MB file with only 100KB symbol table loads in ~100ms for 5000 rows ✅
Large symbol table (fields with unique values per row): Slow loading even with maxRows
- Example: A 500MB file with 266MB symbol table takes ~6 seconds for any row count ❌

To check your QVD's symbol table size, look at the NoOfSymbols in field metadata - values close to the total row count indicate high cardinality.

When lazy loading works best:

Previewing data from very large QVD files without loading the entire file into memory
Files where most fields have reusable values (low cardinality)
Data exploration and schema inspection of large datasets
Faster loading times when you only need a subset of the data

Performance Optimizations

The library includes intelligent symbol table parsing that dramatically improves performance when loading partial data with maxRows:

Smart Symbol Loading:

When you specify maxRows, the library:

Analyzes which symbols are actually needed for the requested rows
Parses only those symbols from the symbol table
Skips parsing unused symbols entirely (not just filtering after parsing)

Performance Impact:

Compared to parsing all symbols regardless of maxRows:

~10x faster load times - Loading 5,000 rows from a 20M row file: 8.3 seconds → 0.9 seconds
~10x less memory usage - Same operation: 1,616 MB → 180 MB
Native or better efficiency - Memory overhead reduced from 6.0x to 0.7x of raw data size

Real-World Example:

// File: orders_20m.qvd (20M rows, 266MB symbol table)
const df = await QvdDataFrame.fromQvd('orders_20m.qvd', {maxRows: 5000});
// Load time: ~0.9 seconds
// Memory usage: ~180 MB
// Only 17,228 out of millions of symbols parsed

Key Benefits:

Much faster previews of large QVD files
Lower memory footprint for data exploration
Efficient handling of files with large symbol tables
Automatic optimization - no configuration needed

This optimization is particularly effective for files with many unique values (high cardinality) where the symbol table is large but you only need to preview a small portion of the data.

QVD File Size Limitations

Simple Explanation:

Node.js has memory limits that affect how large QVD files you can work with. By default, Node.js can use up to about 4GB of memory. This means if you try to load a very large QVD file, you might run out of memory and get an error. Think of it like trying to open a very large document on a computer with limited RAM - if the document is too big, it won't open.

What happens when files are too large:

When opening large files: If a QVD file exceeds available memory, Node.js will throw an out-of-memory error (typically "JavaScript heap out of memory" or "FATAL ERROR: Reached heap limit"). The process will crash before completing the file load.
When saving large files: Writing very large QVD files can similarly exhaust memory during symbol table and index table construction, causing the same out-of-memory errors before the file is written to disk.
Performance degradation: Even before running out of memory completely, you may notice significant slowdowns, high memory usage, and system swapping as files approach memory limits.

The good news is that Node.js memory limits can be increased (though there will always be some limit), and the actual file size you can handle depends on your data.

The most common question at this point is usually:

"How large of a QVD file can I work with using qvdjs?"

There is unfortunately no simple answer to this question, as it very much depends on the characteristics of what data is inside the QVD file. See below for more details.

Technical Details:

The maximum QVD file size you can handle with qvdjs depends on several factors and there is no single fixed limit:

Node.js Memory Limits: By default, Node.js limits heap memory to approximately 4GB (varies by architecture and Node.js version). The library automatically detects your configured heap size and scales its safety limits accordingly. You can increase heap size using the --max-old-space-size flag (e.g., node --max-old-space-size=16384 script.js for 16GB), and qvdjs will automatically allow larger files.
For larger heap configurations, you can also adjust the memorySafetyFactor option (default 0.3 = 30%) to make more efficient use of available memory:
```
const df = await QvdDataFrame.fromQvd('large-file.qvd', {
  memorySafetyFactor: 0.5, // Use 50% of heap instead of default 30%
});
```
To phrase it differently: There is no magic here - viewing a 40 GB QVD on a laptop with 24 GB RAM will not work. That laptop may in fact struggle with QVD files larger than 4-6 GB depending on data characteristics - or happily work with 10+ GB files if the data is very friendly.
Data Characteristics: The actual memory consumption depends heavily on what's inside your QVD:
- Field Cardinality: Files with high-cardinality fields (many unique values per field, like unique IDs or timestamps) require more memory for symbol tables. This is usually the biggest factor, at least if there are many rows too.
- Number of Rows: More rows mean larger index tables in memory
- Number of Fields: More columns increase overall memory requirements
- Data Types: String data generally uses more memory than numeric data
Operation Type: Reading typically uses less memory than writing, especially when using lazy loading (maxRows option). Writing requires building complete symbol and index tables in memory.
Practical Guidance:
- For typical business data with moderate cardinality, files up to 1-2GB usually work well with default Node.js settings
- High-cardinality data (unique values in most rows) may limit you to smaller files (hundreds of MB)
- With increased heap (e.g., 16GB+), you can handle proportionally larger files by adjusting memorySafetyFactor
- Use lazy loading (maxRows option) when possible to reduce memory footprint when reading
- Monitor memory usage with tools like process.memoryUsage() for your specific use cases
- Consider processing large datasets in chunks or using streaming approaches if you hit memory limits. Clever things can be done by doing multiple passes over the file instead of loading everything at once.

If you consistently work with very large QVD files, consider increasing Node.js memory limits (with matching memorySafetyFactor adjustment) or splitting your data into multiple smaller QVD files.

Why Safety Limits Exist

The library implements dynamic safety limits to prevent catastrophic crashes. Without these limits:

Your application will crash hard - Node.js terminates with FATAL ERROR: Reached heap limit when attempting to load files that are too large
No error handling is possible - JavaScript try-catch blocks cannot intercept out-of-memory (OOM) crashes at the V8 engine level
The entire process dies - Not just the QVD operation, but your entire application terminates ungracefully

The safety limits prevent these crashes by checking available memory before attempting to load files, throwing graceful QvdValidationError exceptions that you can catch and handle. While the multi-tier safety system may seem complex, it ensures your application stays running and provides helpful error messages with recommendations (like using maxRows parameter or increasing heap size) instead of cryptic fatal errors.

Example without safety limits:

// Process crashes with no chance to recover
const df = await QvdDataFrame.fromQvd('huge-file.qvd'); // 💥 FATAL ERROR
// Your application is now terminated

Example with safety limits:

try {
  const df = await QvdDataFrame.fromQvd('huge-file.qvd');
} catch (error) {
  if (error.name === 'QvdValidationError') {
    console.log('File too large, trying with maxRows:', error.context.recommendedMaxRows);
    // ✅ Your application continues running
  }
}

For more technical details about the memory safety system, including specific thresholds and the four-tier protection model, see docs/DYNAMIC_SAFETY_LIMITS.md. For practical examples, see docs/examples/heap-scaling-example.md.

Progress Tracking for Large File Writes

When writing large QVD files, the toQvd() operation can take significant time. The library provides optional progress callbacks to track the write operation in real-time:

import {QvdDataFrame} from 'qvdjs';

const df = await QvdDataFrame.fromDict({
  columns: ['ID', 'Name', 'Value'],
  data: largeDataArray, // e.g., 100,000+ rows
});

await df.toQvd('output.qvd', {
  onProgress: (progress) => {
    console.log(`${progress.stage}: ${progress.percent}% complete`);
  },
});

Progress Object Properties:

The progress parameter passed to the onProgress callback contains the following properties:

stage (string): Current operation stage being performed
current (number): Current progress value (e.g., rows processed, columns completed)
total (number): Total progress value for the current stage
percent (number): Progress percentage (0-100) calculated as (current / total) * 100

Progress stages:

symbol-table: Building unique value tables for each column
index-table: Processing all data rows and creating index mappings
header: Generating XML header metadata
write: Writing data to disk

Performance optimizations:

The library uses optimized algorithms for large dataset processing:

Single-pass symbol table building: All columns are processed in one pass through the data (previously required one pass per column)
Map-based lookups: O(1) symbol index lookups instead of O(n) findIndex operations
Reduced algorithmic complexity: From O(n×m×s) to O(n×m) where n=rows, m=columns, s=symbols

These optimizations can reduce write times by 80-90% for large datasets (100K+ rows), compared to earlier versions of the library.

Working with Metadata

The library provides full access to QVD file and field metadata:

import {QvdDataFrame} from 'qvdjs';

// Load QVD file
const df = await QvdDataFrame.fromQvd('path/to/file.qvd');

// Access file-level metadata
console.log(df.fileMetadata.tableName);
console.log(df.fileMetadata.createUtcTime);
console.log(df.fileMetadata.noOfRecords);

// Access field-level metadata
const fieldMeta = df.getFieldMetadata('ProductKey');
console.log(fieldMeta.comment);
console.log(fieldMeta.numberFormat);
console.log(fieldMeta.tags);

// Get all field metadata
const allFields = df.getAllFieldMetadata();
allFields.forEach((field) => {
  console.log(`${field.fieldName}: ${field.noOfSymbols} symbols`);
});

// Modify metadata (only modifiable properties can be changed)
df.setFileMetadata({
  tableName: 'UpdatedProducts',
  comment: 'Modified product data',
});

df.setFieldMetadata('ProductKey', {
  comment: 'Primary key for products',
  tags: {String: ['$key', '$numeric']},
});

// Metadata is preserved when writing
await df.toQvd('path/to/output.qvd');

Security Considerations

The library includes built-in protection against path traversal attacks.

Default Security Behavior:

By default, all file operations are restricted to the current working directory (CWD) and its subdirectories. This means:

✅ Files within CWD can be accessed: ./data/file.qvd or data/file.qvd
❌ Files outside CWD are blocked: /etc/passwd or ../../../sensitive.qvd
❌ Path traversal attempts are detected and blocked

This default behavior protects against path traversal attacks without requiring additional configuration.

Custom Directory Restriction:

You can explicitly specify a different base directory using the allowedDir option:

import {QvdDataFrame} from 'qvdjs';

// Restrict reading to a specific directory
const allowedDataDir = '/var/data/qvd-files';
const df = await QvdDataFrame.fromQvd('reports/sales.qvd', {
  allowedDir: allowedDataDir,
});

// Restrict writing to a specific directory
const allowedOutputDir = '/var/output';
await df.toQvd('processed/sales-filtered.qvd', {
  allowedDir: allowedOutputDir,
});

Security Features:

Path Normalization: All paths are automatically normalized using path.resolve() to eliminate .. and . segments
Null Byte Protection: Detects and blocks null byte injection attempts
Default CWD Restriction: By default, file operations are restricted to the current working directory (CWD) and its subdirectories to prevent path traversal attacks
Custom Directory Restriction: Optional allowedDir parameter allows you to specify a different base directory
Security Errors: Throws QvdSecurityError with detailed context when security violations are detected

Best Practices:

Understand default security: Files are restricted to CWD by default - no additional configuration needed for basic protection
Use explicit allowedDir in production: When your application's CWD differs from your data directory, specify an explicit allowedDir for user-provided file paths
Validate user input: Even with built-in protections, validate and sanitize any user-provided paths
Principle of least privilege: Use the most restrictive allowedDir possible for your use case
Monitor security errors: Log and monitor QvdSecurityError exceptions as they may indicate attack attempts

Examples:

import {QvdDataFrame, QvdSecurityError} from 'qvdjs';

// Example 1: Default behavior (restricted to CWD)
// This is safe by default - no path traversal possible
try {
  const df = await QvdDataFrame.fromQvd('data/file.qvd'); // ✅ Works (within CWD)
  const df2 = await QvdDataFrame.fromQvd('../../../etc/passwd'); // ❌ Throws QvdSecurityError
} catch (error) {
  if (error instanceof QvdSecurityError) {
    console.error('Path traversal blocked:', error.message);
  }
}

// Example 2: Custom allowedDir for specific use cases
try {
  const df = await QvdDataFrame.fromQvd(userProvidedPath, {
    allowedDir: '/safe/directory',
  });
  // Process data...
} catch (error) {
  if (error instanceof QvdSecurityError) {
    console.error('Security violation detected:', error.message);
    console.error('Context:', error.context);
    // Log security incident
  } else {
    throw error;
  }
}

QVD File Format

The QVD file format is a binary file format that is used by QlikView to store data. The format is proprietary. However, the format is well documented and can be parsed without the need of a QlikView installation. In fact, a QVD file consists of three parts: a XML header, and two binary parts, the symbol and the index table. The XML header contains meta information about the QVD file, such as the number of data records and the names of the fields. The symbol table contains the actual distinct values of the fields. The index table contains the actual data records. The index table is a list of indices which point to values in the symbol table.

XML Header

The XML header contains meta information about the QVD file. The header is always located at the beginning of the file and is in human readable text format. The header contains information about the number of data records, the names of the fields, and the data types of the fields.

Symbol Table

The symbol table contains the distinct/unique values of the fields and is located directly after the XML header. The order of columns in the symbol table corresponds to the order of the fields in the XML header. The length and offset of the symbol sections of each column are also stored in the XML header.

Important: The offset values in the XML header are relative to the start of the symbol table section, not absolute file positions. For example, if a field has Offset=1143, this means its symbol data starts 1143 bytes after the symbol table section begins (which itself starts immediately after the XML header ends).

Each symbol section consists of the unique symbols of the respective column. The type of a single symbol is determined by a type byte prefixed to the respective symbol value. The following type of symbols are supported:

| Code | Type | Description | | ---- | ------------ | --------------------------------------------------------------------------------------------- | | 1 | Integer | signed 4-byte integer (little endian) | | 2 | Float | signed 8-byte IEEE floating point number (little endian) | | 4 | String | null terminated string | | 5 | Dual Integer | signed 4-byte integer (little endian) followed by a null terminated string | | 6 | Dual Float | signed 8-byte IEEE floating point number (little endian) followed by a null terminated string |

Index Table

After the symbol table, the index table follows. The index table contains the actual data records. The index table contains binary indices that refrences to the values of each row in the symbol table. The order of the columns in the index table corresponds to the order of the fields in the XML header. Hence, the index table does not contain the actual values of a data record, but only the indices that point to the values in the symbol table.

Empty QVD Files

QVD files can be empty in the sense that they contain zero data rows while still maintaining valid field definitions and metadata. This is a valid use case in Qlik applications where table structures need to be preserved even when no data is available.

Characteristics of Empty QVD Files:

NoOfRecords: Set to 0 in the XML header
RecordByteSize: Set to 1 (following Qlik Sense's convention)
Fields: Field definitions are present and complete with metadata
Symbol Table: Each field has zero symbols (NoOfSymbols=0, Offset=0, Length=0)
Index Table: Empty with Length=0
BitWidth: Can vary per field (typically 0 or 8)

Example Empty QVD XML Header:

<QvdTableHeader>
  <TableName>EmptyTable</TableName>
  <Fields>
    <QvdFieldHeader>
      <FieldName>Country</FieldName>
      <BitOffset>0</BitOffset>
      <BitWidth>0</BitWidth>
      <Bias>0</Bias>
      <NoOfSymbols>0</NoOfSymbols>
      <Offset>0</Offset>
      <Length>0</Length>
    </QvdFieldHeader>
  </Fields>
  <RecordByteSize>1</RecordByteSize>
  <NoOfRecords>0</NoOfRecords>
  <Offset>0</Offset>
  <Length>0</Length>
</QvdTableHeader>

Working with Empty QVDs:

import {QvdDataFrame} from 'qvdjs';

// Create an empty QVD with field structure but no data
const emptyDf = new QvdDataFrame(
  [], // No data rows
  ['Country', 'Year', 'Sales'], // Column definitions
);

// Set metadata
emptyDf.setFileMetadata({
  tableName: 'EmptyTable',
  comment: 'Template table structure',
});

// Write empty QVD (compatible with Qlik Sense)
await emptyDf.toQvd('empty.qvd');

// Read it back
const loadedDf = await QvdDataFrame.fromQvd('empty.qvd');
console.log(loadedDf.shape); // [0, 3] - zero rows, three columns
console.log(loadedDf.columns); // ['Country', 'Year', 'Sales']

Empty QVDs are fully supported for both reading and writing, maintaining compatibility with files created by Qlik Sense and QlikView.

API Documentation

QvdDataFrame

The QvdDataFrame class represents the data frame stored inside of a finally parsed QVD file. It provides a high-level abstraction access to the QVD file content. This includes meta information as well as access to the actual data records.

| Property | Type | Description | | -------------- | ---------- | ------------------------------------------------------------------------------------------------------------------ | | shape | number[] | The shape of the data table. The first element is the number of rows, the second element is the number of columns. | | data | any[][] | The actual data records of the QVD file. The first dimension represents the single rows. | | columns | string[] | The names of the fields that are contained in the QVD file. | | metadata | object | The complete metadata object from the QVD file header, or null if not loaded from a QVD file. | | fileMetadata | object | File-level metadata from the QVD header (qvBuildNo, tableName, createUtcTime, etc.). |

`static fromQvd(path: string, options?: object): Promise<QvdDataFrame>`

The static method QvdDataFrame.fromQvd loads a QVD file from the given path and parses it. The method returns a promise that resolves to a QvdDataFrame instance.

Parameters:

path (string): The path to the QVD file.
options (object, optional): Loading options
- maxRows (number, optional): Maximum number of rows to load. If not specified, all rows are loaded. This is useful for loading only a subset of data from large QVD files to improve performance and reduce memory usage.
- allowedDir (string, optional): Base directory for file access validation. Defaults to current working directory (CWD). The file path must resolve to a location within this directory to prevent path traversal attacks. Set to a specific directory in production environments with user-provided paths.
- memorySafetyFactor (number, optional): Memory safety factor (0.0-1.0). Default is 0.3 (30%). Determines what percentage of available memory (or V8 heap limit, whichever is smaller) can be used. Increase this (e.g., to 0.5 or 0.7) when running with larger heap sizes via --max-old-space-size to allow processing of proportionally larger files.

Example:

// Load all rows (default behavior)
const df = await QvdDataFrame.fromQvd('path/to/file.qvd');

// Load only the first 1000 rows
const dfLazy = await QvdDataFrame.fromQvd('path/to/file.qvd', {maxRows: 1000});

// Load with security restriction (recommended for production)
const dfSecure = await QvdDataFrame.fromQvd('reports/sales.qvd', {
  allowedDir: '/var/data/qvd-files',
});

// Load with increased memory usage for large heap configurations
const dfLarge = await QvdDataFrame.fromQvd('large-file.qvd', {
  memorySafetyFactor: 0.5, // Use 50% of heap instead of default 30%
});

`static fromDict(dict: object): Promise<QvdDataFrame>`

The static method QvdDataFrame.fromDict constructs a data frame from a dictionary. The dictionary must contain the columns and the actual data as properties. The columns property is an array of strings that contains the names of the fields in the QVD file. The data property is an array of arrays that contains the actual data records. The order of the values in the inner arrays corresponds to the order of the fields in the QVD file.

`head(n: number): QvdDataFrame`

The method head returns the first n rows of the data frame.

`tail(n: number): QvdDataFrame`

The method tail returns the last n rows of the data frame.

`rows(...args: number): QvdDataFrame`

The method rows returns a new data frame that contains only the specified rows.

`at(row: number, column: string): any`

The method at returns the value at the specified row and column.

`select(...args: string): QvdDataFrame`

The method select returns a new data frame that contains only the specified columns.

`toDict(): Promise<object>`

The method toDict returns the data frame as a dictionary. The dictionary contains the columns and the actual data as properties. The columns property is an array of strings that contains the names of the fields in the QVD file. The data property is an array of arrays that contains the actual data records. The order of the values in the inner arrays corresponds to the order of the fields in the QVD file.

`toQvd(path: string, options?: object): Promise<void>`

The method toQvd writes the data frame to a QVD file at the specified path.

Parameters:

path (string): The path where the QVD file should be written.
options (object, optional): Writing options
- allowedDir (string, optional): Base directory for file write validation. Defaults to current working directory (CWD). The file path must resolve to a location within this directory to prevent path traversal attacks. Set to a specific directory in production environments with user-provided paths.
- onProgress (function, optional): Progress callback function for tracking write operations. Receives progress updates during symbol table building, index table building, header generation, and data writing.

Progress Callback:

The onProgress callback receives an object with the following properties:

stage (string): Current operation stage - 'symbol-table', 'index-table', 'header', or 'write'
current (number): Current progress value (e.g., rows processed, columns completed)
total (number): Total progress value
percent (number): Progress percentage (0-100)

This is particularly useful for large QVD files where write operations can take significant time.

Examples:

// Write to file (default behavior)
await df.toQvd('output/data.qvd');

// Write with security restriction (recommended for production)
await df.toQvd('processed/data.qvd', {
  allowedDir: '/var/output/qvd-files',
});

// Write with progress tracking for large files
await df.toQvd('large-output.qvd', {
  onProgress: (progress) => {
    console.log(`${progress.stage}: ${progress.percent}% (${progress.current}/${progress.total})`);
  },
});

// Write with detailed progress bar
await df.toQvd('data.qvd', {
  onProgress: (progress) => {
    const bar = '█'.repeat(Math.floor(progress.percent / 2)) + '░'.repeat(50 - Math.floor(progress.percent / 2));
    process.stdout.write(`\r[${progress.stage}] ${bar} ${progress.percent}%`);
    if (progress.current === progress.total) console.log(' ✓');
  },
});

`getFieldMetadata(fieldName: string): object | null`

The method getFieldMetadata returns the metadata for a specific field/column from the QVD header. Returns null if the field is not found or metadata is not available.

The returned object contains:

fieldName: Name of the field
bitOffset: Bit offset in the index table
bitWidth: Bit width in the index table
bias: Bias value for index calculation
noOfSymbols: Number of unique symbols/values
offset: Byte offset in the symbol table
length: Byte length in the symbol table
comment: Field comment (modifiable)
numberFormat: Number format settings (modifiable)
tags: Field tags (modifiable)

Note: Properties like offset, length, bitOffset, bitWidth, bias, and noOfSymbols are immutable and relate to internal data storage.

`getAllFieldMetadata(): object[]`

The method getAllFieldMetadata returns an array of metadata objects for all fields in the data frame. Each object has the same structure as returned by getFieldMetadata.

`setFileMetadata(metadata: object): void`

The method setFileMetadata allows modifying file-level metadata. Only modifiable properties are updated; immutable properties related to data storage are ignored.

Modifiable properties:

qvBuildNo: QlikView build number
creatorDoc: Document GUID that created the QVD
createUtcTime: Creation timestamp
sourceCreateUtcTime: Source creation timestamp
sourceFileUtcTime: Source file timestamp
sourceFileSize: Source file size
staleUtcTime: Stale timestamp
tableName: Table name
compression: Compression method
comment: Table comment
encryptionInfo: Encryption information
tableTags: Table tags
profilingData: Profiling data
lineage: Data lineage information

Immutable properties (cannot be modified):

noOfRecords: Number of records
recordByteSize: Record byte size
offset: Byte offset in file
length: Byte length in file

`setFieldMetadata(fieldName: string, metadata: object): void`

The method setFieldMetadata allows modifying field-level metadata for a specific field. Only modifiable properties are updated; immutable properties related to data storage are ignored.

Modifiable properties:

comment: Field comment/description
numberFormat: Number format settings (Type, nDec, UseThou, Fmt, Dec, Thou)
tags: Field tags (typically used for field classification)

Immutable properties (cannot be modified):

offset: Byte offset in symbol table
length: Byte length in symbol table
bitOffset: Bit offset in index table
bitWidth: Bit width in index table
bias: Bias value
noOfSymbols: Number of symbols

Documentation

Comprehensive documentation is available to help you understand, use, and contribute to qvdjs:

For Users

README.md (this file) - Quick start guide and API reference
QVD_FORMAT.md - Complete QVD file format specification
- Binary structure details
- Symbol type encodings
- Bit packing algorithms
- Example file breakdowns

For Contributors

CONTRIBUTING.md - How to contribute to the project
- Development setup
- Code style guidelines
- Commit message conventions
- Pull request process
- Bug reporting and feature requests
DEVELOPMENT.md - Technical development guide
- Architecture overview
- Implementation details
- Design patterns
- Performance considerations
- Error handling strategies
- Debugging tips
ARCHITECTURE.md - High-level architecture
- Component diagrams
- Class relationships
- Data flow visualization
- Design decisions and rationale
- Extension points
- Future considerations
docs/TESTING.md - Comprehensive testing guide
- Testing philosophy and strategy
- How to write unit and integration tests
- Performance testing guidelines
- Coverage targets
- Multi-platform testing setup

Quick Links by Task

| I want to... | See... | | ----------------------- | ------------------------------------------------------------------------- | | Use the library | README.md - Usage section | | Understand QVD format | QVD_FORMAT.md | | Report a bug | CONTRIBUTING.md | | Suggest a feature | CONTRIBUTING.md | | Contribute code | CONTRIBUTING.md + DEVELOPMENT.md | | Understand architecture | ARCHITECTURE.md | | Write tests | docs/TESTING.md | | Debug an issue | DEVELOPMENT.md |

Testing

qvdjs has comprehensive test coverage with automated multi-platform testing. For detailed information about the testing infrastructure, including:

Test architecture and coverage breakdown
Multi-platform support (Windows, macOS, Linux)
Security testing approach
Self-hosted runner setup
Performance benchmarking

See the Testing Documentation in the docs/ directory.

Quick links:

Testing Summary - Executive overview and quick start
Complete Design - Full technical specification

Running Tests

# Run all tests
npm test

# Run tests with coverage
npm run coverage

# Run specific test file
npm test -- __tests__/reader.test.js

Current test coverage: 91.7% across 95+ tests covering unit, integration, and error handling scenarios.

Contributing

We welcome contributions! When contributing to this project, please follow the Conventional Commits specification for commit messages. This enables automatic version management and changelog generation.

For detailed information about the release process, see RELEASING.md.

Contributors

Göran Sander and Ptarmigan Labs - Creator of qvdjs. Added features to qvd4js library (see below), including improved error handling, expose metadata from XML headers, lazy loading of symbol and index tables, ESM/CJS support, multi-platform testing, TypeScript typings, security hardening, bug fixes, automated npm release process and more
Constantin Müller - Author of the original qvd4js library, from which qvdjs inherited initial versions of the core functions for reading and writing QVD files.

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

qvdjs

⚠️ Important Disclaimer

Install

Usage

Lazy Loading

Important: Symbol Table and High-Cardinality Fields

Performance Optimizations

QVD File Size Limitations

Why Safety Limits Exist

Progress Tracking for Large File Writes

Working with Metadata

Security Considerations

QVD File Format

XML Header

Symbol Table

Index Table

Empty QVD Files

API Documentation

QvdDataFrame

static fromQvd(path: string, options?: object): Promise<QvdDataFrame>

static fromDict(dict: object): Promise<QvdDataFrame>

head(n: number): QvdDataFrame

tail(n: number): QvdDataFrame

rows(...args: number): QvdDataFrame

at(row: number, column: string): any

select(...args: string): QvdDataFrame

toDict(): Promise<object>

toQvd(path: string, options?: object): Promise<void>

getFieldMetadata(fieldName: string): object | null

getAllFieldMetadata(): object[]

setFileMetadata(metadata: object): void

setFieldMetadata(fieldName: string, metadata: object): void

Documentation

For Users

For Contributors

Quick Links by Task

Testing

Running Tests

Contributing

Contributors

`static fromQvd(path: string, options?: object): Promise<QvdDataFrame>`

`static fromDict(dict: object): Promise<QvdDataFrame>`

`head(n: number): QvdDataFrame`

`tail(n: number): QvdDataFrame`

`rows(...args: number): QvdDataFrame`

`at(row: number, column: string): any`

`select(...args: string): QvdDataFrame`

`toDict(): Promise<object>`

`toQvd(path: string, options?: object): Promise<void>`

`getFieldMetadata(fieldName: string): object | null`

`getAllFieldMetadata(): object[]`

`setFileMetadata(metadata: object): void`

`setFieldMetadata(fieldName: string, metadata: object): void`