@e-infra/tokenized-search

A React-based tokenized search component package with advanced filtering, autocomplete suggestions, date range filtering, field-based search restrictions, and query conversion utilities. Extracted from a production Next.js application and packaged for reuse across projects.

Features

• Tokenized Search Input — Build structured search queries as visual tokens (Model.Field Operator Value) with keyboard-driven navigation.
• Autocomplete Suggestions — Field-level suggestions fetched from the backend with debouncing, abort handling, and fallback pipelines for unindexed fields.
• Date Range Filtering — Built-in calendar picker for single dates and date ranges with configurable locale and labels.
• Field-Based Search Restrictions — Exclude fields from filter dialogs and whitelist suggestion-eligible fields per model.
• Query Conversion Utilities — Convert form data to API filters, merge filter states, and build tokenized API request bodies.
• Search History & Saved Searches — Cookie-backed history and localStorage-backed saved searches with full token persistence.
• Schema-Driven Forms — JSON Forms integration for auto-generated metadata forms with enum detection, validation, and matrix inputs.
• OpenAPI Schema Discovery — Runtime model bootstrapping from OpenAPI schemas with fallback schema support.

Installation

npm install @muni-ics/tokenized-search

Peer Dependencies

The package requires the following peer dependencies:

npm install react react-dom @tanstack/react-query @jsonforms/core @jsonforms/react @jsonforms/material-renderers

Optional peer dependency for routing integration:

npm install react-router-dom

Quick Start

import {
  SearchProvider,
  TokenizedSearch,
  configureSearch,
  configureApiClient,
} from "@muni-ics/tokenized-search";
import "@muni-ics/tokenized-search/dist/tokenized-search.css";

// 1. Configure the API client
configureApiClient({
  baseURL: import.meta.env.VITE_API_URL,
  getAccessToken: async () => localStorage.getItem("access_token"),
});

// 2. Configure search behavior
configureSearch({
  modelMapping: [
    {
      schemaName: "DatasetResponse",
      displayName: "Datasets",
      apiModel: "Dataset",
      trigramSearchFields: ["name", "description"],
    },
  ],
  apiEndpoints: {
    search: "/api/search/",
    suggestions: "/api/search/suggestions/",
    schemas: "/api/schema/",
  },
});

// 3. Render
export function App() {
  return (
    <SearchProvider>
      <TokenizedSearch />
    </SearchProvider>
  );
}

Main Exports

Components

| Export | Source | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------- | | TokenizedSearch | src/components/TokenizedSearch.tsx | Main search bar component with tokenized input. | | SearchProvider | src/providers/SearchProvider.tsx | React context provider for search state, history, and API integration. | | EnhancedFilterDialog | src/components/filters/EnhancedFilterDialog.tsx | Advanced filter dialog with schema-driven forms and base filters. | | AutoFilters | src/components/filters/auto-filters.tsx | Automatic filter inputs generated from model config. | | CommonFilters | src/components/filters/common-filters.tsx | Common filter controls shared across filter UIs. | | SaveSearchDialog | src/components/saved-searches/SaveSearchDialog.tsx | Dialog for saving the current search. | | SavedSearchesList | src/components/saved-searches/SavedSearchesList.tsx | List of saved searches with load/delete actions. | | SearchResultsDialog | src/components/results/SearchResultsDialog.tsx | Modal dialog for displaying search results. | | SearchResultsPage | src/components/results/SearchResultsPage.tsx | Full-page search results layout. | | FilterableAttributes | src/components/results/FilterableAttributes.tsx | Display filterable attributes for a result item. |

Hooks

| Export | Source | Description | | ---------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------------- | | useSearch | src/providers/SearchProvider.tsx | Access the search context (tokens, results, history, performSearch). | | useTokenBuilder | src/hooks/useTokenBuilder.ts | Reducer-driven state machine for building search tokens step-by-step. | | useFieldSuggestions | src/hooks/use-field-suggestions.ts | Fetch field-level suggestions with debouncing and abort support. | | useDetailSuggestions | src/hooks/use-detail-suggestions.ts | Fallback suggestion hook for unindexed metadata fields. | | useDebouncedCallback | src/hooks/useDebouncedCallback.ts | Generic debounced callback hook. |

Services

| Export | Source | Description | | ----------------------------------- | ------------------------------------------------------------------------------ | ---------------------------------------------------------- | | searchApi | src/services/SearchService.ts | Execute a search request. | | searchSuggestionsApi | src/services/SearchService.ts | Fetch suggestions from the backend. | | buildApiQueryParamsFromTokens | src/services/SearchService.ts | Convert tokens and free-text to API request body. | | buildApiQueryParamsForSuggestions | src/services/SearchService.ts | Build query params for suggestion requests. | | SearchHistoryService | src/services/HistoryService.ts | Cookie-backed search history (add, get, clear). | | SavedSearchesService | src/services/SavedSearchesService.ts | localStorage-backed saved search persistence. | | bootstrapSearchModels | src/services/BootstrapService.ts | Runtime OpenAPI schema discovery and model initialization. |

Utilities

| Export | Source | Description | | -------------------------------- | ---------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | formatTokenDisplayValue | src/utils/tokenDisplay.ts | Format a token's value for UI display (handles dates, ranges). | | convertFormDataToFilters | src/utils/queryConverter.ts | Convert JSON Forms data to AutoFilterState. | | mergeFormDataWithFilters | src/utils/queryConverter.ts | Merge form data with existing manual filters. | | validateFormDataForSearch | src/utils/queryConverter.ts | Validate form data before converting to filters. | | extractSuggestionsFromResponse | src/utils/suggestion-utils.ts | Parse suggestion strings from API highlight responses. | | extractValuesFromDatasets | src/utils/extract-values-from-datasets.ts | Extract primitive values from nested dataset metadata. | | isUnindexedMetadataField | src/utils/is-unindexed-metadata-field.ts | Determine if a field should use the fallback suggestion pipeline. | | unwrapMetadata | src/utils/metadataUnwrapper.ts | Unwrap a JSON schema into sections and flat fields. | | createFlatFilterOptions | src/utils/metadataUnwrapper.ts | Flatten unwrapped metadata into filter options. | | configureSearch | src/config/search-config.ts | Set the global search configuration. | | getSearchConfig | src/config/search-config.ts | Retrieve the current global configuration. |

Configuration Files

The package supports three JSON configuration files that control field visibility, suggestion eligibility, and fallback behavior.

1. extended-search-restrictions.json

Defines per-model field exclusion lists. Fields listed here are hidden from the Enhanced Filter Dialog and AutoFilters. A special common key applies exclusions to all models.

Location: src/configuration/extended-search-restrictions.json

{
  "datasets": [
    "onedata_dataset_id",
    "onedata_space_id",
    "onedata_share_id",
    "onedata_visit_id",
    "onedata_file_id",
    "reservationId"
  ],
  "common": ["shares", "perms"],
  "collections": ["onedata_space_id"],
  "templates": ["uischema", "schema", "version"]
}

| Key | Description | | ---------------------------------------- | ------------------------------------------------------------------- | | common | Fields excluded from every model's filter list. | | datasets / collections / templates | Model-specific exclusions. Keys must match the lowercase model key. |

2. search-suggestion-fields.json

Whitelists fields per model that are eligible for autocomplete suggestions. Only fields listed here will trigger API suggestion calls.

Location: src/configuration/search-suggestion-fields.json

{
  "datasets": ["name", "description", "created", "modified"],
  "collections": ["name", "description", "created", "modified"],
  "templates": ["name", "description", "created", "modified"]
}

| Key | Description | | ---------------------------------------- | ------------------------------------------------------------ | | datasets / collections / templates | Array of field keys eligible for suggestions for that model. |

Note: In v1.1.0+, static JSON imports are removed. Pass equivalent data via suggestionFieldsConfig in configureSearch().

3. unindexed-field-config.json

Controls fallback suggestion behavior for fields that are not indexed by the search backend.

Location: src/components/tokenized-search/configuration/unindexed-field-config.json

{
  "maxDatasetResults": 5,
  "maxSuggestions": 10,
  "debounceMs": 300,
  "metadataPrefix": "metadata."
}

| Property | Type | Description | | ------------------- | -------- | ------------------------------------------------------------- | | maxDatasetResults | number | Max detail records fetched for fallback extraction. | | maxSuggestions | number | Max suggestions returned after filtering. | | debounceMs | number | Debounce delay for fallback suggestion input. | | metadataPrefix | string | Dot-notation prefix for metadata fields (e.g. "metadata."). |

Note: In v1.1.0+, these defaults are configurable via detailSuggestionDefaults in configureSearch().

configureSearch() API

Call configureSearch() once before rendering <SearchProvider>. It stores a global configuration object used by all components and services.

import { configureSearch } from "@muni-ics/tokenized-search";

configureSearch(config: TokenizedSearchConfig);

TokenizedSearchConfig

| Property | Type | Required | Description | | -------------------------------- | --------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------- | | modelMapping | ModelMapping[] | Yes | Maps schema names to UI labels and API model names. | | primarySchema | object \| string \| (() => Promise<object>) | No | OpenAPI schema object, URL string, or async fetcher. | | fallbackSchema | object | No | Secondary schema consulted when the primary schema lacks a field. | | apiBaseUrl | string | No | Runtime override for the API base URL. | | apiEndpoints | ApiEndpointsConfig | No | Endpoint paths for search, suggestions, schemas, and detail. | | excludedFields | string[] | No | Fields excluded from filter generation. Default: ["id", "deleted_at", "__v", "_id", "metadata"] | | metadataPath | string | No | Prefix for nested metadata fields. Default: "metadata" | | dropdownFieldMatchers | DropdownFieldMatcher[] | No | Rules that map field names to dropdown types. | | suggestionFieldsConfig | SuggestionFieldConfig[] | No | Per-model whitelist of fields eligible for suggestions. | | filterExcludeConfig | FilterExcludeConfig | No | Per-model lists of fields to exclude from the filter dialog. | | modelsWithSchemaDropdown | string[] | No | Model display names that should show a schema selector dropdown. | | defaultFieldType | InputType | No | Fallback input type when schema inference fails. Default: "string" | | simpleFieldTypes | string[] | No | Schema types considered "simple". Default: ["string", "number", "boolean", "integer"] | | isUnindexedField | (fieldKey: string) => boolean | No | Custom predicate for unindexed metadata fields. | | adapter | SearchAdapter | No | Adapter for non-conforming backends. | | detailSuggestionDefaults | object | No | Defaults for detail fallback: { maxResults?, maxSuggestions?, debounceMs?, metadataPrefix? } | | router | object | No | { navigate, location, searchParams } for navigation-aware behavior. | | components | object | No | Override internal components: { TemplateSelect?, ProjectSelect?, DateRangePicker?, Loading? } | | auth | object | No | { getAccessToken: () => Promise<string \| null> } | | locale | string | No | Locale for date formatting. Default: "en-US" | | dateRangeLabels | DateRangeLabels | No | Custom labels for date range tokens. | | defaultSliderRange | object | No | Default min/max for slider inputs. Default: { min: 1900, max: 2100 } | | defaultDateRange | object | No | Default date range for date pickers. | | resultTypeMap | Record<string, string> | No | Maps result type strings to display types. | | getModelConfigForResultType | (type: string) => ModelConfig \| undefined | No | Resolves a result type string to its ModelConfig. | | searchResultsPath | string | No | Path to navigate for search results. Default: "/search" | | shouldClearTokensOnRouteChange | (pathname: string) => boolean | No | Determines whether tokens should be cleared on route changes. | | unindexedFieldPrefixes | string[] | No | Prefixes that indicate an unindexed field (e.g. ["metadata."]). | | knownIndexedFields | string[] | No | Explicitly known indexed fields. | | formatLabel | (fieldName: string) => string | No | Custom label formatter for field names. |

ModelMapping

interface ModelMapping {
  schemaName: string; // Name in the OpenAPI schema (e.g. "DatasetResponse")
  displayName: string; // UI label (e.g. "Datasets")
  apiModel: string; // Value sent to search API (e.g. "Dataset")
  trigramSearchFields?: readonly string[];
  fieldOverrides?: Record<string, Partial<FieldOverride>>;
}

ApiEndpointsConfig

interface ApiEndpointsConfig {
  search: string; // default: "/api/search/"
  suggestions: string; // default: "/api/search/suggestions/"
  schemas?: string; // default: "/api/schema/"
  detail?: (model: string, id: string) => string;
}

Component Usage

<TokenizedSearch />

The main search bar component. All props are optional; when omitted, values fall back to the global config set via configureSearch().

interface TokenizedSearchProps {
  modelMapping?: ModelMapping[];
  excludedFields?: string[];
  suggestionFieldsConfig?: SuggestionFieldConfig[];
  modelsWithSchemaDropdown?: string[];
  onSearch?: (query: any) => void;
  onTokenChange?: (tokens: any[]) => void;
  renderToken?: (token: any) => React.ReactNode;
  renderSuggestion?: (suggestion: string) => React.ReactNode;
  filtersToTokens?: (
    filters: Record<string, any>,
    model?: string,
  ) => Array<{
    model: string;
    field: string;
    operator: Operator;
    value: any;
    displayValue: string;
    apiField?: string;
  }>;
}

Example:

import { TokenizedSearch, SearchProvider } from "@muni-ics/tokenized-search";

function App() {
  return (
    <SearchProvider>
      <TokenizedSearch
        onSearch={(query) => console.log("Search query:", query)}
        onTokenChange={(tokens) => console.log("Tokens:", tokens)}
      />
    </SearchProvider>
  );
}

<SearchProvider />

Wraps your application (or search page) and provides the search context.

import { SearchProvider, useSearch } from "@muni-ics/tokenized-search";

function SearchPage() {
  const {
    tokens,
    setTokens,
    freeTextTokens,
    setFreeTextTokens,
    isSearching,
    searchResults,
    performSearch,
    addToHistory,
    getHistory,
  } = useSearch();

  return (
    <div>
      <TokenizedSearch />
      {isSearching && <p>Searching...</p>}
      {searchResults && <p>Found {searchResults.total} results</p>}
    </div>
  );
}

function App() {
  return (
    <SearchProvider>
      <SearchPage />
    </SearchProvider>
  );
}

<EnhancedFilterDialog />

Advanced filter dialog with model selection, schema dropdown, base filters, and JSON Forms integration.

import { EnhancedFilterDialog } from "@muni-ics/tokenized-search";

function MyComponent() {
  const [open, setOpen] = useState(false);

  const handleApply = (
    type: string,
    filters: AutoFilterState,
    schemaId?: string,
  ) => {
    console.log("Applied:", { type, filters, schemaId });
    setOpen(false);
  };

  return (
    <>
      <button onClick={() => setOpen(true)}>Open Filters</button>
      <EnhancedFilterDialog
        open={open}
        onClose={() => setOpen(false)}
        onApply={handleApply}
      />
    </>
  );
}

Types & Models

Core types are defined in src/types/index.ts and src/types/config.ts.

Token

interface Token {
  model: string;
  field: string;
  operator: Operator;
  value:
    | string
    | number
    | Date
    | boolean
    | { from?: Date | string; to?: Date | string };
  displayValue: string;
  apiField?: string;
}

Operator

type Operator = "=" | "!=" | "<" | ">" | "<=" | ">=" | "contains" | "regex";

FilterOption

interface FilterOption {
  key: string;
  label: string;
  inputType: InputType;
  min?: number;
  max?: number;
  inputTypeOverride?: InputType;
  labelOverride?: string;
  dataSourceKey?: string;
  apiField?: string;
}

InputType

type InputType = "string" | "number" | "boolean" | "date" | "enum" | "matrix";

ModelConfig

interface ModelConfig {
  label: string;
  apiModel: string;
  filters: FilterOption[];
  trigramSearchFields?: readonly string[];
}

SearchHistoryEntry

interface SearchHistoryEntry {
  tokens: Token[];
  freeTextQuery?: string; // @deprecated
  freeTextTokens?: string[];
}

MetadataField & MetadataSection

interface MetadataField {
  key: string;
  label: string;
  inputType: InputType;
  description?: string;
  required?: boolean;
  min?: number;
  max?: number;
  step?: number;
  placeholder?: string;
  suggestions?: SuggestionConfig;
  matrix?: MatrixConfig;
  nested?: MetadataField[];
}

interface MetadataSection {
  key: string;
  label: string;
  description?: string;
  fields: MetadataField[];
  isCollapsible?: boolean;
  defaultExpanded?: boolean;
}

Utilities

query-converter.ts

Converts JSON Forms output into filter states compatible with the search API.

import {
  convertFormDataToFilters,
  mergeFormDataWithFilters,
  validateFormDataForSearch,
  type AutoFilterState,
} from "@muni-ics/tokenized-search";

// Convert form data to flat filter state
const filters: AutoFilterState = convertFormDataToFilters({
  metadata: {
    title: "Test Dataset",
    solvent: "water",
    concentration: 50,
  },
});
// => { "metadata.title": "Test Dataset", "metadata.solvent": "water", "metadata.concentration": 50 }

// Merge with existing filters
const merged = mergeFormDataWithFilters(formsData, existingFilters);

// Validate before search
const { isValid, errors } = validateFormDataForSearch(formData);

suggestion-utils.ts

Parses suggestion responses from the backend.

import { extractSuggestionsFromResponse } from "@muni-ics/tokenized-search";

const suggestions = extractSuggestionsFromResponse(response, "name", "foo");
// => ["foobar", "foo-bar", "food"]

Supports two highlight formats:

1. Object format (new API): { "metadata.dataset_metadata.title": "value" }
2. String array format (legacy): ["fieldName → value", "fieldName: value"]

token-display.ts

Formats token values for UI display.

import { formatTokenDisplayValue } from "@muni-ics/tokenized-search";

const display = formatTokenDisplayValue(token);
// Handles Date, date ranges, and objects automatically.

metadata-unwrapper.ts

Unwraps JSON schemas into structured sections and flat fields for form generation.

import {
  unwrapMetadata,
  createFlatFilterOptions,
} from "@muni-ics/tokenized-search";

const { sections, flatFields, matrixFields, suggestionFields } =
  unwrapMetadata(schema);
const allFields = createFlatFilterOptions({
  sections,
  flatFields,
  matrixFields,
  suggestionFields,
});

extract-values-from-datasets.ts

Extracts primitive string values from nested dataset metadata using dot-notation paths.

import { extractValuesFromDatasets } from "@muni-ics/tokenized-search";

const values = extractValuesFromDatasets(
  datasets,
  "metadata.plant_info.stress_type",
);
// => ["drought", "heat", "salt"]

is-unindexed-metadata-field.ts

Determines whether a field should use the fallback suggestion pipeline because it is not indexed.

import { isUnindexedMetadataField } from "@muni-ics/tokenized-search";

const unindexed = isUnindexedMetadataField("plant_info.stress_type");
// => true (dot-notation fields are treated as metadata by default)

Rules:

1. Custom config.isUnindexedField predicate takes precedence.
2. Field keys starting with configured unindexedFieldPrefixes are unindexed.
3. Fields not in knownIndexedFields are unindexed.
4. Any dot-notation field not explicitly known as indexed is treated as metadata.

Styling

The package includes Tailwind CSS-based styles. Import them once in your application entry point:

import "@muni-ics/tokenized-search/dist/tokenized-search.css";

The stylesheet defines CSS custom properties for theming:

:root {
  --background: oklch(1 0 0);
  --foreground: oklch(26.862% 0.00003 271.152);
  --primary: #257400;
  --primary-foreground: #fff;
  --radius: 0.625rem;
  /* ... */
}

Ensure your build setup processes Tailwind CSS v4 and PostCSS. The package uses @tailwindcss/postcss for CSS processing.

Demo / Example

A demo component is provided at src/components/demo/enhanced-filter-demo.tsx. It demonstrates:

• Opening the EnhancedFilterDialog
• Handling onApply callbacks
• Displaying applied filters
• Schema-driven enum dropdown generation

import { EnhancedFilterDemo } from "@muni-ics/tokenized-search";

function App() {
  return <EnhancedFilterDemo />;
}

The demo includes an example schema with an enum field (solvent) that automatically renders as a <Select> dropdown inside the JSON Forms integration.

Integration Notes

Search History

Search history is persisted in cookies (search_history) with a 365-day expiry and a limit of 10 entries. It is managed automatically by SearchProvider when searches are performed via performSearch() or addToHistory().

import { SearchHistoryService } from "@muni-ics/tokenized-search";

// Manual access
const history = SearchHistoryService.getHistory();
SearchHistoryService.addSearch(tokens, freeTextTokens);
SearchHistoryService.clearHistory();

Saved Searches

Saved searches are persisted in localStorage (key: tokenized_search_saved_searches). The service supports create, read, update, delete, and duplicate detection.

import { savedSearchesService } from "@muni-ics/tokenized-search";

// Create
const saved = await savedSearchesService.createSavedSearch({
  name: "My Search",
  url: "/search?q=...",
  filters: { tokens, queryBody },
});

// Load
const searches = await savedSearchesService.getSavedSearches();

// Check if current search is already saved
const existing = await savedSearchesService.isSearchSaved({ tokens });

Routing Integration

SearchProvider integrates with react-router-dom for navigation-aware behavior:

• Restores tokens and free-text from URL query parameters (?q=, ?tokens=, ?freeText=).
• Clears tokens on route changes when shouldClearTokensOnRouteChange(pathname) returns false.
• Navigates to searchResultsPath (default: /search) with serialized query state.

Adapter Pattern

For backends that do not conform to the default API contract, implement a SearchAdapter:

configureSearch({
  adapter: {
    executeSearch: async (req) => {
      const res = await myApi.search(req);
      return { results: res.hits, count: res.total };
    },
    fetchSuggestions: async (params) => {
      const res = await myApi.autocomplete(params.field, params.query);
      return res.options;
    },
    fetchDetailSuggestions: async (params) => {
      const res = await myApi.detailValues(params.model, params.fieldKey);
      return res.values;
    },
  },
});

License

ISC