@e-infra/tokenized-search

A React-based tokenized search component package with advanced filtering, autocomplete suggestions, date range filtering, field-based search restrictions, and query conversion utilities. Now supports both React Router and Next.js App Router!

Features

• Tokenized Search Input — Build structured search queries as visual tokens (Model.Field Operator Value) with keyboard-driven navigation.
• Autocomplete Suggestions — Field-level suggestions fetched from the backend with debouncing, abort handling, and fallback pipelines for unindexed fields.
• Date Range Filtering — Built-in calendar picker for single dates and date ranges with configurable locale and labels.
• Field-Based Search Restrictions — Exclude fields from filter dialogs and whitelist suggestion-eligible fields per model.
• Query Conversion Utilities — Convert form data to API filters, merge filter states, and build tokenized API request bodies.
• Search History & Saved Searches — Cookie-backed history and localStorage-backed saved searches with full token persistence.
• Schema-Driven Forms — JSON Forms integration for auto-generated metadata forms with enum detection, validation, and matrix inputs.
• OpenAPI Schema Discovery — Runtime model bootstrapping from OpenAPI schemas with fallback schema support.
• Framework Agnostic — Works with React Router v7+ and Next.js App Router out of the box!

Installation

npm install @e-infra/tokenized-search

Peer Dependencies

The package requires the following peer dependencies:

npm install react react-dom @tanstack/react-query @jsonforms/core @jsonforms/react @jsonforms/material-renderers

Optional peer dependency for routing integration (not needed for Next.js):

npm install react-router-dom

Quick Start

For React Router Projects

import {
  SearchProvider,
  TokenizedSearch,
  configureSearch,
  configureApiClient,
} from "@e-infra/tokenized-search";
import "@e-infra/tokenized-search/styles";

// 1. Configure the API client
configureApiClient({
  baseURL: import.meta.env.VITE_API_URL,
  getAccessToken: async () => localStorage.getItem("access_token"),
});

// 2. Configure search behavior
configureSearch({
  modelMapping: [
    {
      schemaName: "DatasetResponse",
      displayName: "Datasets",
      apiModel: "Dataset",
      trigramSearchFields: ["name", "description"],
    },
  ],
  apiEndpoints: {
    search: "/api/search/",
    suggestions: "/api/search/suggestions/",
    schemas: "/api/schema/",
  },
});

// 3. Render - react-router-dom is used automatically
export function App() {
  return (
    <SearchProvider>
      <TokenizedSearch />
    </SearchProvider>
  );
}

For Next.js App Router Projects

See the Next.js Integration Guide for detailed instructions. Here's a quick start:

1. Create a Router Initializer component:

// src/components/router-initializer.tsx
'use client';

import { useEffect } from 'react';
import { setRouterAdapter } from '@e-infra/tokenized-search';
import { useRouter, usePathname, useSearchParams } from 'next/navigation';

export function RouterInitializer() {
  const router = useRouter();
  const pathname = usePathname();
  const searchParams = useSearchParams();

  useEffect(() => {
    setRouterAdapter({
      navigate: (path: string) => router.push(path),
      pathname,
      search: searchParams.toString(),
      searchParams: new URLSearchParams(searchParams.toString()),
    });
  }, [pathname, searchParams.toString(), router]);

  return null;
}

2. Add to your root layout:

// src/app/layout.tsx
import { RouterInitializer } from '@/components/router-initializer';
import { SearchProvider } from '@e-infra/tokenized-search';
import '@e-infra/tokenized-search/styles';

export default function RootLayout({ children }: { children: React.ReactNode }) {
  return (
    <html lang="en">
      <body>
        <RouterInitializer />
        <SearchProvider>{children}</SearchProvider>
      </body>
    </html>
  );
}

Main Exports

Components

| Export | Description | | ---------------------- | ---------------------------------------------------------------------- | | TokenizedSearch | Main search bar component with tokenized input. | | SearchProvider | React context provider for search state, history, and API integration. | | EnhancedFilterDialog | Advanced filter dialog with schema-driven forms and base filters. | | AutoFilters | Automatic filter inputs generated from model config. | | CommonFilters | Common filter controls shared across filter UIs. | | SaveSearchDialog | Dialog for saving the current search. | | SavedSearchesList | List of saved searches with load/delete actions. | | SearchResultsDialog | Modal dialog for displaying search results. | | SearchResultsPage | Full-page search results layout. | | FilterableAttributes | Display filterable attributes for a result item. |

Router Adapter (Next.js Support)

| Export | Description | | --------------------- | -------------------------------------------------------- | | setRouterAdapter() | Set custom router adapter for Next.js or other frameworks | | getRouterAdapter() | Get the current router adapter | | useRouterAdapter() | Hook for reactive router access | | isValidRouterAdapter() | Type guard for router adapters |

Hooks

| Export | Description | | ---------------------- | --------------------------------------------------------------------- | | useSearch | Access the search context (tokens, results, history, performSearch). | | useTokenBuilder | Reducer-driven state machine for building search tokens step-by-step. | | useFieldSuggestions | Fetch field-level suggestions with debouncing and abort support. | | useDetailSuggestions | Fallback suggestion hook for unindexed metadata fields. | | useDebouncedCallback | Generic debounced callback hook. |

Services

| Export | Description | | ----------------------------------- | ---------------------------------------------------------- | | searchApi | Execute a search request. | | searchSuggestionsApi | Fetch suggestions from the backend. | | buildApiQueryParamsFromTokens | Convert tokens and free-text to API request body. | | buildApiQueryParamsForSuggestions | Build query params for suggestion requests. | | SearchHistoryService | Cookie-backed search history (add, get, clear). | | SavedSearchesService | localStorage-backed saved search persistence. | | bootstrapSearchModels | Runtime OpenAPI schema discovery and model initialization. |

Utilities

| Export | Description | | -------------------------------- | ----------------------------------------------------------------- | | formatTokenDisplayValue | Format a token's value for UI display (handles dates, ranges). | | convertFormDataToFilters | Convert JSON Forms data to AutoFilterState. | | mergeFormDataWithFilters | Merge form data with existing manual filters. | | validateFormDataForSearch | Validate form data before converting to filters. | | extractSuggestionsFromResponse | Parse suggestion strings from API highlight responses. | | extractValuesFromDatasets | Extract primitive values from nested dataset metadata. | | isUnindexedMetadataField | Determine if a field should use the fallback suggestion pipeline. | | unwrapMetadata | Unwrap a JSON schema into sections and flat fields. | | createFlatFilterOptions | Flatten unwrapped metadata into filter options. | | configureSearch | Set the global search configuration. | | getSearchConfig | Retrieve the current global configuration. |

Configuration Files

The package supports three JSON configuration files that control field visibility, suggestion eligibility, and fallback behavior.

1. extended-search-restrictions.json

Defines per-model field exclusion lists. Fields listed here are hidden from the Enhanced Filter Dialog and AutoFilters. A special common key applies exclusions to all models.

Location: src/configuration/extended-search-restrictions.json

{
  "datasets": [
    "onedata_dataset_id",
    "onedata_space_id",
    "onedata_share_id",
    "onedata_visit_id",
    "onedata_file_id",
    "reservationId"
  ],
  "common": ["shares", "perms"],
  "collections": ["onedata_space_id"],
  "templates": ["uischema", "schema", "version"]
}

2. search-suggestion-fields.json

Whitelists fields per model that are eligible for autocomplete suggestions. Only fields listed here will trigger API suggestion calls.

Location: src/configuration/search-suggestion-fields.json

{
  "datasets": ["name", "description", "created", "modified"],
  "collections": ["name", "description", "created", "modified"],
  "templates": ["name", "description", "created", "modified"]
}

Note: In v1.1.0+, static JSON imports are removed. Pass equivalent data via suggestionFieldsConfig in configureSearch().

3. unindexed-field-config.json

Controls fallback suggestion behavior for fields that are not indexed by the search backend.

Location: src/components/tokenized-search/configuration/unindexed-field-config.json

{
  "maxDatasetResults": 5,
  "maxSuggestions": 10,
  "debounceMs": 300,
  "metadataPrefix": "metadata."
}

Note: In v1.1.0+, these defaults are configurable via detailSuggestionDefaults in configureSearch().

configureSearch() API

Call configureSearch() once before rendering <SearchProvider>. It stores a global configuration object used by all components and services.

import { configureSearch } from "@e-infra/tokenized-search";

configureSearch(config: TokenizedSearchConfig);

TokenizedSearchConfig

| Property | Type | Required | Description | | -------------------------------- | --------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------- | | modelMapping | ModelMapping[] | Yes | Maps schema names to UI labels and API model names. | | primarySchema | object \| string \| (() => Promise<object>) | No | OpenAPI schema object, URL string, or async fetcher. | | fallbackSchema | object | No | Secondary schema consulted when the primary schema lacks a field. | | apiBaseUrl | string | No | Runtime override for the API base URL. | | apiEndpoints | ApiEndpointsConfig | No | Endpoint paths for search, suggestions, schemas, and detail. | | excludedFields | string[] | No | Fields excluded from filter generation. Default: ["id", "deleted_at", "__v", "_id", "metadata"] | | metadataPath | string | No | Prefix for nested metadata fields. Default: "metadata" | | dropdownFieldMatchers | DropdownFieldMatcher[] | No | Rules that map field names to dropdown types. | | suggestionFieldsConfig | SuggestionFieldConfig[] | No | Per-model whitelist of fields eligible for suggestions. | | filterExcludeConfig | FilterExcludeConfig | No | Per-model lists of fields to exclude from the filter dialog. | | modelsWithSchemaDropdown | string[] | No | Model display names that should show a schema selector dropdown. | | defaultFieldType | InputType | No | Fallback input type when schema inference fails. Default: "string" | | simpleFieldTypes | string[] | No | Schema types considered "simple". Default: ["string", "number", "boolean", "integer"] | | isUnindexedField | (fieldKey: string) => boolean | No | Custom predicate for unindexed metadata fields. | | adapter | SearchAdapter | No | Adapter for non-conforming backends. | | detailSuggestionDefaults | object | No | Defaults for detail fallback: { maxResults?, maxSuggestions?, debounceMs?, metadataPrefix? } | | router | object | No | { navigate, location, searchParams } for navigation-aware behavior. | | components | object | No | Override internal components: { TemplateSelect?, ProjectSelect?, DateRangePicker?, Loading? } | | auth | object | No | { getAccessToken: () => Promise<string \| null> } | | locale | string | No | Locale for date formatting. Default: "en-US" | | dateRangeLabels | DateRangeLabels | No | Custom labels for date range tokens. | | defaultSliderRange | object | No | Default min/max for slider inputs. Default: { min: 0, max: 10000 } | | defaultDateRange | object | No | Default date range for date pickers. | | resultTypeMap | Record<string, string> | No | Maps result type strings to display types. | | getModelConfigForResultType | (type: string) => ModelConfig \| undefined | No | Resolves a result type string to its ModelConfig. | | searchResultsPath | string | No | Path to navigate for search results. Default: "/search" | | shouldClearTokensOnRouteChange | (pathname: string) => boolean | No | Determines whether tokens should be cleared on route changes. | | unindexedFieldPrefixes | string[] | No | Prefixes that indicate an unindexed field (e.g. ["metadata."]). | | knownIndexedFields | string[] | No | Explicitly known indexed fields. | | formatLabel | (fieldName: string) => string | No | Custom label formatter for field names. |

Component Usage

<TokenizedSearch />

The main search bar component. All props are optional; when omitted, values fall back to the global config set via configureSearch().

interface TokenizedSearchProps {
  modelMapping?: ModelMapping[];
  excludedFields?: string[];
  suggestionFieldsConfig?: SuggestionFieldConfig[];
  modelsWithSchemaDropdown?: string[];
  onSearch?: (query: any) => void;
  onTokenChange?: (tokens: any[]) => void;
  renderToken?: (token: any) => React.ReactNode;
  renderSuggestion?: (suggestion: string) => React.ReactNode;
  filtersToTokens?: (
    filters: Record<string, any>,
    model?: string,
  ) => Array<{
    model: string;
    field: string;
    operator: Operator;
    value: any;
    displayValue: string;
    apiField?: string;
  }>;
}

Example:

import { TokenizedSearch, SearchProvider } from "@e-infra/tokenized-search";

function App() {
  return (
    <SearchProvider>
      <TokenizedSearch
        onSearch={(query) => console.log("Search query:", query)}
        onTokenChange={(tokens) => console.log("Tokens:", tokens)}
      />
    </SearchProvider>
  );
}

<SearchProvider />

Wraps your application (or search page) and provides the search context.

import { SearchProvider, useSearch } from "@e-infra/tokenized-search";

function SearchPage() {
  const {
    tokens,
    setTokens,
    freeTextTokens,
    setFreeTextTokens,
    isSearching,
    searchResults,
    performSearch,
    addToHistory,
    getHistory,
  } = useSearch();

  return (
    <div>
      <TokenizedSearch />
      {isSearching && <p>Searching...</p>}
      {searchResults && <p>Found {searchResults.total} results</p>}
    </div>
  );
}

function App() {
  return (
    <SearchProvider>
      <SearchPage />
    </SearchProvider>
  );
}

Styling

The package includes Tailwind CSS-based styles. Import them once in your application entry point:

import "@e-infra/tokenized-search/styles";

The stylesheet defines CSS custom properties for theming. Ensure your build setup processes Tailwind CSS v4 and PostCSS.

Integration Notes

Routing Integration

React Router

When react-router-dom is installed, the package automatically uses its routing hooks for navigation-aware behavior:

• Restores tokens and free-text from URL query parameters (?q=, ?tokens=, ?freeText=).
• Clears tokens on route changes when shouldClearTokensOnRouteChange(pathname) returns false.
• Navigates to searchResultsPath (default: /search) with serialized query state.

Next.js App Router

For Next.js projects, use the setRouterAdapter() function to provide a Next.js-compatible router:

import { setRouterAdapter } from '@e-infra/tokenized-search';
import { useRouter, usePathname, useSearchParams } from 'next/navigation';

// In your root layout
useEffect(() => {
  setRouterAdapter({
    navigate: (path: string) => router.push(path),
    pathname,
    search: searchParams.toString(),
    searchParams: new URLSearchParams(searchParams.toString()),
  });
}, [pathname, searchParams.toString(), router]);

See the Next.js Integration Guide for a complete example.

Search History

Search history is persisted in cookies (search_history) with a 365-day expiry and a limit of 10 entries. It is managed automatically by SearchProvider when searches are performed via performSearch() or addToHistory().

import { SearchHistoryService } from "@e-infra/tokenized-search";

// Manual access
const history = SearchHistoryService.getHistory();
SearchHistoryService.addSearch(tokens, freeTextTokens);
SearchHistoryService.clearHistory();

Saved Searches

Saved searches are persisted in localStorage (key: tokenized_search_saved_searches). The service supports create, read, update, delete, and duplicate detection.

import { savedSearchesService } from "@e-infra/tokenized-search";

// Create
const saved = await savedSearchesService.createSavedSearch({
  name: "My Search",
  url: "/search?q=...",
  filters: { tokens, queryBody },
});

// Load
const searches = await savedSearchesService.getSavedSearches();

License

ISC