@sanity/assist

Table of contents

• Table of contents
• About Sanity AI Assist
• Installation
• Setup
  • Add the plugin
  • Enabling the AI Assist API
  • Permissions
  • Conditional user access
  • Assist configuration
• Schema configuration
  • Disable AI Assist for a schema type
  • Disable for a field
  • Disable for an array type
  • Unsupported types
  • Date and datetime
  • Hidden and readOnly fields
  • Reference support
  • Troubleshooting
• Included document types
• Field and type filters
• Image description generation
• Image generation
• Full document translation
  • What it solves
  • Configure
• Field level translations
  • What it solves
  • Configure
• Adding translation actions to fields
• Translation style guide
• Custom field actions
  • useExampleFieldActions
  • Define helpers
  • useUserInput
• License

About Sanity AI Assist

Free your team to do more of what they’re great at (and less busywork) with the AI assistant that works with structured content. Attach reusable AI instructions to fields and documents to supercharge your editorial workflow. You create the instructions; Sanity AI Assist does the rest. Learn more about writing instructions in the Sanity documentation. Read the release announcement here.

Using this feature requires Sanity to send data to OpenAI.com for processing. It uses generative AI; you should verify the data before using it.

Installation

In your Studio project folder, install the following plugin dependency:

npm install @sanity/assist sanity@latest

This plugin requires sanity version 3.26 or greater.

Setup

Note: AI Assist is available for all projects on the Growth plan and up.

Add the plugin

In sanity.config.ts, add assist to the plugins array:

import {assist} from '@sanity/assist'
export default defineConfig({
  /* other config */
  plugins: [
    /* other plugins */
    assist(),
  ],
})

Enabling the AI Assist API

After installing and adding the plugin and having the AI Assist feature enabled for your project and its datasets, you need to create a token for the plugin to access the AI Assist API. This needs to be done by a member of the project with token creation permissions (typically someone with an admin or developer role).

• Start the studio and open any document
• Click *the sparkle icon** (✨) in the document header near the close document X-button
• Then select Manage instructions
• Selecting Manage instructions will open an inspector panel
• Click the Enable AI assistance button to create a token and enable AI Assist for everyone with access to the project You will find a new API token entry for your project named “Sanity AI” in your project's API settings on sanity.io/manage. The plugin will now work for any dataset in your project. Note: You can revoke this token at any time to disable Sanity AI Assist service. A new token has to be generated via the plugin UI for it to work again.

Permissions

If your project is using custom roles (Enterprise), there are some additional considerations. To see AI Assist presence when running instructions, users will need read access to documents of _type=="sanity.assist.task.status". To edit instructions, users will need read and write access to documents of _type=="sanity.assist.schemaType.annotations". Note that instructions run using the permissions of the user invoking it, so only fields that the user themselves can edit can be changed by the instruction instance.

Conditional user access

To limit which users can see the AI Assist actions in the Studio, use a custom-plugin after assist() that filters out the inspector and actions, based on user properties:

import {CurrentUser, defineConfig} from 'sanity'
import {assist} from '@sanity/assist'
export default defineConfig({
  // ...
  plugins: [
    // ...
    assist(),
    {
      name: 'disable-ai-assist',
      document: {
        inspectors: (prev, {currentUser}) =>
          isAiAssistAllowed(currentUser)
            ? prev
            : prev.filter((inspector) => inspector.name !== 'ai-assistance'),
        unstable_fieldActions: (prev, {currentUser}) =>
          isAiAssistAllowed(currentUser)
            ? prev
            : prev.filter((fieldActions) => fieldActions.name !== 'sanity-assist-actions'),
      },
    },
  ],
})
const ALLOWED_ROLES = ['administrator']
function isAiAssistAllowed(user?: CurrentUser | null) {
  return user && user.roles.some((role) => ALLOWED_ROLES.includes(role.name))
}

Assist configuration (optional)

assist({
  // Showing default values
  assist: {
    localeSettings: () => Intl.DateTimeFormat().resolvedOptions(),
    maxPathDepth: 4,
    temperature: 0.3,
  },
  translate: {
    /* see sections about document and field translation */
  },
})

• localeSettings: See section on date and datetime
• maxPathDepth: The max depth for document paths AI Assist will write to.
• temperature: Influences how much the output of an instruction will vary between runs. For more details, please review the TSDocs of the individual config parameters in assistTypes.ts

Schema configuration

By default, most string, object, and array field types (including Portable Text!) have AI writing assistance enabled. Your assistant can write to all compatible fields that it detects.

Disable AI Assist for a schema type

// this will disable AI assistance wherever it is used,
// ie: as field, document, array types
defineType({
  name: 'policy',
  type: 'document',
  options: {
    aiAssist: {exclude: true},
  },
  fields: [
    // ...
  ],
})

Disable for a field

// this disables AI assistance only for the specific field
defineType({
  name: 'product',
  type: 'object',
  fields: [
    defineField({
      name: 'sku',
      type: 'string',
      options: {
        aiAssist: {exclude: true},
      },
    }),
  ],
})

Disable for an array type

// this disables AI assistance for the specific array member
// if all types in the `of` array are excluded, the array type is also considered excluded
defineType({
  name: 'product',
  type: 'array',
  of: [
    defineArrayMember({
      type: 'customProduct',
      options: {
        aiAssist: {exclude: true},
      },
    }),
  ],
})

Unsupported types

The following types are not supported, and behave as excluded types:

• Image (supported when image has custom fields)
• File (never supported, even when file has custom fields)
• Reference (supported when configured with embeddingsIndex) Fields with these types will not be changed by the assistant, do not have AI Assist actions, and cannot be referenced in instructions. Objects where all fields are excluded or unsupported and arrays where all member types are excluded or unsupported will also be excluded.

Date and datetime

• Date
• Datetime Starting from v3.0.0, AI Assist can write to date and datetime fields. Instructions can use language like "tomorrow at noon" or "next year", and when Assist writes to the field, it will be converted to a field-compatible value. Language about time is locale and timeZone dependant. By default instructions will use the locale and timezone provided by the browser (Intl.DateTimeFormat().resolvedOptions()). Alternatively, you can configure the plugin per user with an assist.localeSettings function that should return LocaleSettings.

Example

assist({
  assist: {
    localeSettings: ({currentUser, defaultSettings}) => {
      if (currentUser.roles.includes('admin')) {
        // forces locale and timeZone for admins
        return {
          locale: 'en-US',
          timeZone: 'America/New_York',
        }
      }
      // defaultSettings is the same as using:
      // const {locale, timeZone} = Intl.DateTimeFormat().resolvedOptions()
      return defaultSettings
    },
  },
})

For a list of allowed values for these parameters, see the following resources:

• locale: Mozilla on Intl
• timeZone: Wiki on time zones

Hidden and readOnly fields

In AI Assist 2.0 and later, conditionally hidden and readOnly fields can have instructions. These fields can be written to by an instruction, as long as the field is non-hidden and writable when the instruction starts running. Fields with hidden or readOnly set to literal true will be skipped by AI Assist. Note: An instruction will not re-evaluate these states during a run. I.e., if an instruction makes a change during its execution that triggers another field to change its hidden or readOnly status, the running instruction will still consider these as if in their original state. If it is essential that AI Assist never writes to a conditional field, it should be marked with options.aiAssist.exclude: true.

Reference support

Create an Embeddings-index

To enable AI assist for references, first, your project must have an existing embeddings-index that includes the document types it should be able to reference. You can manage your indexes directly in the studio using the Embeddings Index Dashboard plugin.

Set schema options

Set options.aiAssist.embeddingsIndex for reference fields/types you want to enable reference instructions for. Reference fields with this option set can have instructions attached and will be visited when running instructions for object fields and arrays. AI Assist will use the embeddings-index, further filtered by the types allowed by the reference, to look up contextually relevant references. For arrays or portable text fields with references, one or more references can be added. Use the instruction text to control this. Example:

import {defineArrayMember} from 'sanity'
defineField({
  type: 'reference',
  name: 'articleReference',
  title: 'Article reference',
  to: [{type: 'article'}],
  options: {
    aiAssist: {
      embeddingsIndex: 'article-index',
    },
  },
})

An example instruction attached to this field could be: Given <Document field: Title> suggest a related article Running it would use the article-index to find a related article based on the current document title.

Troubleshooting

There are limits to how much text the assistant can handle when processing an instruction. Under the hood, AI Assist will add information about your schema, which adds to what's commonly referred to as “the context window.” If you have a very large schema (that is, many document and field types), it can be necessary to exclude types to limit how much of the context window is used for the schema itself. We recommend excluding any and all types that aren't likely to benefit from automated workflows. A quick win is typically to exclude array types. It can be a good idea to exclude most non-block types from Portable Text arrays. This will ensure that AI Assist outputs mostly formatted text. It is also possible to exclude fields/types when creating an instruction. See Field and type filters for more.

Included document types

This plugin adds an AI Context document type. If your Studio uses Structure Builder to configure the studio structure, you might have to add this document type to your structure. The document type name can be imported from the plugin:

import {contextDocumentTypeName} from '@sanity/assist'
// put into structure in structure
S.documentTypeListItem(contextDocumentTypeName)

Field and type filters

When creating instructions for documents, object fields, array fields, or portable text fields, you can explicitly control what will be visited by AI Assist. By default, the assistant will include all compatible fields and types. Opting out fields/types per instruction is done using the respective field/type filter checkboxes under the instruction. When using these filters, it is not necessary to tell AI Assist what to include in the instruction text itself. Note that once the schema targeted by the instruction changes, the following behavior applies:

• instructions that include all fields or types will automatically also include the new fields or types
• instructions that have excluded one or more fields or types will NOT include the new fields or types

Image description generation

AI Assist can optionally generate descriptions for images. This has to be enabled on an image-type/field, by setting the options.aiAssist.imageDescriptionField on the image type, where imageDescriptionField is the field name of a custom string-field on the image object:

defineField({
  type: 'image',
  name: 'inlineImage',
  title: 'Image',
  fields: [
    defineField({
      type: 'string',
      name: 'altText',
      title: 'Alternative text',
    }),
  ],
  options: {
    aiAssist: {
      imageDescriptionField: 'altText',
    },
  },
})

This will add a Generate image description action to the configured field. The Generate image description action will automatically run whenever the image changes. imageDescriptionField can be a nested field, if the image has an object field, i.e. imageDescriptionField: 'wrapper.altText'. Fields within array items are not supported. By default, the caption field will regenerate whenever the image asset changes. To disable this behavior use the following configuration:

{
  imageDescriptionField: {
    path: 'wrapper.altText',
      updateOnImageChange: false
  }
}

Image generation