capacitor-digitalink

Allows use of Google's MLKit DigitalInk models

You must provide a set of coordinates with or without time values to properly use this plugin. There are various implementations out there, but the most common way to accomplish this is by using a Canvas, moving the Canvas Context to draw with pixels, then sending the data over to the plugin.

For example, the picture at predrawn-inks/hi/hi.png:

Would have this data:

// predrawn-inks/hi/hi.json
// [ [x1[], y1[]], [x2[], y2[]], [x3[], y3[]] ]
[
  [ [86.67578125, 86.67578125, 86.859375, ...], [103.3359375, 103.40234375, 104.69921875, ...] ], // h
  [ [203.86328125, 203.86328125, 203.7578125, ...], [159.9921875, 160.05859375, 161.91796875, ..] ], // i stem
  [ [206.203125, 206.203125, 205.99609375, ...], [123.25390625, 123.18359375, 122.91015625, ...] ] // i point
]

Again, the time values are optional.

Once the data has been sent to the plugin, call the doRecognition method to perform recognition. See the API docs below for details on params, returns, etc.

Install

npm install capacitor-digitalink
npx cap sync

Todos / Notes

• Provide functionality for finding LocalModel (en-US) as a default model, and don't allow en-US as a param in deleting models?

QA Check - check Asana

API

• initializePlugin()
• erase()
• logStrokes(...)
• doRecognition(...)
• downloadSingularModel(...)
• downloadMultipleModels(...)
• deleteModel(...)
• getDownloadedModels()
• Interfaces

initializePlugin()

initializePlugin() => Promise<{ ok: boolean; msg: string; }>

Initializes notifications on iOS -- NOT supported in Android due to the use of other event listeners

Returns: Promise<{ ok: boolean; msg: string; }>

erase()

erase() => Promise<{ ok: boolean; msg: string; }>

Erases natively stored stroke/point/ink data

Returns: Promise<{ ok: boolean; msg: string; }>

logStrokes(...)

logStrokes(options: XYTOptions) => Promise<{ ok: boolean; msg: string; options: XYTOptions; }>

Sends XY coordinate data to native code to be prepared for model inference Can include/exclude time values Ensure all units for coordinates/time are consistent between logs. Unit types don't matter, they just need to be the same -- all values are normalized

| Param | Type | Description | | ------------- | ------------------------------------------------- | -------------------------------------------------------- | | options | XYTOptions | - [ X coordinate, Y coordinate, T time in milliseconds ] |

Returns: Promise<{ ok: boolean; msg: string; options: XYTOptions; }>

doRecognition(...)

doRecognition(options: RecognitionOptions) => Promise<{ ok: boolean; msg: string; results: { candidates: string[]; scores: number[]; }; options: RecognitionOptions; }>

Runs inference either on the provided model via the model param, or on the default English model. All params are optional.

| Param | Type | | ------------- | ----------------------------------------------------------------- | | options | RecognitionOptions |

Returns: Promise<{ ok: boolean; msg: string; results: { candidates: string[]; scores: number[]; }; options: RecognitionOptions; }>

downloadSingularModel(...)

downloadSingularModel(model: Model, callback: SingularModelCallback) => Promise<CallbackID>

Downloads singular model. Last callback has the 'done' property set to true, and signals the last callback.

| Param | Type | Description | | -------------- | --------------------------------------------------------- | --------------------------------------------------------------------------------------- | | model | Model | model to download. Native code checks if model is valid and if it's already downloaded. | | callback | (response: Response, error?: any) => void | callback function that runs each time data is sent from the native code. |

Returns: Promise<string>

downloadMultipleModels(...)

downloadMultipleModels(models: Models, callback: MultipleModelCallback) => Promise<CallbackID>

Downloads multiple models from a given array. Callback function will return a response or an error dependent on whether a given model has already been downloaded, is a valid/invalid model, or is finished being downloaded. The last model will be have the 'done' property set to true and signals the last callback.

| Param | Type | Description | | -------------- | --------------------------------------------------------- | --------------------------------------------------------------- | | models | Models | array of models to download. | | callback | (response: Response, error?: any) => void | callback that runs each time data is sent from the native code. |

Returns: Promise<string>

deleteModel(...)

deleteModel(options: DeleteModelOptions, callback: DeleteModelCallback) => Promise<CallbackID>

Deletes a singular/collection of models downloaded to the device, or all models.

| Param | Type | Description | | -------------- | ----------------------------------------------------------------- | ----------------------------------------------------------- | | options | DeleteModelOptions | delete all models, a singular model, or an array of models. | | callback | (response: Response, error?: any) => void | |

Returns: Promise<string>

getDownloadedModels()

getDownloadedModels() => Promise<{ ok: true; msg: string; models: string[]; }>

Returns: Promise<{ ok: true; msg: string; models: string[]; }>

Interfaces

XYTOptions

| Prop | Type | | ------- | --------------------- | | x | number[] | | y | number[] | | t | number[] |

RecognitionOptions

| Prop | Type | | ----------------- | -------------------------------------- | | model | string | | context | string | | writingArea | { w: number; h: number; } |

Model

| Prop | Type | | ----------- | ------------------- | | model | string |

Models

| Prop | Type | | ------------ | --------------------- | | models | string[] |

DeleteModelOptions

| Prop | Type | | ------------ | --------------------- | | all | boolean | | model | string | | models | string[] |