@cyberlangke/tokkit-bytedance-seed
v1.11.0
Published
ByteDance-Seed tokenizer families for tokkit.
Readme
@cyberlangke/tokkit-bytedance-seed
ByteDance-Seed 官方 tokenizer 的 tokkit 子包,当前包含 academic-ds、Seed-OSS、Seed-Coder、Stable-DiffCoder 主线文本模型内置 family。
支持的模型
ByteDance-Seed/academic-ds-9BByteDance-Seed/Seed-OSS-36B-BaseByteDance-Seed/Seed-Coder-8B-BaseByteDance-Seed/Stable-DiffCoder-8B-Base
使用方法
npm install @cyberlangke/tokkit-bytedance-seedimport { getTokenizer } from "@cyberlangke/tokkit-bytedance-seed"
const tokenizer = await getTokenizer("seed-coder")
const sameTokenizer = await getTokenizer("ByteDance-Seed/Seed-Coder-8B-Reasoning")
console.log(tokenizer === sameTokenizer) // true