tokenizer

js/text/tokenizer.ts

fino:text/tokenizer - text to token ids, in pure TypeScript.

Loads the tokenizers published models actually ship with — a Hugging Face tokenizer.json carrying a BPE or WordPiece vocabulary, or a tiktoken ranks file — and reproduces their encoding exactly, including the normalizer, pre-tokenizer, post-processor, and decoder stages that decide where token boundaries fall. There is no native dependency: tokenizers is Rust with no C ABI and sentencepiece is C++ only, so binding either would mean shipping a compiled artifact, while BPE and WordPiece over a vocabulary are string processing that TypeScript does well.

Encoding is deterministic and stateless. A tokenizer serializes back to the spec it was built from, so a DataLoader worker can rebuild an identical one from a structured-cloneable value and produce byte-identical ids — which is also how dataset-scale tokenization gets parallelized, since a single encode is CPU-bound and short.

Encoding.offsets index the original input string, so a prediction can be mapped back onto the caller's own text. Special tokens the post-processor added carry a zero-width offset to mark that they came from no input.

Unigram vocabularies and tokenizer training are deliberately absent: Unigram needs the sentencepiece character map that accompanies it, and loading one without the other yields ids that look right and are not. Both throw a specific error rather than degrading quietly. A llama.cpp GGUF vocabulary is reached the same way any other source is — by building a spec object and handing it to Tokenizer.fromJSON — which keeps this module independent of llama.cpp.

import { Tokenizer } from 'fino:text/tokenizer';

const tokenizer = await Tokenizer.fromFile('./tokenizer.json');
const encoded = tokenizer.encode('Hello, world!');
console.log(encoded.ids, encoded.tokens);
console.log(tokenizer.decode(encoded.ids));

Interfaces

`interface AddedTokenSpec {`

An added-token entry as a tokenizer.json spells it.

Properties

`id: number`

`content: string`

`single_word?: boolean`

`lstrip?: boolean`

`rstrip?: boolean`

`normalized?: boolean`

`special?: boolean`

`interface TokenizerSpec {`

A Hugging Face tokenizer.json.

Only the fields that affect encoding are read; version, training metadata, and unknown keys are preserved by toJSON() but otherwise ignored.

Properties

`version?: string`

`truncation?: Record<string, unknown> | null`

`padding?: Record<string, unknown> | null`

`added_tokens?: AddedTokenSpec[]`

`normalizer?: NormalizerSpec | null`

`pre_tokenizer?: PreTokenizerSpec | null`

`post_processor?: PostProcessorSpec | null`

`decoder?: DecoderSpec | null`

`model: Record<string, unknown>`

`interface EncodeOptions {`

Per-call encoding options.

Properties

`addSpecialTokens?: boolean`

Apply the post-processor's special tokens. Defaults to true.

`truncation?: TruncationOptions | null`

Truncation for this call, overriding the tokenizer's default.

`padding?: PaddingOptions | null`

Padding for this call, overriding the tokenizer's default.

`interface DecodeOptions {`

Options for decode.

Properties

`skipSpecialTokens?: boolean`

Drop tokens flagged special. Defaults to true.

`interface TokenizerOptions {`

Construction options beyond the spec.

Properties

`truncation?: TruncationOptions | null`

`padding?: PaddingOptions | null`

`interface TiktokenOptions {`

Building a tokenizer from a tiktoken ranks table.

Properties

`vocab: Record<string, number>`

Ranks as {token: rank} in the printable byte alphabet.

`pattern?: string`

Word-splitting pattern; defaults to the named encoding's.

`specialTokens?: Record<string, number>`

Literal tokens matched ahead of the model.

`interface Encoding {`

The result of encoding one input.

All arrays are parallel and the same length. offsets index the original input string, so text.slice(...offsets[i]) is the source of token i; a zero-width offset marks a token that came from no input text, such as a special token the post-processor added.

Properties

`ids: number[]`

Vocabulary ids.

`tokens: string[]`

Vocabulary surface forms.

`typeIds: number[]`

Sequence membership: 0 for the first sequence, 1 for the pair.

`attentionMask: number[]`

1 for real tokens, 0 for padding.

`specialTokensMask: number[]`

1 for special tokens the caller did not write, 0 otherwise.

`offsets: Array<[number, number]>`

Original [start, end) character range per token.

`sequenceIds: Array<number | null>`

Which input sequence each token belongs to, or null for added tokens.

`overflowing: Encoding[]`

Sequences that did not fit under a truncation budget.

`interface PaddingOptions {`

Padding configuration.

Properties

`length?: number`

Pad every sequence to this length; omit to pad to the longest in a batch.

`padToMultipleOf?: number`

Round the padded length up to a multiple of this value.

`direction?: Direction`

`padId?: number`

`padTypeId?: number`

`padToken?: string`

`interface TruncationOptions {`

Truncation configuration.

Properties

`maxLength: number`

`strategy?: TruncationStrategy`

`stride?: number`

`direction?: Direction`

`interface AddedToken {`

A token matched literally rather than by the model.

Properties

`id: number`

Vocabulary id.

`content: string`

The literal text to match.

`singleWord: boolean`

Match only when not surrounded by word characters.

`lstrip: boolean`

Extend the match left over whitespace.

`rstrip: boolean`

Extend the match right over whitespace.

`normalized: boolean`

Match against normalized text rather than the raw input.

`special: boolean`

Report as a special token in specialTokensMask.

`interface TiktokenEncoding {`

The pieces a tokenizer needs beyond the ranks table.

Properties

`pattern: string`

Word-splitting pattern applied before merging.

`specialTokens: Record<string, number>`

Literal tokens matched ahead of the model.

Types

`type EncodeInput = string | readonly [string, string]`

A single input, or a sequence and its pair.

`type Direction = 'right' | 'left'`

How to truncate or pad relative to the sequence.

`type TruncationStrategy = 'longest_first' | 'only_first' | 'only_second'`

Truncation strategy for sequences longer than maxLength.

`type AddedTokenOptions = Partial<Omit<AddedToken, 'id' | 'content'>>`

Options accepted when adding a token, beyond its content.

Classes

`class Tokenizer {`

A loaded tokenizer.

Instances are immutable apart from the added vocabulary, which addTokens and addSpecialTokens extend. Encoding holds no state between calls, so one instance is safe to share across concurrent encodes.

Constructors

`constructor(spec: TokenizerSpec, options: TokenizerOptions = {})`

Static Methods

`static fromJSON(source: string | TokenizerSpec, options?: TokenizerOptions): Tokenizer`

Parse a tokenizer.json, given as text or as an already-parsed object.

`static async fromFile(path: string, options?: TokenizerOptions): Promise<Tokenizer>`

Read and parse a tokenizer.json from disk.

`static fromTiktoken(options: TiktokenOptions, tokenizerOptions?: TokenizerOptions): Tokenizer`

Build a tokenizer from a tiktoken ranks table.

Ranks alone do not describe a tokenizer, so pattern is required unless the table came from Tokenizer.fromTiktokenFile with a known encoding name.

`static async fromTiktokenFile( path: string, encoding: string | TiktokenEncoding, options?: TokenizerOptions, ): Promise<Tokenizer>`

Read a .tiktoken ranks file, taking the pattern and special tokens from a published encoding name such as cl100k_base or o200k_base.

Methods

`toJSON(): TokenizerSpec`

The spec this tokenizer encodes with, including any tokens added since.

`vocabSize(withAddedTokens = true): number`

Size of the base vocabulary, optionally including added tokens.

`getVocab(withAddedTokens = true): Record<string, number>`

The vocabulary as a plain object, optionally including added tokens.

`tokenToId(token: string): number | null`

The id for a token, or null if it is not in the vocabulary.

`idToToken(id: number): string | null`

The token for an id, or null if the id is out of vocabulary.

`addTokens(tokens: ReadonlyArray<string | (AddedTokenOptions & { content: string })>): number`

A token already in the vocabulary keeps its id; a new one is assigned the next free id. Returns how many were newly added.

`addSpecialTokens( tokens: ReadonlyArray<string | (AddedTokenOptions & { content: string })>, ): number`

`encode(input: EncodeInput, options: EncodeOptions = {}): Encoding`

Encode one input, or a sequence and its pair.

`encodeBatch(inputs: readonly EncodeInput[], options: EncodeOptions = {}): Encoding[]`

Encode several inputs.

With padding enabled and no explicit length, every result is padded to the longest in this batch — which is what makes a batch stackable into one tensor.

`decode(ids: readonly number[], options: DecodeOptions = {}): string`

Decode ids back into text.

`decodeBatch(batch: ReadonlyArray<readonly number[]>, options?: DecodeOptions): string[]`

Decode several id sequences.

Getters

`get truncation(): TruncationOptions | null`

Truncation applied when a call does not override it.

`get padding(): PaddingOptions | null`

Padding applied when a call does not override it.

Constants

`const TIKTOKEN_ENCODINGS: Readonly<Record<string, TiktokenEncoding>>`

The published tiktoken encodings.

The ranks tables themselves are large downloads rather than baked-in data, so these describe only the surrounding configuration; pair one with a ranks file fetched through fino:model/hub or shipped alongside the model.