Function buildTokenizer

buildTokenizer(
docs: string[],
): { BOS: number; uchars: string[]; vocabSize: number }
Build a character-level tokenizer from a list of documents.
Parameters
- docs: string[]
  Array of document strings.
Returns { BOS: number; uchars: string[]; vocabSize: number }
uchars (sorted unique characters), BOS token id, and vocabSize.
- Defined in tokenizer.ts:19