Function tokenize

tokenize(doc: string | undefined, uchars: string[], BOS: number): number[]
Tokenize a document string into an array of token ids, surrounded by BOS.
Parameters
- doc: string | undefined
  The document string.
- uchars: string[]
  Sorted unique characters (from buildTokenizer).
- BOS: number
  The BOS token id.
Returns number[]
Array of token ids: [BOS, ...charIds, BOS].
- Defined in tokenizer.ts:53