Instance Propertyswift-transformers 0.1.17Tokenizers
tokenIds
TokenLattice.swift:117var tokenIds: [Int] { get }
var tokenIds: [Int] { get }
import Tokenizers
struct TokenLattice
Implements a TokenLattice to implement the Viterbi algorithm We could make it generic so TokenLatticeNode stores an opaque type, but it’s overkill right now. Based on https://github.com/huggingface/tokenizers/blob/b58227c7f1ccf8b73ee2268354336da56d91e492/tokenizers/src/models/unigram/lattice.rs#L137 and https://github.com/xenova/transformers.js/blob/b07336d8f7ff57453cc164cc68aead2a79cbd57e/src/utils/data-structures.js#L269C28-L269C28
@frozen struct Int
A signed integer value type.
var tokens: [String] { get }
mutating func insert(startOffset: Int, length: Int, score: Float, tokenId: Int)
Insert a new token into the node lattice.