Structureswift-transformers 0.1.17Tokenizers
TokenLattice
Implements a TokenLattice to implement the Viterbi algorithm We could make it generic so TokenLatticeNode stores an opaque type, but it’s overkill right now. Based on https://github.com/huggingface/tokenizers/blob/b58227c7f1ccf8b73ee2268354336da56d91e492/tokenizers/src/models/unigram/lattice.rs#L137 and https://github.com/xenova/transformers.js/blob/b07336d8f7ff57453cc164cc68aead2a79cbd57e/src/utils/data-structures.js#L269C28-L269C28
struct TokenLattice