Library Moduleswift-transformers 0.1.17Tokenizers
Tokenizers
import Tokenizers
Module information
- Declarations
- 151
- Symbols
- 202
import Tokenizers
protocol Decoder
protocol Normalizer
protocol PostProcessor
protocol PreTokenizer
protocol PreTrainedTokenizerModel : TokenizingModel
A tokenizer model that is set up with Hub configuration data
protocol Tokenizer
protocol TokenizingModel
struct AutoTokenizer
class BertTokenizer
enum ChatTemplateArgument
struct LeavesWithCommonPrefixIterator<T> where T : Hashable
enum PreTokenizerOption
class PreTrainedTokenizer
enum SplitDelimiterBehavior
struct TokenLattice
Implements a TokenLattice to implement the Viterbi algorithm We could make it generic so TokenLatticeNode stores an opaque type, but it’s overkill right now. Based on https://github.com/huggingface/tokenizers/blob/b58227c7f1ccf8b73ee2268354336da56d91e492/tokenizers/src/models/unigram/lattice.rs#L137 and https://github.com/xenova/transformers.js/blob/b07336d8f7ff57453cc164cc68aead2a79cbd57e/src/utils/data-structures.js#L269C28-L269C28
enum TokenizerError
struct Trie<T> where T : Hashable
class TrieNode<T> where T : Hashable
typealias Message = [String : Any]
typealias PreTokenizerOptions = Set<PreTokenizerOption>
typealias ToolSpec = [String : Any]
import Generation
import Hub
import HubCLI
import Models
import TensorUtils
import TransformersCLI