FTS5TokenizerDescriptor

For example:

try db.create(virtualTable: "book", using: FTS5()) { t in
    t.tokenizer = .unicode61() // FTS5TokenizerDescriptor
}

Related SQLite documentation: https://www.sqlite.org/fts5.html#tokenizers

Creating Tokenizer Descriptors

• init(components: [String])

  Creates an FTS5 tokenizer descriptor.

• static func ascii(separators: Set<Character>, tokenCharacters: Set<Character>) -> FTS5TokenizerDescriptor

  The “ascii” tokenizer.

• static func porter(wrapping: FTS5TokenizerDescriptor?) -> FTS5TokenizerDescriptor

  The “porter” tokenizer.

• static func unicode61(diacritics: FTS5.Diacritics, categories: String, separators: Set<Character>, tokenCharacters: Set<Character>) -> FTS5TokenizerDescriptor

  The “unicode61” tokenizer.

• enum Diacritics

  Options for Latin script characters. Matches the raw “remove_diacritics” tokenizer argument.

Instantiating Tokenizers

• func makeTokenizer(FTS5TokenizerDescriptor) throws -> any FTS5Tokenizer

  Creates an FTS5 tokenizer, given its descriptor.