What problem does the SentencePiece library solve that earlier tokenizers did not?
-
A
It eliminates the need for any vocabulary
-
B
It performs tokenization without relying on language-specific whitespace rules, treating raw text as input
-
C
It automatically detects the language of the input text
-
D
It converts tokens to dense vector embeddings