Tokenization Models
Last updated
Last updated
Tokenization models are frameworks that define how are structured. Xyxyx introduces two foundational tokenization models — 1x1 and A4 —, which adhere to different intended use cases.
A model for tokenizing short text records
A model for tokenizing long text records