Tokenization Models

Tokenization models are frameworks that define how Text-Based Tokens are structured. Xyxyx introduces two foundational tokenization models — 1x1 and A4 —, which adhere to different intended use cases.

Model	Description
1x1	A model for tokenizing short text records
A4	A model for tokenizing long text records