If the method succeeds, the return value is the estimated
number of tokens in the specified string. If there are no text
characters, or the string only contains whitespace, the method
will return zero. If the method fails, the return value is -1.
To get extended error information, check the value of the
LastError property.
The EstimateTokens method provides an approximate count of
the number of tokens that would be required to represent the specified
text for the current provider. This can be useful when estimating prompt
size and ensuring that requests remain within model limits. This method
does not require a connection to a provider and can be used independently
to estimate token usage before establishing a connection or sending
a request.
The MessageText parameter must contain valid text and cannot
contain arbitrary binary data or embedded null characters. If malformed
Unicode is detected in the string, the method will fail and return -1,
with the last error code set to stErrorInvalidUnicode. If
the string consists only of whitespace characters, including non-breaking
spaces, the method will return zero.
A token is a unit of text used internally by large language models.
Tokens are not the same as characters or words. In general, a token may
represent a single word, part of a word, or a short sequence of characters.
For example, common words may be represented as a single token, while
less common or longer words may be split into multiple tokens. Punctuation
and whitespace may also be represented as separate tokens.
As a general rule of thumb, one token corresponds to approximately
three to four characters of English text. This can vary significantly
depending on the content, language, and tokenization rules used by the
provider.
Most providers distinguish between input tokens (the text sent to the
model) and output tokens (the text generated by the model in response).
Usage and billing are typically based on the total number of tokens
processed, including both the prompt and the generated response. Monitoring
token usage can help applications control costs and ensure that requests
remain within the limits imposed by the selected model.
The tokenization rules used by each provider can vary, and the exact
number of tokens used in a request is determined by the provider at runtime.
The value returned by this method is an estimate only and should not be
relied on as an exact measurement. The value of the Provider
property can affect how the method calculates the estimated number of
tokens in the string.
To determine how many input and output tokens have been used in a
session, use the TotalTokens property.