If the function succeeds, the return value is the estimated
number of tokens in the specified string. If there are no text
characters, or the string only contains whitespace, the
function will return zero. If the function fails, the return
value is LLM_ERROR. To get extended error information, call
LlmGetLastError.
The LlmEstimateTokenCount function provides an approximate count
of the number of tokens that would be required to represent the specified
text for a given provider. This can be useful when estimating prompt size
and ensuring that requests remain within model limits.
The lpszMessageText parameter must contain valid Unicode
text and cannot contain arbitrary binary data or embedded null
characters. If malformed Unicode is detected, the function will fail
and return LLM_ERROR, with the last error code set to
ST_ERROR_INVALID_UNICODE. If the string consists only of whitespace
characters, including non-breaking spaces, the function will return
zero.
The ANSI version of this function expects the input string to
contain either ASCII or UTF-8 encoded text. Strings encoded using a
localized ANSI code page are not supported and may fail validation if
they contain non-ASCII characters. To ensure internationalized text is
processed correctly, it is recommended that you use the Unicode
version of this function whenever possible.
A token is a unit of text used internally by large language models. Tokens
are not the same as characters or words. In general, a token may represent a
single word, part of a word, or a short sequence of characters. For example,
common words may be represented as a single token, while less common or
longer words may be split into multiple tokens. Punctuation and whitespace
may also be represented as separate tokens.
As a general rule of thumb, one token corresponds to approximately
three to four characters of English text. This can vary significantly depending
on the content, language, and tokenization rules used by the provider.
Most providers distinguish between input tokens (the text sent to the
model) and output tokens (the text generated by the model in response).
Usage and billing are typically based on the total number of tokens
processed, including both the prompt and the generated response.
Monitoring token usage can help applications control costs and ensure that
requests remain within the limits imposed by the selected model.
The tokenization rules used by each provider can vary, and the exact number
of tokens used in a request is determined by the provider at runtime. The
value returned by this function is an estimate only and should not be relied
on as an exact measurement.
This function does not require an active client session and can be used
independently to estimate token usage before establishing a connection or
sending a request.
To determine how many input and output tokens have been used in a
session, use the LlmGetTokenUsage function.
LPCTSTR lpszMessageText = _T("Explain how a TCP connection is established.");
LONG nTokens = LlmEstimateTokenCount(LLM_PROVIDER_OPENAI, lpszMessageText, -1);
if (nTokens != LLM_ERROR)
{
_tprintf(_T("Estimated token count: %d\n"), nTokens);
}
else
{
DWORD dwError = LlmGetLastError();
_tprintf(_T("Unable to estimate token count, error 0x%08lx\n"), dwError);
}