The LlmSetCacheTime function controls how long the client
caches provider and model metadata. This includes information such as
the list of available models and details about the currently selected
model.
Caching reduces the number of network requests required when
retrieving model lists or model information, which can improve
performance and reduce latency. However, because model availability
and capabilities may change over time, cached data may become stale
if it is retained for extended periods.
Changing this value will have no effect if the LLM_OPTION_NOCACHE
option has been specified.
Setting the cache time to zero disables caching and is equivalent
to specifying the LLM_OPTION_NOCACHE option. Note that this differs
from specifying a cache time of zero in the LLM_SESSION structure
during initialization, where a value of zero indicates that a
default cache lifetime will be used.
The cache time is specified in seconds. If a value greater than or
equal to 1000 is specified, it is assumed that the value may have been
provided in milliseconds, and it will be automatically scaled to
seconds. The specified cache time is clamped to an internal minimum and
maximum value if it falls outside the supported range. The default
cache time is one hour.
In most cases, the default cache time provides a good balance between
performance and accuracy. Applications that frequently query model
information or require real-time updates may choose to reduce or
disable caching.