| |
Working with large language models (LLMs) introduces a number of new
terms and concepts that may not be familiar, even to experienced developers.
This page provides clear, practical definitions of the terminology used
throughout this documentation, helping you get up to speed quickly without unnecessary
complexity.
Sections
Basic Concepts
Provider
A provider is a service which hosts one or more language models and
processes requests from your application. This can include cloud-based
services, as well as locally hosted environments running on your system
or within a private network. Examples of providers include OpenAI
(ChatGPT), Anthropic (Claude) and Google (Gemini). Applications such
as LM Studio and Ollama can be used to host models locally.
The control uses a common interface to work with multiple providers,
allowing your application to switch between services with minimal
changes to your code. Providers may differ in terms of available models,
capabilities, performance and usage costs. For an overview of currently
supported providers, please refer to the
providers page.
Model
A model is a specific language model provided by a service. It is the
component responsible for generating responses based on the input that
your application provides. Each provider may offer multiple models, with
different capabilities, performance characteristics and token limits.
Models are typically identified by name, which must be specified when
creating a session or sending a request. For example, model names such as
gpt-5.4-nano, claude-3-haiku or
gemini-2.5-flash may be used, depending on the provider.
The control can be used to retrieve a list of available models from a
provider at runtime, allowing your application to dynamically select an
appropriate model or present options to the user.
Because models can vary significantly between providers, the
control provides a consistent interface for selecting and using them. This makes
it easier to experiment with different options and choose the model that
best fits your requirements.
API Key
An API key is a unique identifier used to authenticate requests to a
provider. It is typically issued by the service provider and is required
to access their API. Depending on how your application is designed, the
API key may be supplied directly, read from an environment variable, or
retrieved from a secure location such as the Windows Credential Manager.
Because an API key grants access to a provider account, it should be
treated as sensitive information. It is not recommended to embed API keys
directly in application source code, particularly in applications that
are distributed to end users or stored in source control systems where
the key may be exposed. Instead, consider using environment variables or
secure storage mechanisms to protect the key.
Use of a provider's API may incur usage-based charges. It is the
responsibility of the developer to monitor usage and manage any costs
associated with their API key. Developers are encouraged to review
provider pricing policies and configure usage limits or alerts where
available.
Some applications may allow end users to provide their own API key.
In this case, the user is responsible for managing their usage and any
associated costs.
Endpoint / Base URL
An endpoint, or base URL, is the network address used to communicate
with a provider's API. It defines where requests are sent when your
application interacts with a language model.
Most providers have a default endpoint that is used automatically by
the control, so it is not usually necessary to specify this value explicitly.
However, some providers require a custom endpoint, such as when using a
dedicated service instance or a locally hosted model.
For example, cloud services may use provider-specific endpoints, while
local environments such as LM Studio or Ollama typically use a base URL
like http://localhost:port to accept requests from your
application. For more information about the endpoint used by a
specific provider, please refer to their documentation.
Session
A session represents an active connection between your application
and a provider. It maintains configuration settings and conversation
history for a sequence of requests. Using a session allows your application
to perform multiple requests while preserving context, enabling more
natural, conversational interactions with the model.
The Connect method is used to create a new session,
establishing a connection to the specified service provider and
selecting the model.
Inference
Inference is the process of a language model generating a
response based on the input it receives. When your application
sends a prompt to a model and receives output, the model is
performing inference.
During inference, the model analyzes the input text, considers
the available context and generates tokens one at a time until the
response is complete. The speed and cost of inference can vary
depending on the model, the amount of context provided and whether
advanced reasoning features are enabled.
Inference is different from training. Training is the process
of building or updating a model using large datasets, while
inference refers only to using an already trained model to
generate output.
Training
Training is the process used to create or improve a language
model. During training, the model is exposed to very large amounts
of text and other data so it can learn patterns in language,
relationships between words and how information is typically
structured. This process allows the model to generate responses
that resemble natural human communication.
Training is different from inference. Training is the process
of building or updating a model, while inference refers to using
an already trained model to generate responses. When your
application sends a prompt and receives output from a model, the
model is performing inference rather than training.
Prompts and Messages
Prompt
A prompt is the input text provided to a language model. It represents
the question, instruction or request that your application sends to the
model in order to generate a response. Prompts can be simple, such as a
single sentence, or more detailed, including instructions that guide how
the model should respond. The quality and clarity of the prompt can have
a significant impact on the output that is generated.
A prompt may be provided as a single input, or as part of
an ongoing session. The control manages the underlying message history
and context for you, allowing your application to maintain conversations
without needing to manually track previous requests and responses.
System Prompt
A system prompt is a special type of instruction that defines how the
model should behave when generating responses. It is typically used to
establish context, set the tone of the output, or provide guidance on how
the model should interpret user input. It might instruct the model
to respond in a specific style, limit the level of detail in its
answers, or follow a particular format.
System prompts are often used to control the overall behavior of a
session rather than responding to a single request. For example, a
simple instruction to control the format of the model's output could
look something like this:
Respond using plain text only. Do not use Markdown formatting,
special characters or emojis. Limit output to standard ASCII text.
This helps ensure that responses are returned in a consistent format
that is easier for your application to process.
Not all providers support system prompts in the same way, and some
models may not support them at all. In these cases, instructions may need
to be included as part of the standard input instead. The control handles
these differences internally where possible, allowing your application to
specify system instructions without needing to account for provider-specific
behavior.
Prompt Injection
Prompt injection is a technique where input provided to a language
model is crafted to override or manipulate its intended behavior. This
can occur when user-supplied input includes instructions that attempt to
change how the model responds or bypass system-level guidance.
For example, a user might include text that instructs the model to
ignore previous instructions or reveal information that should not be
disclosed. Because language models interpret all input as text, they may
not distinguish between trusted instructions and untrusted user input.
When developing applications that accept external input, it is
important to treat all user-provided content as untrusted. Sensitive
information should not be included in prompts, and applications should
validate or constrain input where appropriate. The control provides
mechanisms for defining system-level behavior, but it is the
responsibility of the application to ensure that user input is handled
safely.
Message
A message is a unit of communication within a conversation. Each
message contains a piece of text and is associated with a role, such as
user, assistant or system.
Messages are used to represent the interaction between your
application and the model. For example, a user message contains the
input provided by your application, while an assistant message contains
the response generated by the model. The control
maintains the conversation history for you, allowing subsequent requests
to include the context of previous interactions without requiring your
application to explicitly track each message.
Message Role
A message role defines the purpose of a message within a conversation.
Each message is associated with a role that indicates how the model
should interpret its content.
The most common roles are user, assistant and
system. A user message represents input provided by your
application, an assistant message represents the response generated by
the model, and a system message provides instructions that guide the
model's behavior.
Not all providers handle message roles in the same way, and some
models may not fully support all roles. The control manages these
differences internally where possible, allowing your application to work
with a consistent set of roles across providers.
Message Turn
A message turn represents a single interaction between your
application and the model. It typically consists of a user message
(input) and the corresponding assistant message (response). Each turn
builds on the previous conversation, allowing the model to generate
context-aware responses based on earlier interactions within the
session.
Message History
Message history refers to the collection of previous message turns
maintained as part of a client session. The history represents the
ongoing conversation between your application and the model and is used
to provide conversational context for subsequent requests.
Each entry in the message history typically consists of both the
original user message (prompt) and the corresponding assistant response
generated by the model. Together, these message turns form the complete
conversational state for the session.
The message history is managed internally by the control.
When conversation history is enabled, previous message turns are
automatically included as context for future requests, allowing the
model to generate responses that are aware of earlier interactions.
The current message history can be enumerated, exported or imported
using methods such as GetMessage,
ExportHistory and ImportHistory.
Hallucination
A hallucination is output generated by a language model that
appears plausible or well-formed, but is incorrect, misleading or
fabricated. This can include inaccurate factual information,
nonexistent references, incorrect code examples or responses that
confidently present unsupported conclusions.
Hallucinations can occur because language models generate output
based on patterns learned during training rather than performing
independent verification of the information they produce. As a
result, a response may sound convincing even when parts of the
content are incomplete or inaccurate.
The likelihood of hallucinations can often be reduced by
providing clear, specific prompts and limiting ambiguous or
open-ended instructions. Supplying relevant context, requesting
concise responses and asking the model to avoid speculation may also
improve reliability. For applications that require accurate or
verifiable information, responses generated by a model should always
be validated before being trusted or used in production systems.
Hallucinations are a normal limitation of language models and can
occur with both cloud-based and locally hosted models. The frequency
and severity of hallucinations may vary depending on the model, the
prompt and the complexity of the request. Applications that depend
on accurate, complete or safety-critical information should always
independently verify model output before relying on it.
Latency
Latency refers to the amount of time required for a model to
begin generating or complete a response after a request has been
sent. Higher latency results in slower responses, while lower
latency improves responsiveness and can provide a better user
experience.
The latency of a request can vary depending on several factors,
including the selected model, the size of the prompt and
conversation history, network conditions and the current load on the
provider's service. Requests that involve reasoning, large context
windows or longer generated responses may take additional time to
process.
Locally hosted models may also experience increased latency
depending on the performance of the system hardware, such as CPU
speed, GPU capability and available memory. Cloud-based providers
may generally offer faster performance, but can still experience
delays during periods of high demand.
Applications that interact with language models should be
designed to account for variable latency. This can involve using
visual indicators in the application and delegating complex requests
to background worker threads. The default timeout period for client
sessions is 60 seconds; however, requests which require more
reasoning effort may require larger timeout values.
Tokens and Limits
Token
A token is a unit of text used by a language model when processing
input and generating output. Tokens are not the same as characters or
words; depending on the text, a single word may be represented as one or
more tokens. For example, common words may be represented as a single
token, while longer or less common words may be split into multiple
tokens. Punctuation, spaces and formatting may also be counted as
tokens.
Tokens are important because they are used to measure both the size of
a request and the amount of generated output. Most models have limits on
the total number of tokens that can be processed in a single request, and
usage-based pricing is typically based on the number of tokens used.
While tokens are primarily used to measure text input and output,
other types of content such as images, audio and video may be processed
differently depending on the model. In these cases, usage may be measured
using different units or converted into tokens internally by the
provider.
The control provides the EstimateTokens method that can be used to estimate the number of
tokens in a string before sending a request. However, token counts are
ultimately determined by the provider, and the reported usage may differ
from the estimates. Token usage information is provided as part of the
response, allowing your application to monitor usage and remain within
model limits.
Context / Context Window
The context, or context window, refers to the total amount of text that
a model can consider when generating a response. This includes the input
provided by your application, any prior conversation history and the
generated output. The context is measured in tokens, and all messages
within a session contribute to the total context size. As a conversation
continues, the amount of context increases as additional messages are added.
The control manages the conversation history for you as
part of the session. This allows the model to generate responses that
take previous interactions into account without requiring your
application to explicitly manage the context.
Context Limit
The context limit is the maximum number of tokens that a model can
process in a single request. This limit includes both the input tokens
and the tokens generated in the response. If the total number of tokens
exceeds the model's context limit, the request may fail or be truncated
depending on the provider. For this reason, it is important to be aware
of the context size when working with longer inputs or extended
conversations.
Context limits can vary significantly between models and providers.
Modern cloud-based models often support context windows ranging from
tens of thousands to hundreds of thousands of tokens. Locally hosted
models may have smaller default limits, but these can often be adjusted
depending on the available system resources, such as memory and GPU
capacity. Increasing the context size for local models may impact
performance and resource usage.
As a general guideline, one token typically represents a few
characters of text. For example, a context limit of 128,000 tokens can
represent tens of thousands of words, or several hundred pages of
text. This is only a rough approximation, as the exact number of
tokens depends on the content and how it is processed by the model.
The control provides information about model limits and token usage,
which can be used to help ensure that requests remain within supported
constraints.
Token Usage
Token usage refers to the number of tokens processed as part of a
request. This typically includes both the input tokens provided by your
application and the tokens generated in the response. The total number
of tokens used depends not only on the size of the input, but also on
the complexity and length of the generated output. More detailed
responses, or those that involve reasoning or multi-step processing,
may use significantly more tokens.
Tracking token usage is important because it is commonly used to
determine usage-based costs. It can also be used to monitor how close
a request is to a model's context limit.
The control has several properties, such as TotalTokens and
ContextUsed, which can return information about token usage
for the session.
Generation Settings
Temperature
Temperature is a parameter that controls the randomness of the model's
output. Lower values produce more predictable and consistent responses,
while higher values allow for more varied and creative output.
For example, a low temperature setting may result in more direct and
deterministic answers, which can be useful for tasks such as generating
structured data or answering factual questions. Higher values may produce
more diverse responses, which can be useful for creative tasks such as
writing or brainstorming.
In most cases, the default temperature value is appropriate and does
not need to be adjusted. Changing this value can significantly affect the
style and consistency of the output, so it is generally recommended only
when you have a specific reason to do so and understand the impact it may
have on the results.
Top-p (Nucleus Sampling)
Top-p, also known as nucleus sampling, is a parameter that controls how
the model selects the next token when generating a response. Instead of
considering all possible tokens, the model selects from a subset whose
combined probability meets a specified threshold.
Lower values restrict the model to a smaller set of more likely tokens,
resulting in more focused and predictable output. Higher values allow a
broader range of tokens to be considered, which can produce more varied
responses. In most cases, adjusting the top-p value is not necessary. It is
generally recommended to use the default setting unless you have a
specific need to fine-tune how the model generates output.
Top-k
Top-k is a parameter that limits the number of possible tokens the
model considers when generating the next part of a response. Instead of
evaluating all potential tokens, the model selects from the top-k most
likely options. Lower values restrict the model to a smaller set of highly probable
tokens, resulting in more consistent output. Higher values allow more
variation by increasing the number of possible choices.
Like top-p, this parameter is typically used for advanced control over
the generation process. In most applications, the default value is
appropriate and does not need to be modified.
Reasoning
Reasoning refers to a model's ability to process more complex or
multi-step tasks, such as analyzing information, solving problems or
following detailed instructions. Models that support reasoning are
typically designed to produce more structured and thoughtful responses.
For example, reasoning may be useful when summarizing complex
content or generating step-by-step instructions.
In practice, reasoning may result in responses that take longer to
generate and use more tokens, as the model performs additional internal
processing before producing an answer. This can be useful for tasks that
require accuracy or step-by-step logic, but may not be necessary for
simpler requests.
Some providers offer models or options that enable or adjust reasoning
behavior. Reasoning can be controlled using the ReasoningLevel
property
when supported by the selected model. Because this feature can impact
performance and cost, it is generally recommended to use the default
settings unless your application requires more advanced behavior.
Other Concepts
Structured Output
Structured output refers to responses that are generated in a
predictable format, such as JSON or another machine-readable structure.
This allows your application to reliably parse and process the model's
output rather than treating it as plain text. For example, a model may
be instructed to return data in JSON format with specific fields,
making it easier to extract values and integrate the response into
your application.
In practice, structured output is most commonly generated using
formats such as JSON, which are widely supported and easier to validate.
Other formats, such as XML, Markdown or TOML, may also be used depending
on the application. However, simpler and more strictly defined formats
tend to produce more consistent results, particularly when the output
must be parsed programmatically. More verbose formats, such as XML, may
require additional validation to ensure the structure is correct.
The control provides support for structured output where available,
allowing your application to request and receive formatted responses in
a consistent manner. However, it is the responsibility of the application
to parse and process that output based on its specific requirements.
Some models may include Markdown formatting in their responses, even
when a structured format has not been explicitly requested. If plain
text or a specific output format is required, this should be clearly
specified in the prompt or system instructions.
Rate Limit
A rate limit is a restriction imposed by a provider on how many
requests can be made within a given period of time. If this limit is
exceeded, additional requests may be rejected or delayed. Rate limits
vary between providers and may depend on factors such as account type
or usage level. Applications should be designed to handle rate limit
responses gracefully, for example by retrying requests after a delay.
To reduce the likelihood of hitting rate limits, applications can use
strategies such as limiting the frequency of requests, batching work
where appropriate, caching responses when possible and avoiding
unnecessary or repeated requests. Implementing retry logic with a delay
or backoff strategy can also help ensure that temporary limits do not
disrupt normal operation.
Local Model
A local model is a language model that runs on your own system or
within a private network, rather than being hosted by a cloud-based
provider. Local models can be accessed using applications or services
that expose an API endpoint compatible with the control.
Using a local model can provide advantages such as increased privacy,
reduced dependency on external services and the ability to operate
without an internet connection. It may also allow greater control over
model configuration and resource usage. However, local models typically
require more system resources and may have lower performance or smaller
context limits compared to cloud-based models. Running a model locally
may also require additional setup and configuration.
Applications such as LM Studio and Ollama can be used to host and run
models locally, providing an environment that can be used for development
and testing, or for use in private deployments.
Quantization
Quantization is the process of reducing the precision used to
store a model's internal data. This decreases memory requirements
and can improve performance, allowing larger models to run on
consumer hardware. Most locally hosted models are distributed in a
quantized format, which makes them smaller and faster to load while
preserving most of their original capabilities. Different
quantization levels provide different trade-offs between model size,
speed, memory usage, and response quality.
Developers using cloud-based AI providers generally do not need
to be concerned with quantization because it is handled by the
service provider. The term is most commonly encountered when
downloading and running models locally using tools such as LM Studio
or Ollama.
See Also
LlmClient Control Overview
LlmClient ActiveX Control
Supported Providers
|
|