The NormalizeText method performs a multi-stage normalization
and cleanup of Unicode text, intended to simplify and standardize input
that may contain visually similar (confusable) characters, diacritical marks,
or inconsistent whitespace. It applies NFKD (Compatibility Decomposition)
normalization converting characters into their canonical forms and
then replaces any confusable Unicode characters with their ASCII equivalents
based on an internal mapping table. This method is similar to
calling the NormalizeString function in the Windows API,
however it performs additional checks and is more strict with how the
input is validated.
If the message text is provided as a byte array, it is validated to
ensure it contains only valid UTF-8 encoding. If the text contains
invalid UTF-8 sequences, this method will fail. You should never pass
in a localized byte array encoded using the system code page using
the StrConv function. If the message text is provided as a string,
it is validated to ensure it contains only well-formed UTF-16. If
the text includes malformed Unicode, embedded nulls, or arbitrary
binary data, the method will fail.
This method helps mitigate issues with characters that are visually
indistinguishable from standard Latin letters, such as using the Cyrillic
"a" (U+0430) instead of the Latin "a" (U+0061). It also removes symbols
(such as emojis, flags and pictographs) which can confuse pattern matching
and removes combining marks that can be used to visually modify characters
or obscure text. Unicode whitespace is trimmed from both ends of the string,
including zero-width and non-breaking spaces.
If you want a normalized version of the text in a message, you can
use the FilteredText property. It performs many of the same
normalizations that this method does, but allows for more flexibility
through options which allow you to control how the message text is
processed and filtered. It is particularly useful for filtering extraneous
HTML from the message, enabling you to display a version of the message
that is safer for the user (such as removing unsafe links from the message
contents).
The FilteredText property is tolerant of malformed Unicode
text in a message, while the NormalizeText method is designed
be stricter with how it processes the text. If there are any
malformed characters (such as unpaired surrogates, over-long or
truncated UTF-8 characters, invalid continuation sequences or
characters outside of valid Unicode code points), this method will
fail and throw an exception which must be handled appropriately by
your application. The intent is to enable you to detect what may be
intentionally malformed text in the contents of a message or a header
value.
Note that standard Visual Basic controls such as Label
and TextBox do not fully support displaying Unicode
characters, especially those outside the Basic Multilingual Plane
(such as emoji or mathematical symbols). These controls attempt to
convert Unicode strings to ANSI using the system locale, which can
result in garbled or missing characters. To correctly display full
Unicode text, you must use controls that support Unicode natively,
such as those in the Microsoft Forms 2.0 Object Library (FM20.DLL)
with appropriate font settings, or render the text using Windows API
functions.
A string containing bold mathematical Unicode characters is
normalized to produce a simplified, plain text version. If you
wanted to display the obfuscated Unicode text in this example,
you would need to use a Unicode-aware control such as the
TextBox control in the Microsoft Forms 2.0 Object Library.
You would also need to select a font which is capable of displaying
Unicode math symbols, such as the Cambria Math font.
The advantage of using the NormalizeText method is that it
will convert this type of Unicode text to something which is
displayable without requiring specific UI components and fonts.
Dim sUnicode As String
sUnicode = ChrW(&HD835) & ChrW(&HDC07) & ChrW(&HD835) & ChrW(&HDC1E) & ChrW(&HD835) & ChrW(&HDC25) & _
ChrW(&HD835) & ChrW(&HDC25) & ChrW(&HD835) & ChrW(&HDC28) & "!"
Dim sPlainText As String
sPlainText = MailMessage1.NormalizeText(sUnicode)
MsgBox sPlainText, vbInformation, "Normalized Text"