The FilterUnicode method filters Unicode text to
normalize visually confusable characters, remove symbolic or
non-textual code points (such as emojis), and standardize whitespace.
It is designed to produce safe, readable output by converting
characters to their most recognizable and printable forms.
This method performs multi-stage normalization and filtering
of Unicode text, designed to produce clean, printable output from
potentially deceptive or malformed input. It applies Compatibility
Decomposition (NFKD) to break down complex characters into their
canonical equivalents, such as replacing ligatures or accented
characters with their base ASCII forms. It also removes Unicode
combining marks that could be used to visually obscure or alter text.
Leading and trailing Unicode whitespace, including zero-width and
non-breaking spaces, is trimmed to improve consistency.
To mitigate spoofing and visual ambiguity, it replaces confusable
Unicode homoglyphs with their standard Latin counterparts. For example,
the Cyrillic small letter "a" (U+0430) is replaced with the Latin "a" (U+0061).
It also detects and maps mathematical alphanumeric symbols and fullwidth
characters to their ASCII equivalents. Non-textual code points, including
emojis, pictographs, private-use characters, and deprecated symbols are
removed from the input string. This is especially useful in security-sensitive
contexts, such as sanitizing domain names, email addresses, usernames,
or filenames. It can help prevent spoofing, phishing, and filter evasion
through the use of homoglyphs or invisible characters.
If the input text is provided as a byte array, it
is validated to ensure it contains only valid UTF-8 encoding. It
removes invalid character sequences such as overlong encodings,
truncated multibyte sequences, illegal continuation bytes, and code
points outside the valid Unicode range. You should never pass in a
localized ANSI string encoded using the system code page using the
StrConv function. If you wish to convert a native string type to a
valid UTF-8 encoded byte array, use the StrToUtf8 method.
If the input text is provided as a string, it is
validated to ensure it contains only well-formed UTF-16. This includes
removing unpaired surrogates, invalid code points, unprintable control
characters, and private-use characters. If the input includes malformed
Unicode, embedded nulls, or arbitrary binary data, it may result in
the loss of non-text content during filtering.
This method is deliberately biased toward security rather than readability
and can affect the output when applied to text containing legitimate non-Latin
characters. It is intended for use on text that is expected to contain only
printable plaintext characters. For example, it will filter and normalize text
containing certain Greek or Cyrillic characters that could be used to spoof
ASCII letters or URL delimiters. While this helps mitigate obfuscated or
malicious content, it can also make legitimate text in other languages unreadable.
The standard Visual Basic controls such as Label
and TextBox do not fully support displaying Unicode
characters, especially those outside the Basic Multilingual Plane
(such as emoji or mathematical symbols). These controls attempt to
convert Unicode strings to ANSI using the system locale, which can
result in garbled or missing characters. To correctly display full
Unicode text, you must use controls that support Unicode natively,
such as those in the Microsoft Forms 2.0 Object Library (FM20.DLL)
with appropriate font settings, or render the text using Windows API
functions.
This example demonstrates how to filter and convert text using
both UTF-16 and UTF-8 representations. A string containing bold
mathematical Unicode characters is filtered to produce a simplified,
plain text version. The string is then converted to a UTF-8 encoded
byte array, which is also filtered. Finally, the UTF-8 data is
converted back to a string and compared to the original to verify
round-trip accuracy. This illustrates how the methods can be used
interchangeably with both native strings and UTF-8 encoded data.
If you wanted to display the obfuscated Unicode text in this
example, you would need to use a Unicode-aware control such as the
TextBox control in the Microsoft Forms 2.0 Object Library.
You would also need to select a font which is capable of displaying
Unicode math symbols, such as the Cambria Math font.
The advantage of using the FilterUnicode method is that it will convert
this type of Unicode text to something which is displayable without
requiring specific UI components and fonts.
Dim sUnicode As String
sUnicode = ChrW(&HD835) & ChrW(&HDC07) & ChrW(&HD835) & ChrW(&HDC1E) & ChrW(&HD835) & ChrW(&HDC25) & _
ChrW(&HD835) & ChrW(&HDC25) & ChrW(&HD835) & ChrW(&HDC28) & "!"
Dim sFiltered As String
sFiltered = FileEncoder1.FilterUnicode(sUnicode)
MsgBox sFiltered, vbInformation, "UTF-16"
Dim arrUtf8() As Byte
arrUtf8 = FileEncoder1.StrToUtf8(sUnicode)
sFiltered = FileEncoder1.FilterUnicode(arrUtf8)
MsgBox sFiltered, vbInformation, "UTF-8"
Dim sConverted As String
sConverted = FileEncoder1.Utf8ToStr(arrUtf8)
If sConverted = sUnicode Then
MsgBox "The conversion between UTF-16 and UTF-8 was successful", vbInformation
Else
MsgBox "The conversion between UTF-16 and UTF-8 failed", vbExclamation
End If