Fast UTF-8 handling for modern C++ with support for legacy C

Fast UTF-8 handling for modern C++ with support for legacy C ♦ Overview

What’s here An introduction to the FastUtf8::Uniseries C++ class; A reason to target just UTF-8 for C/C++ internationalization: it slashes case folding bloat associated with unnecessary encodings; An introduction to the C-compatible functions that do the heavy lifting for FastUtf8 ; and A discussion of included methods for expressive queries against semistructured internationalized content and their outstanding performance results relative to prior methods.

The footprint of the global datasphere is estimated to be in the hundreds of zettabytes. Most of it is unstructured or semistructured data. There are hundreds of natural languages represented on the Internet, with less than half of websites in English. Mature programming languages, including C/C++, stem from a time frame when the Internet wasn’t this way.

Providing for natural language support, in steps composed entirely via imperative programming, can lead to layers of sophisticated code. For example, to pick out a title from a semistructured list comprising titles of books and corresponding authors, one step might involve case folding for a given author’s name. Another step might involve separating the list into discrete lines based on tokens. Yet another step, where mixed natural languages are involved, might include matching various further tokens used in those languages. The result: slow, kludgy, narrowly focused code. Such code has worked in the past, and it still can – but isn’t there a better fit for the scope of data we have today? In the world of structured data, such as what’s associated with relational databases, declarative queries are the norm. Expressive syntax is handled via fast, tested methods that hide the complexities of finding content . As distributed systems consultant Martin Kleppmann puts it, “In a declarative query language, like SQL or relational algebra, you just specify the pattern of the data you want... but not how to achieve that goal.” Can any sort of declarative query be applied, in C/C++, to semistructured data? Even modest steps toward making it happen could result in more reliable and fast internationalized applications.

Declarative queries for semistructured natural language

Aiming for a rich declarative arrangement, like SQL but targeted at semistructured data, may not be helpful. There are a great many natural languages that C/C++ code might parse, and each may have a way of its own for delimiting content. On the other hand, to a certain degree, semistructured data formats tend to be shared across those languages. A declarative query that works for one locale can work for another, so long as there’s enough flexibility in the query, and in the mechanism enabling it, to account for cross-locale distinctions. For that reason, any such mechanism may best follow this rule of thumb: the simpler its interface, the better.

Neither an SQL-style arrangement nor a multi-step method, such as in the book title example of the second paragraph above, is particularly simple. That is, methods for handling semistructured data in C/C++ have centered around carefully-arranged loops that a developer must be able to comprehend and modify, for every little change in the data or its format. Still, the very idea of enabling expressive queries for semistructured data has seldom arisen. Over twenty years ago, knowledge base consultant Mike Bergman put the situation this way : Generally, most academic, open source, or other attention to these problems has been at the superficial level of resolving schema or definitions or units. Totally lacking in the entire thrust for a semi-structured data paradigm has been the creation of adequate processing engines... You know, it is very strange. Tremendous effort goes into data representations like XML, but when it comes to positing or designing engines for manipulating that data the approach is to clone kludgy workarounds on to existing relational DBMSs or text search engines. Making Validation Make Sense

For the second question, an impulsive answer would be to validate every byte every time it’s read. If we’d been considering ASCII content, the possibility would hardly have come to mind. The conditions that could clobber content are roughly alike, for both ASCII and UTF-8, and the idea of detection based on the content itself is worth considering – but is a routine that needs to quickly return results most efficacious for the purpose?

My suggestion is that UTF-8 validation can indeed be useful, and that a validation routine can be regularly invoked within any software that prioritizes reliable content handling. I also suggest performing the validation in two conditions: when UTF-8 content is first stored in memory, and via a low-priority thread that can find and revalidate the UTF-8 content already stored in memory. I’m making these suggestions because input provided via a user or program interface is the most likely source of invalid content, and because in the unlikely event that content is clobbered within well-designed software, a low-priority thread can find out about it soon enough for most purposes, without slowing down every single read over the content. Unless the system is kept persistently busy, that low-priority thread very well may discover invalid content before any critical thread reads it – which means the condition may possibly be flagged soon enough for someone to handle it without causing downtime. Waiting to flag invalid content only when it’s read by a thread that needs it would scotch any hope of catching the condition in advance.

Regarding the question of programmatic error handling, some past answers have included either truncating away any invalid portions or haphazardly synthesizing new ones. Those answers are, in other words, to throw away or corrupt potentially irreplaceable data. Why would anyone want to do that?

Instead, my suggestion is to consider how a routine meant to handle valid UTF-8 has found itself faced with something other than valid UTF-8. In other words, retest and debug. In the event that a general-purpose answer is the best available, like in a live application that’s got to do something with unexpected content without waiting for human intervention, then I suggest treating that content as 8-bit ASCII . After all, if a UTF-8-handling routine has been given some content to handle, and that content doesn’t check out as UTF-8, then ASCII is the most prevalent encoding that might’ve been in its place. Translating data cast as 8-bit ASCII text into valid UTF-8 content is lossless: the translation can be undone in case the need arises.

In other words, as of 2026, C++ is an ASCII-centric programming language that would benefit from a declarative natural language search mechanism with a simple interface. Such a mechanism built today would best be built around functionality for general UTF-8 support. Can this be found based on open-source solutions?

Until now, for the C/C++ developer looking for a no-frills approach to UTF-8 enablement, there’s been no simple answer. Methods for handling UTF-8 in C++ have been overloaded with support for extraneous encodings or with a broad range of extraneous functionality. Others have lacked thorough testing and performance comparisons against standard string functionality. Extraneous encodings may create more overhead than other extraneous functionality, particularly if each encoding provides for case-insensitive content matching. To achieve best performance, each case mapping arrangement for internationalized content is based on in-memory tables that can occupy megabytes.

The cost of code bloat can be viewed as a performance penalty, but there’s a bigger-picture view. Increasingly, elimination of overhead is associated with environmentally “greener” software. Low-bloat high-performance code is nowadays perceived as the responsible – not just responsive – solution. What’s needed is a provably reliable modern C++ class, or a family of functions compatible with legacy C, that has a focus on UTF-8 enablement and that provides a basis for declarative queries against semistructured internationalized content.

Can we reasonably speak of a declarative query without positing a query language in which to frame it? For C++, we can code a method that takes an expression as input: a pattern for matching against UTF-8 content, or a set of code points to find among the content, or other arrangements yet to be invented. Outside the UTF-8 context, Martin Kleppmann has described pattern matching within functional programming as a “declarative approach” in which... The logic of the query is expressed with code but it’s repeatedly called by the processing framework. The framework he had in mind when he wrote that statement was a NoSQL datastore, specifically MongoDB . But conceptually, the framework could be as simple as a family of functions for handling UTF-8 content. In any case, as he puts it, “...this declarative approach... makes it possible to introduce performance improvements without requiring any changes to [a] query.”

Imagine a semistructured writeup such as a parts inventory, a list of films organized by their titles and including the directors’ names, or a series of MLA-style references cited in a report. If that writeup is encoded in UTF-8, software can efficiently find most any delimited portion of it, based on just a few lines of code, using one or more of these methods: Targeted wildcard search; Tokenset search; and/or Basic full-text search, of the std::string::find() variety, but UTF-8-enabled.

Searching a relatively large chunk of content for a relatively small chunk that matches a search pattern – one that can contain wildcards – isn’t a new concept. Notepad++ offers a regular expression search that can operate across multiple files. But regular expressions lack standards and typically don’t perform nearly as well as dedicated algorithms for matching wildcards. Yet aside from regular expressions, until now, I haven’t seen a fast wildcard-driven search method unbundled and available as a tested C/C++ function.

There’s no reason wildcards can’t be incorporated into a UTF-8 search pattern passed into C/C++ code along with some UTF-8 content to be searched. In the search pattern, the wildcards can include the code points for ‘*’ (which matches any number of code points within the content) and for ‘?’ (which matches any single code point). Unlike Notepad++, which always finds the first match on the first and last code points within a given search pattern, a more typical use case for C/C++ might be to construct a slice of the content based on the match on a wildcard itself.

Which slice would that be? As it turns out, existing methods for matching wildcards track state that might be relevant in a range of scenarios. Matching wildcards is a backtracking endeavor to begin with, and the location of a prospective match on a wildcard is something that’s necessarily tracked. Arranging logic around the method to find a match within a larger chunk of content means tracking, additionally, the beginning and ending locations of the match on the overall search pattern, as with Notepad++. When a match against a given search pattern is found, any or all of these locations – the location of the first code point in the overall matching content, the location of the match(es) on any first or last wildcard in the search pattern, or the location of the overall match’s last code point – could be natural candidates for slice contruction.

Example based on a few MLA-style references Capote, Truman. Breakfast at Tiffany's: A Short Novel and Three Stories. Random House, 1958. Capote, Truman. In Cold Blood. Random House, 1965. Capote, Truman. Other Voices, Other Rooms. Random House, 1948. Hellman, Lillian. The Children's Hour. 1934. Famous American Plays of the 1930s, edited by Harold Clurman, Dell, 1959. Hellman, Lillian. Toys in the