UTFZ

UTFZ is a small utf16 compression library.

How does it work?

Consider the following utf16 representation of "Hello":

48 00 65 00 6c 00 6c 00 6f 00

You can see that all the odd bytes are 00.

Consider the following utf16 representation of the Farsi word "سلام":

33 06 44 06 27 06 45 06

You can see that all the odd bytes are 06 and this pattern repeats itself. These are wasted bytes. The letters of each language, are incrementally one after another and share the same high byte most of the time. Utfz assumes the high byte is equal to 00 and changes it every time it encounters a 00.

With utfz, the first word becomes:

48 65 6c 6c 6f

and the second one becomes:

00 06 33 44 27 45

Where the first 00 tells the decoder to set the high byte to 06. Words of different languages with different high bytes can be combined:

48 65 6c 6c 6f 00 06 33 44 27 45

To escape a low byte that equals 00, utfz adds the current high byte after the low byte in the sequence.

Check the source code for more info.