VNC specific DES implementation.

Motivation

Find more info on Vidar Holen dedicated webpage

What VNC actually does

The RFB specification says that VNC authentication is done by receiving a 16 byte challenge, encrypting it with DES using the user specified password, and sending back the resulting 16 bytes.

The actual software encrypts the challenge with all the bit fields in each byte of the password mirrored.

Actual password ("COW"):
01000011 01001111 01010111 (plus five bytes of zero)
Key used as encryption:
11000010 11110010 11101010 (plus five bytes of zero)

The actual flipping is, in effect, done in rfb/d3des.c in realvnc.com's unix server. The bytebit array has had its entries mirrored, as the comments mention.

This is no doubt because the most significant bit in 7-bit ascii is always 0. When the DES algorithm creates a 56bit key based on the password, it makes sense to trick it into removing this 0 instead of the the least significant bit which is part of the actual password (otherwise B and C would for authentication be the same character).

Credits

See the dedicated LICENSE file for full credits.

Related

noVNC, a web-based VNC that use this DES specific implementation