Coding - Error control
9.4. QR codes

Quick Response codes, usually called QR codes, are a great way to transmit data from paper to a digital device. All you need to read them is a camera on a mobile device (which is pretty common these days). In fact, most QR codes can be read within a camera app, although if you’re experimenting with them, it can be worth installing a custom QR code scanner app that can give you more details about the contents of the code.

Try scanning the following QR code by pointing a mobile phone camera at it (on some phones you may need to adjust the settings so that a camera recognises a QR code). This pattern represents the very simple text “ABCDEFG”.

The black and white squares on the code just represent binary digits (bits), which might be interpreted as text, URLs, or even pictures. The bits include information about what kind of data it contains.

QR codes are used in all sorts of situations - advertising, making payments, providing information, contact tracing, and even headstones. Because they are often in public places, they might be read under poor lighting conditions and are vulnerable to damage, so they rely on error correction codes to make sure the data can be scanned reliably.

In the previous exercise, you should have found that a small number of changes to the bits doesn’t cause a change in the message. What is happening here is exactly the same principle that’s at work in the parity magic trick. When one bit is changed (either white to black, or black to white), it’s easy for the software to detect that this has happened, and it can change it back. However, QR codes use a stronger error control code than the parity trick; even if you change several bits, it can usually work out the original message. This is error correction at work.

Eventually, when too many bits are changed, QR codes can’t make the correction. What’s really impressive is that they still perform error detection: they won’t read the data incorrectly, instead they just refuse to give any data at all, rather than giving you incorrect data. You would have been able to do this in the parity trick if someone changed two or more cards. You can tell something is wrong, but you can’t be sure about how to fix it. The main thing is that at least you know that something is wrong.

The “damage” to the data (such as colouring in a white square on a QR code) is causing an error in the data; it is sometimes referred to as “corruption” of the data, or “noise”. On a QR code, data corruption could be caused by things like splashing the surface, getting ripped from wear and tear, people bumping it and leaving a mark, or light reflecting off the paper and creating a bright spot where it should be black. In situations other than QR codes, it is caused by physical damage to disks (such as scratching a DVD), and in data transmission it might come from electrical interference messing with wifi transmission, or unreliable equipment randomly changing some of the transmission.

The powerful method that QR codes use to deal with data corruption is called Reed-Solomon error correction, which we’ll discuss in the next section.

The Reed-Solomon method, used in QR codes, works in a similar way to the parity error correction used earlier in this chapter - it adds extra bits to the data so that errors can be corrected. However, the Reed-Solomon code is able to deal with a lot more errors in the data than the parity method can. Reed-Solomon error correction has been around since 1960, and remains one of the most widely used ways of dealing with errors introduced accidentally when data is stored. It is used for hard disks and optical disks (CDs, DVDs and Blu-ray), and, of course, QR codes. The Reed-Solomon method is a form of Forward Error Correction, which is important on file storage devices: if an error is detected in a file on a disk that was saved a year ago, you can’t travel back in time and ask someone to save another copy. (Actually, you kind of can travel back in time: if someone made a backup of the file a year ago, you could use that, but small errors occur in storage devices regularly, so it’s not ideal to go to backups every time there’s a small failure in a disk.)

The maximum number of errors that a Reed-Solomon code can correct is one half of the amount of extra error correcting data added to it. This means that if k bits of useful data are represented using n bits in total, the maximum number of bit errors that can be corrected, is $(n-k)/2$. For example, if a QR code has 55 bytes of data, and 14 bytes of error correction added, then it can correct up to 7 bytes (56 bits) of errors. In this case, $n=55+14$ and $k=55$, so the rate is 79.7%. Reed-Solomon codes can detect twice as many errors as they can correct, so it can detect up to $n-k$ errors (14 bytes in the example). It will probably detect errors if there are more than that many, but at that stage there are so many errors that it might accidentally be treated as an error correction situation for some different data.

A key feature of Reed-Solomon codes is that the code-rate $(k/n)$ can be adjusted depending on how many errors are expected. In the case of QR codes, a mark or splash on the code could easily change several of the bits, and by using a lower code-rate (more redundant bits), larger amounts of corruption can be tolerated. A simplistic way of thinking of Reed-Solomon codes is that they are like the parity error correction method, except instead of being based on single bits being added to a row of data, they are based on much larger groups of bits being added - the “parity” in this case is multiple bits. This allows for the code-rate to be adjusted, and also copes well with a lot of adjacent bits being corrupted (for example, when there’s a scratch on a DVD, a burst of interference in a radio transmission, or dirt splashed on a QR code).

The layout of a QR code is based on a standard that shows where different elements are stored in the code There are 40 possible sizes (called “versions”), and each has 4 different encodings for the data (numeric, alphanumeric, byte and Kanji), and also a choice of 4 error correction levels (low, medium, quartile, and high). Each black or white square (“dot” or “module”) corresponds to one bit of data. The example at the start of this section is the smallest size QR code (version 1), and is 21 dots wide and 21 dots tall. Some of the dots are used up for the alignment markers.

The error correction levels indicate how much of the QR code is used up for error correction. The highest level means that 30% of the data can be corrupted, yet still be corrected. The example above uses low error correction, which means that it can only correct the data if no more than 7% of the data is changed.

For the “quartile” level of error correction, which can correct up to 25% of the data being corrupted, about half of the data in the QR code is used for error correction. For example, a Version 3 QR code (29x29 dots) holds 70 bytes of information, and to get quartile level error correction, it needs to use 34 of the bytes for the data, and 36 bytes for Reed-Solomon codes, which means that the code-rate is 34/70, or 48.5%. That might seem wasteful, but it’s incredibly resilient to errors.

The “low” level of error detection of Version 3 QR codes had 55 bytes for data and 15 bytes for Reed-Solomon codes, which is a rate of 78.6%, yet it can still correct the information if up to 7% of the data (about 5 bytes, or 40 dots) is corrupted. (Theoretically Reed-Solomon codes should be able to deal with about 10% of the data being corrupted in this situation, but there’s a bit more going on in the details that we haven’t covered here!)

In contrast, the 5x5 block of data with parity error correction (used in the magic trick earlier in this chapter) could correct only 1 bit error out of the 25 data bits, which is 4% of the data being corrupted. However, for this case, the code-rate is 25/36, or 69%, since the 25 bits had 11 extra bits added to them. So Reed-Solomon codes can get a good level of error correction without having to have the code-rate too low.

Experiment Project

For the Level 1 QR code above, change one dot at a time, scanning the QR code after each change, and find out at what point it is unable to correct the error. What percentage of corruption is this code able to deal with?

Researching these questions will give you an idea of how error detection and correction have evolved historically, and how they will have an important role in the future. There’s some information below that can help with getting into these ideas.

• How did early tape storage systems depend on error detection?
• When did the modern credit card first appear, and why did it use Luhn’s error detection algorithm?
• What technology made barcode scanning possible?
• When did barcodes start being widely used, and how did they use error detection to make them reliable?
• When did QR codes appear, and what technology was needed before the public could use them conveniently?
• What is RFID, how is it being used instead of barcodes, and what is the role of error correction in this situation?
• How is error correction essential for quantum computing?

The need for error control has been around for a long time. Early computing storage, like reels of tape, stored important information like bank balances, but they couldn’t be read reliably, so it was obvious to add a parity check to them. Even then, a single parity bit added to each 8 bits had a 50/50 chance of detecting multiple errors, but usually the tape would have errors across multiple bytes (perhaps it wasn’t lined up quite right), so it would become obvious that the data was unreliable, and had to be read again - this was an ARQ situation, based on error detection! These days, magnetic media like tapes and disks use more effective error correction codes, typically a Reed-Solomon code, so that they can reliably reconstruct data that has been read incorrectly - that is, they use Forward Error Correction (FEC).

Credit cards appeared in the 1960s, and the last digit of a credit card is a check digit calculated using Luhn’s algorithm, which involves multiplying every second digit by 2. The same algorithm is still used over half a century later for credit cards; there are more reliable checksums that could be calculated, but this form of error detection is embedded into machinery and software all around the world, so it looks like we’re stuck with it!

The idea of scanning product codes using something like a barcode has been around for a long time, but it was only in the 1970s, with the availability of lasers, that it was feasible to scan product codes in a commercial situation (see here for some interesting history of the barcode). Prior to that, all products needed to have a price sticker on them, which made it hard to change prices or have specials; or if they didn’t have prices, checkout operators would need to be able to figure out the price. With the wide range of products appearing in supermarkets at the time, this was becoming challenging, so the barcode was welcomed once it was feasible to produce scanning devices. A Universal Product Code (UPC) standard was developed that included a check digit to make scanning reliable, and this time an error detection checksum was used based on multiplying every second digit by 3. This system can always detect if a single digit is read incorrectly, but if two digits are incorrect, there’s a one in ten chance that they will cancel each other out. If every digit in the product code has been read in randomly (perhaps the barcode is completely messed up), there’s a one in ten chance that the check digit will be correct, and in this case, the problem would only be detected if the product isn’t in the shop’s database. These sorts of undetected errors are very unlikely, and we’ve come to trust barcode scanners - these days, many people don’t even check that they’ve been charged for the right products, thanks to error detection codes.

Two-dimensional (or “matrix”) barcodes allow for the storage of much more data. There are many types of 2D barcodes in use, but the one that the public sees the most often is the “quick response” (QR) code invented in 1994. Because QR codes can store so much more data, it is a lot easier to add error correction bits to them, so they are capable of Forward Error Correction. This in turn has made them useful in places where they might be subject to wear and tear - printed on paper, stuck on notice boards, windows, and places that are exposed to the elements. There are even buildings with QR codes on their roof - these can be scanned from planes or satellite images! The widespread availability of smartphones with cameras from around 2003 meant that the public had all the equipment with them to scan a QR code, and these days the software for decoding a QR code is even built into the camera on a smartphone.

Technology has been changing rapidly, but the role of error detection and correction has been crucial in keeping these systems cheap and reliable. Even in the future, error correction looks to be essential for RFID tags (which might replace bar codes), for high speed data transmission, and even for quantum computing, for dealing with unstable qubits.

• QR codes are explained in some detail on Wikipedia, including details of where the error correction information is stored in the code (in the diagrams, data is labeled D1, D2..., and error correction information is labeled E1, E2..). The exact format is quite complicated, with information zig-zagging around the layout, and it isn’t necessary to understand that level of detail to be able to explain the error correction process.
• There is a fairly gentle explanation of Reed-Solomon coding by Russ Cox available (with examples using the “Go” language), although this is only for those who really want to know more details.
• This Wikiversity page gives an explanation of “Reed–Solomon codes for coders”. It uses Python code to explain the process, but you will also need to understand the mathematical concepts of polynomials and finite fields that Reed-Solomon codes rely on.
• Computerphile covers Reed-Solomon codes in this video (which builds on this video).
• Here’s a blog with “wounded” QR codes.
• This is the website of the company that invented QR codes, which has more details about them.
• There’s more general information about Error Correction on Wikipedia.