How to test corrupted file handling

Real uploads are sometimes broken: a connection drops half-way, a disk fills up, a phone app crashes while saving. Most validation looks at the file name and the first few bytes, so a damaged file is accepted and only fails later, when a thumbnailer, converter or indexer tries to read it. These tests find out where that happens and whether the user hears about it.

Why damaged files get through

A file signature (the “magic bytes”) is at the very start of the file. A file cut off at 60% still starts with %PDF-, and a truncated JPEG still starts with FF D8 FF and still declares its full width and height. Checking the extension, the Content-Type header or the signature therefore passes every file below. Only reading the whole structure reveals the damage.

The test files

Each file page has a hex dump of the first bytes and a checklist for that file.

What a good failure looks like

• The user is told. The message says the file is damaged and suggests exporting or saving it again. It does not say the format is unsupported, which sends people looking for the wrong fix.
• Nothing half-made is published. No half-grey thumbnail, no document listed with 0 pages, no archive whose contents are partly extracted.
• The error is handled. A clear 4xx response or validation message, not a 500, a timeout or an empty preview.
• Partial recovery is a decision. If a library recovers only some of the PDF's 10 pages, either reject the file or keep it and tell the user what is missing. Silently storing a shorter document is the worst outcome.

Detecting damage before you store a file

To catch damage at upload time, open the file with a full parser instead of looking at its first bytes. On the command line:

# PDF: qpdf exits with a non-zero status when it finds errors
qpdf --check sample-pdf-corrupted.pdf; echo "exit status $?"

# ZIP: test every entry against its CRC-32
unzip -t sample-zip-corrupted.zip
7z t sample-zip-corrupted.zip

Both commands exit with status 2 on these files. Note that qpdf --check also fails on a valid password-protected PDF it cannot decrypt, so test for encryption first; see encrypted is not corrupt.

In Python, Pillow opens images lazily, so Image.open() alone succeeds on a truncated JPEG and even reports its size. Decoding the pixels with load() is what fails:

import zipfile
from PIL import Image

with Image.open("sample-jpg-truncated.jpg") as im:
    print(im.size)      # (800, 600): the header is intact
    im.load()           # raises OSError: image file is truncated

# A DOCX is a ZIP package; without its central directory it cannot be opened
zipfile.ZipFile("sample-docx-corrupted.docx")   # raises zipfile.BadZipFile

# A ZIP with a bad entry: testzip() returns the name of the first bad entry
with zipfile.ZipFile("sample-zip-corrupted.zip") as z:
    print(z.testzip())  # data/payload.bin

Pillow's ImageFile.LOAD_TRUNCATED_IMAGES = True setting makes it accept truncated images. If your code sets it, a check like this passes damaged files; make sure it is off where you validate uploads.

Background jobs and retries

Damage often surfaces in a queue job rather than in the upload request. Upload each file and watch the job that processes it:

• The job fails once and is marked as a permanent failure. A damaged file will fail the same way every time, so retrying it forever only fills the logs and the queue.
• The failure reaches the user, for example as a “could not process this file” status, instead of a spinner that never stops.
• Temporary files and partial output are removed.

Encrypted is not corrupt

A password-protected file is valid, but your system cannot read it without the password, and it should not be reported as damaged. sample-pdf-password-protected.pdf opens with the password sample. Without the password it should be reported as encrypted, with a prompt or a clear message; a wrong password should be refused; and each step that reads content, such as text extraction, thumbnails and virus scanning, should have a defined result for a file it cannot open.

Tell the two cases apart before you call a file damaged. With qpdf, --requires-password exits with status 0 when a password is needed:

qpdf --requires-password sample-pdf-password-protected.pdf; echo $?   # 0: needs a password
qpdf --requires-password sample-pdf-corrupted.pdf; echo $?            # not 0: no password needed