Guides5 min read
How I archive data I want to open in 50 years
I archive the backups I want to open in fifty years with tar, zstd in independent chunks, PAR2 recovery files and SHA-256 checksums.
A few years ago I bought a four-bay Synology NAS. Twenty-six terabytes. It felt like infinite space.
Last weekend it was full.
Most of that space was backups. So I sat down with Claude Code to work out how to archive them properly. Very quickly, saving space became the smaller problem.
These are files I want to keep for decades. Will I still be able to open them in twenty years? In fifty? And what happens when part of an archive gets corrupted? Over that kind of timescale, something will.
So I designed my own archive layout. It's not a new file format. Every piece is a tool that already exists and will still exist.
RAID is not a backup
First, the storage. RAID keeps a volume running when a disk dies. It doesn't notice when a file silently rots, it can't undo an accidental overwrite, and it does nothing once the files leave the NAS. RAID is not a backup. I wanted each archive to protect itself, wherever it ends up: on the NAS, on a USB drive or in the cloud.
Open tools, open formats
My first rule was longevity. I have to be able to extract these archives in fifty years. So I only used open-source tools with open formats, the kind the Linux kernel and every major Linux distribution depend on.
Tar was the obvious starting point. Almost everyone uses it, and it works with any compressor. But what does tar actually do? I created two files and ran tar with no compression to find out:
tar --no-mac-metadata --no-xattrs -c foo.txt bar.txtfoo.txt000644 000765 000024 00000000014 15265122427 013052 0ustar00istvanstaff000000 000000 foo content
bar.txt000644 000765 000024 00000000014 15265122446 013034 0ustar00istvanstaff000000 000000 bar contentThat's all it is. For each file, tar writes a header with its path, permissions, owner and timestamp, then the content, one after another in a single stream. Nothing clever. Pipe that into a compressor and you have a compressed archive. Each tool does one job. Moments like this remind me how good the Unix way is.
That simplicity pays off later. Every file in a tar stream starts with a header on a 512-byte boundary, so if a chunk in the middle is lost, you can find the next header and carry on.
Compression in independent chunks
Next problem: if a one-terabyte archive gets damaged, I don't want to lose a terabyte. Most compressors produce one continuous stream. Break it in one place and everything after that point is gone. I needed compression in independent chunks, so the healthy parts stay readable.
It came down to xz or zstd. I picked zstd. The way I use it, recovery is easier: every piece carries a small label saying where it belongs, and plain zstd simply skips the labels. It's fast, it's what Arch, Fedora and Ubuntu now use for their packages, and there's a well-argued essay on why xz is a poor fit for long-term archiving.
A repair kit that travels with the data
Saving the healthy parts is good. Repairing the damage is better. That's where PAR2 comes in.
PAR2 creates recovery files that sit alongside the archive. Think of them as a repair kit that travels with your data, whether it lives on a NAS, a USB drive or in the cloud.
You choose how much extra space to spend on recovery. With ten percent redundancy, you can rebuild up to ten percent of the archive's data blocks, as long as the recovery data survives. One damaged bit makes a whole block count as damaged, so that isn't quite the same as losing ten percent of the file.
It doesn't replace a second copy somewhere else. If the disk holding the archive and its repair kit dies, neither will help you. But for a damaged archive, those extra files can be the difference between losing data and getting it back.
Knowing before you need it
The last piece is knowing something is wrong before I need the data. PAR2 can verify the archive, but I wanted a check that doesn't depend on PAR2 and that works at the level of the files I actually care about. So every archive gets a SHA-256 checksum, and beside it a manifest with the SHA-256 of every file. I can check the archive without unpacking it, and check every file after extracting it.
One tool to tie them together
None of these layers depend on each other. If PAR2 is gone in thirty years, tar and zstd still open the archive. The repair kit is a bonus, not a lock-in.
No single tool does all of this, so I wrote one that ties them together. I called it archivist. I wrote it in Python, not because it's my favourite language, but because Bash gets hard to maintain and test, and Python is already installed on the Debian machine where this runs, and on almost any other Linux box.
I had it working in an afternoon. Since then I've archived terabytes of data, reclaimed about a terabyte of space, and added support for importing Time Machine backups.
What the fundamentals were for
Claude Code helped me research the options and put the pieces together. My engineering experience decided what to ask, judged the answers, and checked whether the result actually met my requirements.
That's what made this interesting to me. Knowing the fundamentals didn't mean I already knew every tool or every answer. It meant I could make much better use of the help.
Try opening your oldest backup, and see whether it still works.
Related reading