The way I’m organizing the main backups to start with is with folder names such as 20250505 Laptop Backup, 20250508 Media Backup, etc.

Eventually I plan on organizing things in bulk folders with simple straightforward names such as Movies, Music, Game ROMs, Virtual Machines, etc.

Yes, thankfully I already got all my main files, music and movies backed up. Right now I’m backing up my software, games, emulator ROMs, etc.

Hopefully that drive finishes backing up before the weather gets bad, cuz I’m definitely shutting things down when there’s lightning around…

Thank you, at least someone understands my stubbornness. 👍

That’s just it, there is no setup, except Linux Mint as the main system. It’s literally a physical bucket of discs and drives in all sorts of various formats…

Please do explain then.

I have multiple drives with various differing directory trees.

‘An’ drive? I mean like 10+ drives, looking to do a master backup.

Don’t put it past me, I already have before.

Also, try converting Big Endian vs Little Endian ROM file formats. I spent many months doing that, via goodtools.

I’m not in any hurry to accidentally overwrite a ROM that’s been corrected for consistency in my archives because some automatic sync software might think they’re supposed to be the same file.

Block level dedupe doesn’t account for random data at the end of the last block. I want a byte for byte hash level and folder comparison, with the file slack space nulled out. I also want to consolidate all related files into logically organized folders, not just a bunch of random folders titled ‘20250505 Backup Turd’

I also have numerous drives with similar folder structures, some just minimalized to fit smaller drives. I also have archives from friends, based on the original structure from like 10 years ago, but their file system structures have varied from mine over the years.

Not my code, I didn’t even have internet access when I started programming.

I don’t even want this data encrypted. Quite the opposite actually.

This is mostly the category of files getting deleted from the Internet Archive every day. I want to preserve what I got before it gets erased…

I get the concept of block level reduplication, no problem.

But some of these drives came from friends that reorganized their copy of files their own way, while I took my main branch they copied from and salvaged damaged files.

Ever heard of goodtools? I’ve spent an awful lot of time salvaging corrupt video game console ROMs. I have all of Atari 2600, most of NES and SNES, a number of N64 and a number of PSP games, along with a lot of other stuff.

I ain’t about to play headgames on what I have and haven’t salvaged already, I must keep track of what device stores what, what filename is what, and what dates are what.

I want an organized file/folder structure. I didn’t spend the past 20+ years to trust everything to automation.

These 4+4+2TB drives are fresh new to me, amazing they all seem to check out.

Right now, the drives I’ll be pulling data from range anywhere from 40GB to 320GB, from a variety of different file systems. And that’s not counting the many optical discs that need to be archived before disc rot sets in (I’m sure some have already, but looking better than I expected).

I don’t necessarily need a 20TB, just one of these 4TB drives ought to do the trick. Besides, its already gonna take me months to pull all my backups from the Internet Archive…

Not everything is an individual file though, a lot of the stuff needs to be stored and maintained as bulk folders.

I mod operating systems and occasionally games, plus write software. I can’t just dump off all text files into a single folder, that’ll just dump off all readme.txt files off into a single TXT folder, losing association with the project folders from which they came.

It’s not about whether I need any of the data or not. It’s about the fact that I have many archives scattered across many smaller driives of things getting deleted from the internet every day.

It’s about data preservation. And suddenly I have 2X 4TB hard drives and a 2TB hard drive? A total of 10TB, just suddenly found in a dumpster, and all the SMART stats check out?! 👍

I’m looking to backup everything I have from the past 25+ years!

Just a drop in the bucket, one of my drives has like almost all the SNES game ROMs…

I guess you’re missing the point then. I’m backing up data coming from many different file systems, FAT12, FAT16, FAT32, EXFAT, NTFS, HPFS, EXT2, 3 and 4, ISOs (of varying degrees of copy protection plus MODE1 and MODE2 discs with audio tracks)…

Plus different date revisions of many files.

You think there’s anything consistent enough where any one solution works?

I need all the recommended software I can throw at it. Sure I’d love a purely automated solution, but i know there’s still gonna be a lot of manual curating on my part as well.

Also, files don’t have to match and filenames aren’t important? Are you a psychopath? That’s exactly what I want, to organize folder and filenames, and match and remove duplicates based on file hashes.

I have like 10+ hard drives and probably 75+ optical discs to back up, and across the different devices and media, the folder and file structure isn’t exactly consistent.

I already know in advance that I’m gonna have to curate this backup myself, it’s not quite as easy to just purely let backup/sync software do it all for me.

But I do need software to help.

Joy oh joy, I got like 75+ optical discs and like 10+ hard drives (whatever still works) to back up.

This is already gonna take months I know, just my free time at the end of the day.

This is gonna be fun. /s

Thank you and everyone for the advice though.

Side note, I think one of my drives has almost all the SNES game ROMS…

Of course.

Goal #1 is to migrate what data I can (which is a fucking lot) all over to the 4TB, in separate folders for each drive. Only after that will I worry with scanning for dupes and organizing things.

I’m just looking for advice on what software is recommend for helping deal with such large tasks in advance.

I’ve actually got 2X 4TB drives plus a single 2TB drive. But yeah, I know the best and easiest way is to consolidate it all on one drive first.

I found Timeshift to be a disappointment. I tested it as I was setting my system up.

• Install Linux Mint, obviously.
• Install most main software I want.
• Do a Timeshift backup.
• Install more software I might want to try eventually.
• Restore the Timeshift backup.

Result: The system still thought all the extra software packages were installed, but none of them actually worked. Like, if Timeshift is gonna uninstall packages that weren’t present in the last backup, shouldn’t it also unregister those packages as well?

To fix all that crap, I had to force reinstall all packages, which takes about as long as a full OS reinstall, but I was already happy with the rest of the configuration, so I ran…

sudo aptitude reinstall '~i'

Also, 1474560 / 1024 = 1440

If anyone could keep up with binary numbers back in the day, floppy disks were literally measured in binary megabytes.