Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Alex Atkin UK]

I just had a disaster moving around my NAS storage.

I had ~/Documents symlinked to /mnt/storage0/alexatkin.  I had done a fresh install on a larger drive so moved /mnt/storage0/alexatkin/* to /home/alexatkin/.
I thought it had been successful so ran a backup (huge mistake here).

Somewhere along the line something went wrong and the Pictures folder is empty.

What I think happened is because I stupidly used mv instead or rsync -a, it had been sat waiting for me to confirm overwriting the default Pictures folder.  I'm sure I remember it asking and saying y to overwriting the folder, but something must have gotten missed in the console error messages of NUT unable to see my UPS (that's another issue, no idea why that wont work).

Of course I did this overnight so whatever happened I did not notice something had gone wrong, and only noticed AFTER running the backup script, wiping out the backup copy.

I have written a ton of files to /mnt/storage0 in the meantime, so recovery from there is impossible.  Unfortunately the backup was also after that, so I'm not hopeful of recovering any of this.  Of course, this is all my personal photo collection, the most devastating folder to lose.

I knew using the OS default Pictures folder would bite me in the ass eventually, even if it was my own stupidity for not sanity checking the move was fully successful.

I'd say live and learn but fibro fog seems to mean I'm doomed to constantly losing my personal photos.  I already lost tons from CD bit rot and this was supposed to be as bullet-proof a solution as I could manage.  Of course, there is no accounting for user error and ignoring the age-old advice of "always make a backup FIRST and make sure everything is where it should be after".

I'm just so angry at myself for not double checking everything had moved over and why the heck I used mv in the first place.  What baffles me is I could have sworn the Pictures folder was gone from the old drive, implying it had successfully moved it, but its nowhere to be found.  I had ample opportunity to spot this mistake and recover from the backup. I could have sworn I checked folder sizes to make sure there was the right number of files in both locations, I don't know how I missed it.

[UPDATE]
Oh, it looks they ARE still on /mnt/storage0/alexatkin, that's a stroke of luck if true.

[burakkucat]

[UPDATE]
Oh, it looks they ARE still on /mnt/storage0/alexatkin, that's a stroke of luck if true.

Hopefully you will be able to perform a recovery. 

[Chrysalis]

Good to hear luck was on your side.

I think we all been there, I know I have.

[Alex Atkin UK]

I was more in shock than anything, especially given I had not that long ago been digging through my CD/DVDs trying to find anything that wasn't on the NAS given how many I've had that failed to read any more.

This was supposed to be the solution to reduce the chances of that happening again so if I had somehow deleted it during the transfer, that would be a massive self-own.  I'm thinking I definitely need to alternate backups for my home directory so that if I DO mess things up, I will at least be able to recover the older data months down the line.  I can't do the whole server because the cost of a second set of backup drives would be insane.

[Weaver]

I once nearly lost months of work at work by copying files from a floppy instead of to a floppy. Luckily I had backups. I was so scared by this near miss that I wrote a tiny batch script file to do copying to a floppy that would only copy in one direction and had all sorts of checks and guards in it. For example, it marked all the files in the src dir as read only first just in case of madness and then wiped the floppy clean in readiness, and checked all the destination files after copying by reading them back.

[Alex Atkin UK]

I tend to mess up with rsync forgetting when to use and not to use a / at the end.  I ended up with multiple copies of my music collection last night, which at least was better than deleting it.

This is why I mostly try to do 1:1 backups and use JBOD rather than RAID or other forms of spanning.  Really can't afford space for parity or the hassle of reconstructing everything.  If one drive dies, I restore the backup on a new drive, that's it.  Though up until now I just buy larger drives before the old ones fail.  Makes me wonder when were going to top-out on HDD sizes actually, it seemed to struggle for them to reach 20TB and restoring data takes forever as read/write speeds have pretty much topped out already.

[Chrysalis]

For me its not realistic to backup "everything".

So essentially anything I can redownload to recover it isnt backed up.

OS is backed up to local drive.
Emails/browser configs backed up to local drive.
Some modding work not on NAS backed up to local drive.
Keeppass auto backs up its DB to second save location on every save.
Documents/pictures/other important files, configs etc. backed up to cloud.
Lots of my modding files, game recordings, are stored on NAS, yet to have an actual backup so just the drive redundancy currently, cloud was the plan, but was waiting for FTTP.  Still better than old situation where they was on single drive no redundancy.
I also use snapshots on both ZFS(NAS) and NTFS, they great for dealing with operator error stuff, and would have saved you Alex.    So e.g. you accidentally overwrite a file, no problem restore from snapshot. (NTFS snapshots will work with shift-delete direct deletes).  To enable this, enable system restore for the drive in question, I suggest then task scheduler task, snapshot every day or something.  It will auto delete oldest snapshot if goes over space limit assigned).  Shadow explorer great app for navigating windows NTFS snapshots.
Laptop OS also backed up.
My browser profile data is on its own partition same with pictures/documents/emails etc. so if I ever need to reinstall OS backup due to failed windows update or something that stuff doesnt get replaced.

[Alex Atkin UK]

That's the thing, when you are storing a lot of movies and TV shows, many not for sale on physical media (and often not even digitally), almost none of it can be redownloaded, at least not in the same quality, and the process of downloading it in the first place is a legal grey area as they do not provide official DRM free ways to do so.  So I may be able to download stuff I bought off Amazon today, but tomorrow they can update the DRM and block it.

Were in a really stupid situation where shows/movies in 4K HDR on streaming services are only at best available in 1080p SDR on physical media, sometimes only on DVD and fairly often not at all.  And thanks to licensing, anything you can't actually buy outright can and does disappear once its no longer popular enough, plus there's no guarantee stuff you DID buy remains available indefinitely either.

[meritez]

Yes the Fantasia edit that's on Disney's streaming service is concerning.


https://youtu.be/DeE-yaev7bM

[Alex Atkin UK]

Disney also have a lot of the Marvel movies in IMAX on there that aren't available elsewhere.

[Weaver]

I was taken to see Fantasia when I was very small. I was a bit too young to fully appreciate it. The bowdlerisation that Meritez showed us is shocking, disgraceful.

Is it possible to use NTFS in Linux and get access to the NTFS versioning / snapshot feature too ?

[meritez]

Today I learnt a new word.
I remember the original vividly, and as they're fantasy creatures I see nothing wrong with the depiction.

[Alex Atkin UK]

I was taken to see Fantasia when I was very small. I was a bit too young to fully appreciate it. The bowdlerisation that Meritez showed us is shocking, disgraceful.

Is it possible to use NTFS in Linux and get access to the NTFS versioning / snapshot feature too ?

Its my understanding there are several file systems vastly superior to NTFS in Linux.  The NAS was reinstalled with btrfs so snapshots are available, but I have no idea how to do them.

[Weaver]

The modern-ish NTFS versions have per-file versioning, so you can go back to an earlier version of a file, or iirc can go to the directory and look at that to see deleted files. Have I got this right ? It has been about twelve years since I used WinNT family o/s’s, so my memory is now pretty foggy. Do the various Linux file systems offer that same kind of capability or do things work somewhat differently ?

[Chrysalis]

Yes thats volume shadow copy (labelled as system restore in windows).

For Linux the best filesystem for that is ZFS, not officially supported in Linux though due to licensing issues.  However Proxmox and TrueNAS scale are based on Debian and do officially support ZFS.  FreeBSD supports ZFS, which also includes pfsense, opnsense and TrueNAS core.