Backing up my data.. program suggestions?

[AloneWolf]

Hey guys. I posted in here a little while ago about wanting to move over to Linux as my main platform. Well, after some unexpected hardware issues and delays in getting stuff replaced, I finally setup my dual boot system with KXStudio and Windows 10. I plan on using Linux native software as much as possible and just keeping my WIndows partition for some games and the odd program here and there.

After doing a lot of shuffling around with which files are stored on what drives, I've got a few things I could really do with some help on if you guys could?

1. I have an external storage/backup drive which is only accessed by Linux. However, I do plan to backup some of my WIndows files onto there via KXStudio (I can just access the WIndows partition from KXStudio I think). Are there any downsides to using NTFS file format for this drive as far as Linux files are concerned (other than more fragmenting)? And what about the other way round - will using EXT4 cause any issues for the WIndows files (other than not being able to access them from Windows, which is not actually a problem for me)?

2. I plan on backing up my KXStudio Home folder to this external drive. Are there any other directories it'd also be useful to backup? For example, on WIndows I like to backup the "AppData" folder as that contains a lot of program settings and makes it easier when reinstalling..

3. What would be a good program for doing these backup tasks? The way I like to do backups is one full, then a bunch of incrementals.. I had a look in Synaptic for backup programs, but so much stuff came up and I had no idea what to get (a lot of it seemed irrelevant too). I find the package manager in Linux Mint to be a bit more user friendly/easy to spot good programs. I feel like I just have a sea of random results in KXStudio :/ Sorry, I don't mean to talk badly of KXStudio, I just feel a bit lost with the package manager at times..

Hope you guys can help.. I'm looking forward to being at a stage where I can start helping others out by answering questions around here..

Thanks

[sysrqer]

I don't know apt package manager but for my backup I create a list of my world file (manually installed stuff) before running the backup. The original (system) file is backed up as well but I like having a copy in /home. I use rsync and backup my whole drive with a couple of exceptions, and run it as a script.

Code: Select all

#!/bin/bash
cp --remove-destination /var/lib/portage/world ~/Documents/Configs/world_list
sudo rsync -aA --exclude-from=excludelist --delete --stats --progress / /media/sdc1-usb-Mass_Storage_Dev

My excludelist is in ~/excludelist and is this:

Code: Select all

/dev
/sys
/proc
/mnt
/media
/tmp
/run
/var/tmp/*
/lost+found
/home/me/.thumbnails

I'm sure apt must have something similar or a command which can create a list of explicitly installed stuff. If you needed to reinstall then you could probably just feed it a file directly, I seem to remember pacman being able to read from a file list.

[AloneWolf]

Thanks for the suggestion, sysrqer.

I'm not feeling in the mood for getting crazy with the terminal just yet I'm trying to keep things simple and avoid it while I get my computer setup for use, then I'll dive into that stuff at a later date once I'm more comfortable.

I did find rsync in the package manager, and it looks like there's some gui's for it. I don't know which ones are good, but maybe I'll try a few out and see what ones let me set things up easily..

Cheers dude

[AloneWolf]

Sorry to double post, but I couldn't figure out how to edit my posts :/ Is that possible?

Anyway, I'm trying out grsync (gui for rsync). It seems to work good for mirroring files. I kinda like this method, other than the fact that it'd also good to have backups of older files incase I end up mirroring corrupt files or something like that.. There was an option to keep copies of older file versions, but when I did that as a test, I wasn't able to open/access the older file versions it backed up. Not quite sure what I'm supposed to do with that..

Also, I forgot to mention this in my original post, but is there something like WIndows's "Volume shadow copy service" I can use? That way I don't have to worry about files being in use when I'm trying to back them up. It seems to be the only way to backup operating system files which are always in use, at least on WIndows anyway..

Thanks again.

[glowrak guy]

I posted the other day about editing, but it was deleted. Login here, go to your posts,
there should be three icons:

Edit

!

Quote

If not, it's a rare browser flaw. I edit a lot, and not because I'm a genious

[AloneWolf]

I don't quite understand your instructions. I tried clicking on "View your posts" at the top of the page, but I didn't see anywhere to edit posts from there. I'm not sure if that's where you mean? Cheers.

[asbak]

rsync really is the way to go
Luckybackup is a rsync front/end (GUI), works a treat.

[AloneWolf]

I did download Luckysync earlier actually. I also downloaded backupninja. I've yet to try either of them though. I will do soon though Cheers.

[Luc]

1. External storage/backup drive:
NTFS is still considered experimental in Linux. It's generally (not 100%) okay for reading, but always suspicious for writing. It's easy to find people who will say they never had a problem with it, but the software that is supposed to support it labels itself as experimental, which is pretty much a red light. It has to. NTFS is a secret file system created by Microsoft. Only Microsoft knows exactly how it works. Those who have reverse engineered it say it's very complex, convoluted, probably complicated on purpose to prevent third-party support and hamper competition. I would never trust that data trap to keep my files.

FAT32 might actually be a better idea. I used FAT32 with Windows 95 and 98 for almost 10 years and never ever lost a file, even when I had sudden power failures (and I had a few) or system crashes (too many to count). Power failures or system crashes have corrupted my files on ext3, but never on FAT32. For 5 years, I had a FAT32 partition that could be accessed by Windows and Linux on a dual boot setup. I never had a problem with it. And it is very compatible, very easy to read with both Windows and Linux. There are drawbacks: it's slow and wasteful. It's a little slower than ext3, the slowest of all Linux native file systems, and if you fill it up with a lot of small files, it can hold about 1/3 as much as the Linux native file systems.

But there is no reason to use either if you do not intend to access it from Windows. You can store anything into any file system, that has no effect on the integrity of the files. Just make sure you really don't need to access them from Windows, because you can't. I know there are Windows programs that can read ext2, but I don't know about ext3 (ext4 is almost the same), and it may not be reliable.

2. Are there any other directories it'd also be useful to backup?
You sound like a genuine Linux newbie, so you may want to backup the /home/ and /etc directories as well. The /home/ directory contains user data and /etc contains system configuration.
I like to backup everything, i.e. clone my entire hard disk onto another so I can just slip it in place of the old one and boot up if something terrible happens. Our fellow forum user sysrqer gave you a very good recipe for exactly that.
On Debian and derivatives (such as Mint or Ubuntu), you can backup the list of installed packages this way:

$ sudo dpkg --get-selections > installedpackages.txt

That does not really backup the content, i.e. the packages. It's just a list. When you need to restore it, you run this:
$ sudo apt-get update
$ sudo apt-get dist-upgrade
$ sudo dpkg --set-selections < installedpackages.txt
$ sudo apt-get dselect-upgrade

You will need an Internet connection for that, for everything will be downloaded *then* installed. Depending on the number of packages, it may take quite a while.

3. What would be a good program for doing these backup tasks?
rsync all the way. Don't be afraid of the command line. Here is a crash course:

1) Mount the external media.
2) Run a line like this:
$ sudo rsync -aSHxv /source /target
or
$ sudo rsync -aSHxv --delete-before /source /target

Read 'man sync' to learn what the options -aSHxv and --delete-before do. Read it, it's not difficult.

Main pitfalls:
/source /target => clones a directory or file system
/source /target/ => (trailing slash on target) copies the entire /source directory into /target/ as a subdirectory. Careful with that!
/source/* /target/ => copies everything inside /source into /target/ BUT --delete-before will not work.
/source/ /target/ => copies everything inside /source into /target/ AND --delete-before will work.

[emarsk]

/source/* /target/ => copies everything inside /source into /target/ BUT --delete-before will not work.

Also, it won't copy hidden files/directories (those starting with "."), not good for $HOME…

Anyway, another vote for rsync here.
Here's what I use for my backups:

Code: Select all

rsync -uFaxhiP --stats --del /source/ /target/ -n

The "-n" switch is for simulation mode. If everything looks good, I run it for real without "-n".
The -F option is a bit obscure, read the manual if you're curious but you can ignore it for the moment.
The "-hiP --stats" options just control the output shown.
The "-uax --del" switches are the important ones, and I add "-H" if I'm backing up some place where I'm not sure there are no hardlinks (or I know there are).

Code: Select all

 -n = --dry-run
 -a = -rlptgoD
    = --recursive -links --perms --times --group --owner --devices --specials
 -x = --one-file-system
 -F = --filter='dir-merge /.rsync-filter' ; -FF : --filter='exclude .rsync-filter'
 -H = --hard-links
 -h = --human-readable ; -hh : 1024 prefixes
 -i = --itemize-changes
 -P = --partial --progress (implies --verbose)
 -u = --update

I keep only full backups. If you're really interested in having differential backups at different points in time, you can look into rsnapshot (which uses rsync, by the way).

AloneWolf, don't be afraid of the command line! It's where the real power of Linux lies. It's where you get the real freedom to do exactly what you want.
There are GUIs for rsync, but all they do is putting checkboxes for some switches, and are inherently limited.

[AloneWolf]

I wondered why WIndows said one of the folders was corrupted yesterday but Linux could still read it fine.. I'm not keen on usin FAT32 for the reasons you mentioned. I also have a lot of large files to store on this drive, and I think Fat32 might have too much of a small file size limit. It's an awkward one eh! I need to store audio samples on this drive so I can access the same library with LInux and WIndows. I'll have to see how it goes with NTFS for a while :/

I'm not a complete LInux noob I used Ubuntu and Ubuntu Studio at various points between about 2010-2013 and I remember using the terminal a bit back then to do things like changing the desktop environment. But I can't remember any of the console commands I used to know.. I have been using Mint a bit prior to my KXStudio install too. But other than that, yeh I'm pretty much a Linux noob I suppose

"etc" sounds like another one I'd want to backup then! I could do a system image, but it I would like more storage space for that, so maybe I should get some. Sometimes it is just better to backup a small amount of files though. Backing up the entire drive is a bit overkill to do every backup..

Thanks for the detailed explanations of terminal commands. One of the reasons I don't just like to copy and paste commands people give out is because I don't actually understand exactly what those commands are doing, and that kinda freaks me out incase I break my system :/

It sounds like rsync is really the thing to use though, so I'll have to get to grips with it and make it do what I want. I'll be coming back to this thread to re-read everyone's advice as I make more backups.

Thanks all. Really appreciate the help

[Luc]

Backing up the entire drive is a bit overkill to do every backup.

Actually, no. The first backup does take a while, but subsequent sessions will only update the changes and take a fraction of the time.

[AloneWolf]

@Luc - That's not doing a full backup everytime though. I thought that's what they were saying..

The mirror/sync thing is pretty cool, I like that. But as I said before, it kinda sucks if you end up mirroring corrupted files and don't have copies of old versions that aren't corrupted. I think what I would like to do now is make a sync/mirror of my entire drive, then additionally use a method like incremental (or something similar) to make backups of the home folder so I have old versions of documents incase my mirrored backup ever backs up corrupted files. I don't know to set this up with rsync yet, but I'll have to try and figure it out!

Also guys, I noticed something that's causing trouble for my plans. The NTFS storage I have needs my password to access it from Linux form some reason. Is there a way I can stop that? I've noticed it's causing issues with programs. For example, I have told Clementine that my music library is on that drive, and it has scanned it, yet nothing shows up in the Clementine library. I suspect this is something to do with it needing a password to be opened?

And I still don't understand if LInux has something like WIndow's Volume shadow copy service. I would need some similar functionality to backup system files that are in use, right?

Thanks all

[Luc]

And I still don't understand if LInux has something like WIndow's Volume shadow copy service. I would need some similar functionality to backup system files that are in use, right?

I don't think so. Linux doesn't lock ordinary files like Windows does. I've never seen or heard about that except in Windows.

Linux locks files in /proc and /sys but you definitely don't need to backup those directories. See the concept of --exclude in the discussion above.

[lazyklimm]

Seafile