• I subscribed to JungleDisk using Amazon S3 storage. It backed up about 30Gb of data (in about 10 days! ouch!) and then ran in the early morning to keep itself in sync.
• I set up an rdiff-backup script to mirror the important stuff to an external USB drive.
• I created a subversion repository for all of my installers, tools, installation CD iso images (which I created using MagicISO)
• I copied my entire media library onto a cheap, huge external SATA drive and brought it to work, which is a more secure location that my house.
• I mirrored my svn repository onto that external drive as well and set up a batch script to update it weekly.

All in all, this was a really good first effort. I was happy and felt secure, though I haven’t once had the need to use this wonderful system. I have, however, found some parts of it to be annoying. First of all, JungleDisk is SLOW. Really, really slow. Well, ok, it’s only slow to upload. It took TEN days to upload about 30Gb of data, which is about 34kb/sec. As a test, I did some uploads to servers around the country (I have friends in fun places), and averaged about 100kb/sec, so in my opinion either Amazon throttles their incoming bandwidth, which I can understand, or my route to S3 stinks. In addition, it slows my computer down when it runs. I feel like that’s really out of the question for a modern application.

These things are annoying, but they aren’t a hill to die on. What was really bothering me was the cost. I had estimated that it would cost about the same each month to use S3 as it does to use mozy.com or other similar options. I was wrong! It costs $4.95 each month for mozy, and about $7 each month for S3 with my usage. It’s not a huge difference, but it is annoying.

Back when I was looking at online storage, I tried mozy seriously and found the biggest weakness to be their Mac client, which I failed to note at the time was just freshly out of private beta. On a whim, I tried it again, still in beta but a tad newer. It still does that thing every time you start it up where it tries to scan your probable backup sets even though I don’t want it to. It does not, however, keep crashing and it is, to my delight, a LOT faster. I got my averaged upload speed when I let it run untethered. And, even when I did that, my mac barely noticed it was doing anything major with its network connection. The client lets you throttle the bandwidth use during a specific time range, so I turned it down to about 48kb/sec when home in the evenings so that the wife doesn’t notice that it’s chugging away. (She definitely noticed with JungleDisk.. low WAF to be sure!) It, therefore, uploaded the same 30Gb of data in under FOUR days. I even got to tail its log file and watch it upload each file. Exciting stuff, I tell you.

BUT, to complicate matters in the mean time, I had started using svn to store my digital images as well, so all of those lovely .svn directories were lying around, and there was no smart way to tell mozy to leave them alone. So, I devised a fairly straightforward workaround: I modified my rdiff-backup job to ignore these files and populated my mozy backup set with the rdiff’d backup set instead. I stagger the cron and mozy jobs to keep everything in sync, up to date and backed up. It works quite well, and I never have to look at it. Ever.

I do, of course, look at it every single day because I’m paranoid. I’d love to say I’ll start trusting it to just keep working, but to be honest, I like knowing for certain that my data are safe in the event of a flood, fire, EMP, etc.

I also installed the windows client at work under a different e-mail account (I only want the 2Gb free service and you can’t use both with the same account) and use it to back up my work documents. We use Acronis at work, but also use PGP whole-disk encryption and I just don’t trust that my drive won’t some day get itself all corrupted in a tizzy. Acronis fails semi-randomly and so far mozy doesn’t, so I’m not going to take any chances with stuff that’s important because, well, that’s the whole point. Besides, mozy’s windows client includes an awesome mapped drive that lets me browse right into my recent backup and grab files as needed. Seriously, when they add that feature to the mac client my backup life will be complete. That was, actually, one of the only things I really liked about JungleDisk.

So, in the end, I only changed direction slightly. I have yet to do any of the other things I needed to do for extra security, though I did lock my mac to my radiator pipes when I went away for two weeks. It made prudent sense at the time, though when I think about it now, it seems silly.

Recently, I thought it would be useful to have all of my digital photos in the repository, both for backup purposes as well as to allow me to, say, have all of my downloaded desktop wallpapers on my machines at work and at home. What better purpose, right? The biggest pain was keeping the repository up to date without constantly needing to issue pseudo-random svn add commands every time we take pictures. (I could use scplugin to make this easier, but as I found out, it kinda sucks.)

Instead, I turned to my love for shell scripting, and fairly quickly banged this out. I used the handy svn status command with impunity to help me handle recursive adds and deletes. Of note, yes, I wish SyntaxHighlighter worked here…

#!/bin/sh

store='/Users/shelton/Pictures'
date=`date`
message="Auto-Commit on ${date}"

# Update the local store
svn update ${store}

# Recursively delete all manually deleted files
svn status ${store} | sed -e '/^!/!d' -e 's/^!//' -e 's/^ *(.*)/1/' -e 's/ *$//' -e 's/ /\ /g' | xargs svn rm

# Recursively add all manually added files
svn status ${store} | sed -e '/^?/!d' -e 's/^?//' -e 's/^ *(.*)/1/' -e 's/ *$//' -e 's/ /\ /g' | xargs svn add

# Commit! Comment includes timestamp.
svn commit -m "${message}" ${store}

exit 0

It would be really nice if there were SyntaxHighlighter support for shell scripts. Maybe that’s worth working on myself, eh? Updated 2009-01-12: Sha-Zam!

This also required some modification to my rdiff-backup cronjob since, well, I didn’t want to back up every single .svn directory and included files as that would effectively double the backup requirements for that directory. So, I played a bit more with rdiff-backup’s options and ended up changing from this:

45 04 * * * rdiff-backup -v5 --print-statistics --exclude /Users/shelton/rdiff-backup.log --include /Users/shelton/Documents --include /Users/shelton/Pictures --include /Users/shelton/Music --exclude '**' /Users/shelton /Volumes/backup/BACKUP >> /Users/shelton/rdiff-backup.log

to this:

rdiff-backup -v5 --print-statistics --include-globbing-filelist /Users/shelton/cfg/rdiff-backup-files.txt /Users/shelton /Volumes/backup/BACKUP >> /Users/shelton/logs/rdiff-backup.log

This is obviously MUCH cleaner. rdiff-backup-files.txt looks like this:
- /Users/shelton/**/.svn
- /Users/shelton/logs
+ /Users/shelton/VM
+ /Users/shelton/Documents
+ /Users/shelton/Pictures
+ /Users/shelton/Music
+ /Users/shelton/Sites
+ /Users/shelton/bin
+ /Users/shelton/cfg
+ /Users/shelton/.tcshrc
+ /Users/shelton/.vimrc
+ /Users/shelton/.crontab
+ /Users/shelton/.ncftp
+ /Users/shelton/.screenrc
+ /Users/shelton/.htpasswd
- **
That first line, - /Users/shelton/**/.svn, is by far the most important, and took way too much trial and error to get right. Order of statements is, apparently, VERY important for rsync/rdiff-backup. Another note about the above list, I’m now backing up things that I wasn’t before, namely my VM directory, which contains my Parallels virtual machines. It started to make prudent sense, though currently the only VM in there is a very tiny cfg fileset to virtual-boot my BootCamp partition. I’ve started doing several things differently with regard to my home backup, which I’ll have to summarize at some point soon.

• were too expensive
• were too time consuming
• had low WAF

I stumbled upon a 500Gb drive for under $90 on TigerDirect, which allowed me to keep data off-site in a high-security location for relatively little effort. That was luck.. keeping it up to date, well that’s tougher. I can bring the drive home and sync it locally, but who wants to do that on a regular basis? Corporate firewalls keep me from realizing that potential. The other mitigating factor is that one of the items I would like in sync between the two is all of my music (all ~200Gb of it). And while the initial copy was done on my LAN at home overnight, my iTunes libraries are in two different OS formats, so even if I did copy all of the changed files back to my home drive, I’m still SOL once the files are there. Somehow I have to know to update my library.

I had a brainstorm in the shower (where all of my good geeky ideas originate) that I could create some sort of complex set of AutoHotKey, iTunesSDK/JavaScript or ActionScript [scripts] that would copy over an updated library XML, shut down the local instance of iTunes, compare the XML files, write out a library modification script, start iTunes, run the script and quit. The trouble for me was determining which library would be the master, and if I didn’t want either to be the master, how do you reconcile changes? Maybe create two sets of modification files and queue up the returning file for delivery later on? What if iTunes is in use?

Anyway, maybe that’s a good project for a rainy/snowy weekend. I’d need some existing rsync system in place first. I’m getting ahead of myself, and WAY off topic.

The drive also keeps a copy of my home SVN respository, which I update via a Windows Scheduled Task once per week.

So, after about a month or so, this is my result:

1. Evaluate online backup options. Done. I ended up sticking with JungleDisk for now, until something better comes along, or I discover mozy’s client sucks less.
2. Set up an automated incremental backup of all of my documents/pictures to a local, external drive since JungleDisk is so painfully slow. I’m using rdiff-backup to keep a local copy, but will switch to Time Machine after I upgrade to OS X 10.5.
3. Install a large SATA drive (>= 300gb) for my workstation (at work) for my music and possible offsite SVN mirror. Done.
4. Investigate other offsite SVN mirror locations (family? cheap web hosting?) Deferred.
5. Get my local SVN repository in order. It’s in pretty good shape but it can’t hurt to re-think the structure a tad. Add a tree for installation media disk images. This is done. I’ve updated it from both sets of “Software I Install” directories on each operating system. In the process, I created a tree for tools (non-install, runtime exe’s) which contain a lot of little things I use once in a while such as the SysInternals (now Microsoft) suite.
6. Convert all DVD and CD media to their respective disk image formats and checked into the SVN repository. Done. This was way too easy.
7. Set up the HDD at work (Done) and rsync my music over. Deferred.
8. Set up the offsite SVN repository and mirror my repository at home. Create a cron job to do it automatically every night. Done. This works very well.
9. Scan all of my really important, hard-copy files into PDFs and add them to the Documents tree in my backups. Deferred.

This concludes the first phase of this project. My important data are safe and easily recoverable. My unimportant data are somewhat less safe, but that’s fine. Everything runs automatically, and I don’t have to think about it much. The best part of this is that my wife doesn’t have to know about how all of this works. It just does, and that’s high WAF if I’ve ever heard it!

What’s Next?

After all of this, I still have some tasks I’d like to accomplish so that I “feel more secure”:

• Locks for each computer. This seems silly since the house locks, but if the workstations are all locked to their desks, then a theif has to decide if they budgeted time to hack apart my desk to get my iMac or not.
• Scan all of my really important, hard-copy files into PDFs and add them to the Documents tree in my backups. I plan to do this by the end of the year.
• Investigate other offsite SVN mirror locations. Ideally, this would be at a family member or friend’s house, connected to a machine on which I could run an SVN update automatically. Finding someone willing to let me do that is the tough part, and the reality is that the best option is a low-power linux box with a ~80gb drive that someone lets me toss into their closet. I have the drives and some of the hardware, so really I just need to allocate a couple hundred clams for the guts of the machine… oh, and find someone with a nice internet pipe willing to let me hide a computer in a closet. Details.
• Rsync music and iTunes library. That’s a project in itself. We’ll see.

I may do some or all of those some day. For now, I feel relatively secure in my data’s security.

Home Backup Project – Part 1: Identify the Problem
Home Backup Project – Part 2: Plan!
Home Backup Project – Part 3: Subversion Clients
Home Backup Project – Part 4: Creating ISO images
Home Backup Project – Part 5: Evaluating Incremental Backup Options

• mozy.com
• JungleDisk and other Amazon S3 utilities
• rdiff-backup
• Time Machine

mozy

Since one of the muses for this project was a post over at Computer Zen, I followed his lead initially and signed up for a free account at mozy.com (using his referral code, of course). Overall, the concept sounded fantastic. They have clients for Windows and Mac, unlimited backups for $5/month (or less, when purchased in bulk) and a fairly solid fanbase of users who are very pleased.

From a security standpoint, mozy also seemed very good. An article over at MacApper noted that files are encrypted with a 448-byte key and all transactions take place via SSL. (The last part seems a no-brainer, but having my data encrypted makes me feel all warm and fuzzy inside.)

I installed the mac client and started to play with it and was instantly turned off. There’s a huge design flaw right up front… the software does not ask me what I want to do, it just starts doing what it thinks I want it to do – spidering my hard drive for likely content to be backed up. Now, even iTunes and WMP ask before taking up significant CPU resources in such a way. Not mozy, no sir. It knows that you want it to do so!

I may be an uncommon geek, but I’m fairly well organized. A child could probably find a file on my computer (without spotlight) simply by knowing what they’re looking for. Is it a document? Yes, it’s probably in Documents.

Anyway, all I wanted was the dialog that says “pick your files to back up”, which is the second part of the interface. I had to wait almost ten minutes before I could do that. But, for the sake of argument, I waited. After that one annoying “feature” everything else about mozy is actually pretty nice. It tells you how much space you have left, how much your backup will take (the trial is limited to 2Gb) and what files you’re backing up. You tell it when to do it’s work, and it goes off into the background and does its business.

So, for $0, I had 1.8Gb of data backed up, which really just amounted to my Documents folder and my iTunes library files. I need about ten times that for my images and other data I want to safeguard this way.

What I liked: Speed of backup; Easy way to download backed-up files or order DVDs as needed; Cost
What I disliked: Annoying up-front user behavior assumption; Immature client – it started crashing randomly after about a week

I’m told that this was a fairly new client.. the Windows client is supposed to be much better, so that’s good to hear. For my purposes, however, all of the data I care about in my house is on my mac. (My wife’s laptop and my linux server are all recoverable with little-to-no effort and don’t contain any data that aren’t anywhere else. My office workstation gets backed up at work, so I don’t need to worry about it here.) I don’t much need the Windows client, so to me, the killer app is the Mac client, which was somewhat lousy. Too bad… Scott had such nice things to say about mozy.

One additional note, however. If I were in Scott’s position, and were backing up multiple computers with multiple platforms, mozy might make even more sense. If, for instance, I had a true family approach to this, and were trying to keep my and my wife’s familys’ computers all backed up in one place, then this might make sense. For us, on one mac, mozy didn’t completely fit my needs, purely from a client perspective. Were the client overall more usable and stable, I think I might have stayed.

JungleDisk / S3

Initially, mozy was the only online option I was investigating intensively. I had looked at others (Carbonite, iDisk, box.com, etc.) but didn’t like various things about their services and stuck with mozy. About a week into the process, I read a comment thread over at LifeHacker about this very idea. Several commenters noted that they had great success with JungleDisk, which uses Amazon’s S3 service for storage. The S3 pricing model is pretty cheap, so its barrier to entry was quite low.

JungleDisk installs as an internet drive (similar to iDisk), though it’s actually connecting to a service running on your machine. When you add/change a file on this drive, it caches it locally and then queues it for upload to S3. First of all, the upload speed is SLOW. There were times, during my initial 20Gb push, that the speed dropped to less than 1 kbps. To me, that’s abysmal. I had to let it run for over a week to complete the upload. It might be faster to download data (I should probably test that, huh?), and that’s what’s really important now that the backup has completed.

One nice thing about S3, however, was that I could upload data to it, and not leave a copy on my mac. I have about 5Gb of old archived data that, if you remember, I “lost” for a few weeks. I don’t need it around all of the time, but I sure do want it somewhere. S3 sounded like a good place.

The client is robust and backs up exactly what you ask it to back up. It runs on whatever schedule you set. I have it running nightly. It costs about $20 after a 30-day trial, but for backup software it’s probably worth it.

What I liked: Overall storage cost; Set it and Forget it; Usable for non synchronous data
What I disliked: Client cost; Upload speed

rdiff-backup

rdiff-backup is a unix command-line util to perform backups by, gasp, recursively diff-ing. I know. The name is a real misnomer.

It’s written in python, and it’s pretty darn snappy. It backed up about 20Gb in 90 minutes (which doesn’t seem that fast, but I was copying to mr. slow external “says it’s 2.0 but acts like 1.1″ removable drive), and then about 12 hours later ran an incremental update in 4 minutes. Not too shabby, I suppose. I also set it up to run in cron every night:

45 04 * * * rdiff-backup -v5 --print-statistics --exclude /Users/shelton/rdiff-backup.log --include /Users/shelton/Documents --include /Users/shelton/Pictures --include /Users/shelton/Music --exclude '**' /Users/shelton /Volumes/backup/BACKUP >> /Users/shelton/rdiff-backup.log

Then, since space is limited on that drive (though not too much), I also set it up to remove increments that are older than three months. I can always trim that down if space becomes a premium:

00 04 01 * * rdiff-backup --remove-older-than 120D /Volumes/backup/BACKUP

..and that’s it. Pretty darn simple, I must say. Once of the really nice things about rdiff-backup is that the most recent revision is always sitting there as normal files. I can use any standard file system command (cp, mv, tar, etc) to move/copy files out of that archive in case I lose something. It’s not as space-conscious as incremental tarballs gzipp’d, but it’s what makes most sense for me.

rdiff-backup also has some nifty restore features that seem promising.

After using it for a few weeks, however, I noticed that the include/exclude flags don’t really work as I’d like them to. For instance, I want to include all of ~/Documents, but not ~/Documents/Parallels (because I have more than one Parallels VM, and that takes up a LOT of space). Shouldn’t –include /Users/shelton/Documents –exclude /Users/shelton/Documents/Parallels work properly? I guess not. It pattern-matches the directories, giving precedence to the include statements, so since /Users/shelton/Documents/Parallels has /Users/shelton/Documents in it, I’m out of luck. I ended up thinking different and moved Parallels up a directory and symlink’d back to it so now I have /Users/shelton/Documents/Parallels -> /Users/shelton/Parallels. It doesn’t know the difference because, well, UNIX is the r0x. I added an exclude statement for that directory and I’m in business, with about 20Gb reclaimed.

While figuring out that mess, I noticed that the pruning task wasn’t quite effective. My drive filled up within two weeks, and I needed to set the prune to 14 days, then 7 days, and then 5 days to recover space enough to even start the following backup. I ended up running the prune task 45 minutes before the backup task every day and setting it to remove anything older than two days, which somewhat defeats the purpose of it keeping previous versions in the first place. I may be too busy (or out of town) to notice something’s been gone in under two days. Again, I could just buy a new hard drive, but I’m also aiming for “cheap” in this project. After getting rid of Parallels, I was able to put this back to 7 days, so all in all, this isn’t such a bad option.

rdiff-backup can be used to backup remotely, but you need to provide a remote machine and unless you can mount it as a drive, you need rdiff-backup to work properly on the remote machine as well. I could not get it to work on my ubuntu server in the basement, so that was somewhat annoying. I thought about using it to sync to S3, but the overall speed of S3 was so slow that it wasn’t worth it.

Time Machine

Leopard isn’t out yet, so I couldn’t fully evaluate Time Machine. However, everything I’ve seen and read leads me to believe that when I do upgrade, I’ll also buy an external drive 2-4 times the size of my internal drive and point Time Machine at it and just pretend it isn’t there. It also doesn’t satisfy the “remote” aspect of this part of the project, but it might be yet another way to keep my data safe.

Conclusion

So what did I end up doing? Well, a little of column A, and a little of column B.

My disgruntled-ness with mozy’s poor client was enough of a detracting factor for me that I ditched it entirely. If I had a windows household, I probably would have kept it. In the end, I stuck with rdiff-backup running every morning. I’m almost certain I’ll switch over to Time Machine once I upgrade to 10.5, but if I do, I’ll probably want a bigger backup drive than what I have.

Home Backup Project – Part 1: Identify the Problem
Home Backup Project – Part 2: Plan!
Home Backup Project – Part 3: Subversion Clients
Home Backup Project – Part 4: Creating ISO images

Worst case scenario, I think.

Anyway, I thought this would be really easy on my mac:
dd if=/dev/disk4 of=/path/to/new.iso
This always worked back in my I only run linux because windows users suck! days.

This created an unmountable file in both windows and os x. The unix gods have failed me! (In retrospect, I was probably forgetting a command line operator, or something, and therefore actually failing myself, but I digress…) I then thought that perhaps Roxio Toast Titanium would make my day, but to no avail. The solutions I had in front of me evaporated rapidly.

I did some quick googling and found MagicISO, but it’s !(free), and I’m trying to do (most of) this on the cheap. Some more research led me to two tools:

• LC Iso Creator – http://www.lucersoft.com/freeware.php
• VaporCD Volume Manager – http://vaporcd.sourceforge.net/

Each seems to create ISOs from CD fairly well. So far, WarCraft III backed up nicely enough for me to install off of the ISO image on my Parallels VM, so that’s confidence inspiring. On the other hand, I received some very weird errors reading one of the Unreal II disks, so neither of these may be a perfect solution as I checked the disk itself and it’s fine.

However, both of these programs are wicked old… VaporCD was last updated six years ago and hasn’t been tested on XP at all. There’s got to be a free-ish, newer solution for this somewhere. Perhaps I just haven’t stumbled across it yet. Rather than continue stumbling, I kept playing around with both and determined they were, for the most part, crap. Plain and simple.

I’d like this fact to explain their free-ness, but all of the mozilla apps are free (as in beer) and they rock my socks off, so that’s right out. (I mean no insult to the developers of either of the above applications. To be frank, I expected something fairly more useful than dd, but my hopes were dashed. End rant.)

I asked for some advice at work and everyone said “Go download MagicISO.” So… I did. And it’s so much better. It meant “spending $30,” but it’s fast, and doesn’t choke on minor block corruption due to surface scratches, and it’s… fast.

Really, really fast.

I blew through all of my games and OS install CDs (about 25 total CDs) in under 90 minutes and just let it run in the background while doing other work. I finished up the rest of my disks the next day, which brought this phase of the “things to do before I get to do what I’ve been wanting to do” to a very quick close.

Sometimes you just have to “pay” for software. Gasp!

Home Backup Project – Part 1: Identify the Problem
Home Backup Project – Part 2: Plan!
Home Backup Project – Part 3: Subversion Clients

On the mac at home… well, that’s a different story. I tried SCPlugin, which seemed to be the Mac version of Tortoise. While it started out looking that way (the icons showed up for existing checked-out repos), it wouldn’t do a base PROPFIND for my local store, and then it crashed. Hard. And stopped working, even after a reboot. Why? I have no idea. That’s the great thing about a mac… software either works or it doesn’t, and SCPlugin doesn’t.

I then tried svnX, which had some promise, but after 15 minutes of poking, I had no idea how to add anything to repository. No clue at all. I’m sure it’s simple once someone tells you how.

Then SmartSVN. Same deal.. I couldn’t add a file, just a directory, and that seemed wonky. It’s probably me, not the software, but I wanted something usable enough that in a groggy stupor, I could update my local store and not need to think twice about what I was doing.

MacSVN also showed some promise, but it crashed on startup needing a newer version of Berkeley-DB than is available for the Mac. I want simple, and hacking up code I didn’t write in the first place to use a non-stable version of another software package falls slightly out of scope.

Then, I went back to the folks at tigris and poked around their site. I found RapidSVN, which is a GUI front-end for SVN with some additional bookmark support. That’s it… and somehow, this works so much better for me than any of the other options. You check out a local store and update it via finder (or any other file system tool, i.e. mv/cp/tar/etc.), then add/delete/commit as you need to through either RapidSVN or using the svn command line, and it all works together nicely.

This was my experience – I don’t plan on maintaining any source code in this repository (for now), so my uses are pretty one-dimensional – it’s a glorified password-protected file storage mechanism with an existing protocol for mirroring, versioning and access from anywhere.

After deciding on a method for utilizing SVN from anywhere, I was able to get my repository in good working order. The other major benefit to using SVN is that I can access the repository via a web interface, and when I need a file from anywhere, that sure is handy. The structure currently looks something like this at the top-most level(s):

• Applications
  • Mac
  • Other
  • Windows
• Disk Images

Of note, I plan to use svnsync locally on my svn server to keep a secondary mirror of the repository on an identical drive. There’s a great wiki article on on Mirroring a Subversion Repository over at the OpenDS wiki.

Home Backup Project – Part 1: Identify the Problem
Home Backup Project – Part 2: Plan!

Another thought is that I might want a hard backup available at a secure, physical location – A bank’s safety deposit box, or a family member’s house, perhaps. If I had a blu-ray or HD-DVD burner, and could afford to have a stack of said media lying around, or if I felt like dropping money on a REV drive and its media, then perhaps that would be an easy solution. As it stands, my only option would be to burn 2-3 Dual-Layer DVDs (or one Blu-Ray) every 3-4 months and mail it to someone’s house. Maybe there’s another one out there, but that will require some investigation. I do not want to buy a set of external drives and ship them around. On the other hand, I do have two 60 Gb IDE drives sitting in the basement doing absolutely nothing, so maybe that’s _not_ a bad option.

This is what I came up with:

To get to Initial Result, the following should be accomplished:

1. Evaluate online backup options, which seem to be:
   • mozy.com
   • JungleDisk or some other client that uses Amazon’s S3 service.
2. Install a large SATA drive (>= 300gb) for my workstation (at work) for my music and possible offsite SVN mirror.
3. Investigate other offsite SVN mirror locations (family? cheap web hosting?)
4. Get my local SVN repository in order. It’s in pretty good shape but it can’t hurt to re-think the structure a tad. Add a tree for installation media disk images.
5. Convert all DVD and CD media to their respective disk image formats and checked into the SVN repository.
6. Set up the HDD at work and rsync my data over. It might be smart to do this locally first since that’s a LOT of data.
7. Set up the offsite SVN repository and mirror my repository at home. Create a cron job to do it automatically every night. This shouldn’t take too much bandwidth after the initial load since I shouldn’t be messing with it too much.
8. Scan all of my really important, hard-copy files into PDFs and add them to the Documents tree in my backups.

To get there, I needed to purchase:

1. A subscription to the online service of my choice:
   • mozy.com is about $54 per year, though if I sign up for 2 years, I get 3 months free.
   • The JungleDisk client is $20, and Amazon’s S3 service is use-based, which also should translate to about $5/month.
2. A large SATA drive. (TigerDirect FTW!)

Overall, what I came up with is not too bad. Some of the tasks, as they are planned now, looked to take a lot of time, but in the end, what they will save me in stress, anguish, annoyance and time spend re-building everything is invaluable.

After even more thinking about this, that last task of scanning important documents seemed a bit out of scope for this project, and I’ll pick it up some other time. Suffice it to say that would take a LOT more time than anything else, and I also have a job and a wife.

Home Backup Project – Part 1: Identify the Problem

1. On a primary OS hard drive, doomed to likely failure in the next 18-24 months
2. On DVD
3. On a generic removable drive that has, at times, decided it no longer wanted to be a hard drive

I set out to figure out how to remedy this in a reasonable and cost-effective way. I don’t want to break the bank or waste my time with ridiculous backup systems that are more of a hassle than anything else.

Data Snapshot

Here’s what I determined I had to worry about losing and where it was stored:

• Documents and other archived files I need/want to keep – ~8 Gb
• Digital Photos – ~16 Gb
• Music – 200 Gb (I’m an avid collector!)
• Digital Video and Movie backups – ?? Probably 2+ TB (yes, that’s terabytes), recorded to DVD+/-R
• Non-CD Application Installers (Mac and PC) – ~18Gb
• Installation Media – ??

Scope of Backup and Other Considerations

Obviously I couldn’t plan to back up EVERYTHING properly. I’m not going to sit down and re-copy all 300ish DVDs of video just to make sure I don’t lose anything. I put them on DVDs for the very reason that they’re something I could technically afford to lose. I have renter’s insurance to cover my purchases. Same goes for my music, but since I access that a LOT more, I’d rather have it backed up digitally somewhere so that in the event that p00p does indeed go down, I can still hear my favorite tunes.

Any regular backup system I came up with needed to be 100% transparent. I don’t want to sit at my computer every morning and approve something, or check something, or audit something. This has a low WAF/PAF ratio.

There are also many files I only have in hard-copy that I should probably have a digital copy of somewhere, so that’s something I’ll need to build into a project plan.

Initial Project Sketch

This is what I came up with to begin this process:

1. Do a quick backup to an external drive NOW just so I’m covered.
2. Figure out the end game for my data as well as what are some acceptable “for now” options
3. Figure out any intermediate steps or tasks to prepare my data to be backed up
4. Assign a time line and any additional resources
5. Budget for anything worth budgeting
6. Implement!

Relevant Links

Each of these were inspiring or helpful in some what throughout this process:

• http://www.engadget.com/2007/03/21/how-to-automatically-back-up-your-computer/
• http://lifehacker.com/software/geek-to-live/geek-to-live-automatically-back-up-your-hard-drive-147855.php
• http://maczealots.com/articles/backup/
• http://www.hanselman.com/blog/OnLosingDataAndAFamilyBackupStrategy.aspx
• http://lifehacker.com/software/mac-tip/create-an-encrypted-disk-image-at-the-command-line-279005.php

In Part II, I’ll get into what my “vision” is regarding my data. In addition, I’ll set up a basic project plan for how I’ll accomplish this new initiative.

First, the fire alarm at my house went off… exactly once. It was a horrifically loud squelch, and I sprung out of bed and raced around the house to find the source of the alarm since there was no smoke, no heat and no ambient light. The backup batteries in the detectors probably needed replacing.

The alarm lasted only for a split second, but it was enough to scare me in thinking that if there had been a fire, I’m not sure I would have had the piece of mind (or time) to grab my DVD binders of movies, my iMac with all of my files on it, the external drive containing all of my music, or the subversion server in my basement with all of my downloaded applications and installation media.

Lucky for me… no fire, just panic… and really, my first thought should never be “grab my data!”, but rather “grab my wife! and the cats!”

About two months later, a very sobering “data loss” happened. From middle school to when I graduated from Grad School, I’ve been archiving every document I wrote, work on every project I ever completed as well as any data from previous jobs I’ve held. I’m sure this is because I’m always wondering if I may at some point need “something”. It could also be for the edification of my future offspring… or at least, it would have been.

I suddenly had this panicked feeling that I didn’t know where my data were… so I started looking. And looking. And sure enough, they were totally gone. Ten years of work. Gone.

A moment of silence was had… then I started thinking – what I can do to prevent the sinking, terrible “your data are gone” feeling? The end result will be a a series of posts going through my analysis and implementation of the proper home backup strategy that I have implemented over the past months.

Incidently, I’ve since discovered that these data were actually not lost, but were so safely stored on my hard drive that even Spotlight had no idea where they were. I’ll have to remember that ~/tmp is un-cataloged… which really, when you think about it, makes good sense.