iTunes Backup

On 1/16/07 at 7:11 AM, Nik <gerberinik.net> wrote:

> Interestingly, SuperDuper! has actually declined in quality, and
> no longer reliably backs up extended attributes.

Can you elaborate on this? Is that based on your own experience/testing,
or did you find another article that reports this?

- Todd

On Jan 17, 2007, at 7:01 AM, Todd Ruston wrote:

> On 1/16/07 at 7:11 AM, Nik <gerberinik.net> wrote:
>
>> Interestingly, SuperDuper! has actually declined in quality, and
>> no longer reliably backs up extended attributes.
>
> Can you elaborate on this? Is that based on your own experience/
> testing,
> or did you find another article that reports this?

This was based on my own testing.

Specifically, I took a text file and manually (via the terminal)
added a resource fork and a custom extended attribute. Additionally,
TextMate left a "cursor position" attribute in the file as well.

I then copied this file to a disk image and used SuperDuper! to copy
the disk image to an empty disk image.

The file copied without problems, but upon examining it with xattr, I
found that while the resource fork stuck around, the attribute I had
added and TextMate's cursor position attribute were both missing.

I repeated the SuperDuper! copy a couple of times, and had the same
results each time.

I cannot say whether ACLs (access control lists) would also have been
affected, since they are also stored in the extended attributes of a
file. I'm pretty unfamiliar with ACLs and thus didn't devise a test
for whether they were maintained.

--Nik

I don't understand why backing up iTunes data (and iPhoto for that matter) has to be so complicated. Here's a simple method that takes zero minutes of your time (with three caveats) once it's up and running.

Get two 256 GB (or greater) hard drives and use Disk Utility to configure them as a RAID pair. Mine are Firewire. This produces an identical pair of drives that are viewed by the system as a single drive. You can use them in real time even though they are the also the backups.

Caveats (fact - all hard drives will eventually fail)

1.- If a hard drive has a probability of X of failing, two drives failing simultaneously has a probability of X*X, So this gives you a window of time to replace the failed drive unless you are incrediblely unlucky. When one drive fails you replace it with a new drive and revive it with surviving the drive. The mainframe system I worked on could do this while surviving drive was being read and written to. I don't know if Apple can do this. You may have to do this without any other access to the drives. You could improve on the recovery window by having a third drive handy.

2.- the drives are co-located and an on-site disasater can take out both drives. An off-site copy is advisable for those concerned by rotating in the above mentioned third drive and using the revive method noted above. Of course you have to plan for the window of exposure that the off-site drive is out of sync with the RAID pair.

3.- a computer crash during a write to the RAID set may cause the volume to be unmountable. I don't know how Apple journaling would protect against this and have no way of testing this. [Before I went to a mirrored pair, my iMac crashed with the Grey Screen of Death in 5 languages no less, while I was downloading iTunes purchases. This crash was equivalent to pulling the firewire cable during a disk update. Bad. This was compounded by a tactical error I made when I took the Mmfgr at their word that the drive was usable out of the box for either Windoz or Mac. Also Bad. Besides the future-think that someday I could interchange the firewire drive with Windoz (my now laptop does not support firewire), the mfgr's default format is inefficient and wastes disk space AND the drive could not be accessed by any number of disk recovery utilities.]

4. OK, so there is a fourth one. If you also use the RAID drives as a backup media, recovery is brute force effort of either copying the whole folder back or doing on file at a time. But that's been covered here before using other uilities.

HTH. Regards, Mike

I have used and recommended Shirt-Pocket's SuperDuper! since its beginning.
I inquired about Nik <gerberinik.net>'s comment and Shirt-Pockets reply
is:

"It's quite possible that there's been a regression in 10.4.8, Dean, and
that's affected our EA copying. We're looking into the issue and will
correct it."

I have always been extremely satisfied with Shirt-Pockets products and
their response to inquiries and problems. This response, which was
instantaneous as usual, was not defensive and was positive to it's end.
Dave Nanian (Shirt-Pocket!) is great.
                                                 
  Dean Tromanhauser | GPAG Systems | Extension: 55699
            E-mail: rdtcapgroup.com
 

Subsequent communication from Shirt-Pocket about SuperDuper!:

"Ah. I've got a very good handle on this now. Here's the deal.

When a file is copied or updated, its EAs are copied as well. They are
definitely copied - I've verified this on a number of systems.

If that file is changed, its EAs are copied along with the file.

If, however, *only* the EAs are changed, and the file itself is not
modified in any other way, the EAs are not updated. But if the file is
modified in any way, the EAs are copied and updated.

We're looking into this case now: obviously, we don't want to have any bugs
in SuperDuper!, and this certainly is one. However, given that it only
happens when just EAs are changed (and that EAs are in very infrequent
use), it's unlikely to affect the vast majority of users.
--
Dave Nanian
Shirt Pocket"


                                                 
  Dean Tromanhauser | GPAG Systems | Extension: 55699
            E-mail: rdtcapgroup.com

On 1/17/2007 6:01 AM, "Todd Ruston" wrote:
>> Interestingly, SuperDuper! has actually declined in quality, and
>> no longer reliably backs up extended attributes.
>
> Can you elaborate on this? Is that based on your own experience/testing,
> or did you find another article that reports this?

I was surprised by this claim, as well, having tested SuperDuper quite
thoroughly and reviewed several versions of it; it's been the only
backup/clone utility I regularly trust with my own data for quite some time
now.

After reading Nik's comment about custom attributes on the referenced blog,
I passed it on to Shirt Pocket and asked for their response. They were
surprised, because, given the APIs they're using, all extended attributes
should be copied. So they did a bunch of testing over the past few days, and
figured out what the issue is:

> Ah. I've got a very good handle on this now. Here's the deal.
>
> When a file is copied or updated, its EAs are copied as well. They
> are definitely copied - I've verified this on a number of systems.
>
> If that file is changed, its EAs are copied along with the file.
>
> If, however, *only* the EAs are changed, and the file itself is not
> modified in any other way, the EAs are not updated. But if the file
> is modified in any way, the EAs are copied and updated.
>
> We're looking into what's involved with fixing this case now:
> obviously, we don't want to have any bugs in SuperDuper!, and this
> certainly is one. However, given that it only happens when just EAs
> are changed (and that EAs are in very infrequent use), it's unlikely
> to affect the vast majority of users.


I have to agree; this bug is very unlikely to affect most users. But it's
good that they're going to fix it anyway ;-)

(Just as impressive, IMO, is that Shirt Pocket regularly responds to email
questions like this within a matter of hours [or even minutes] -- and based
on feedback I've received from Macworld readers, it's not just because I'm a
magazine editor ;-) This is apparently par for the course for the company.)

At 22:17 01/17/07 -0800, mrnoonan wrote:
>1.- If a hard drive has a probability of X of failing, two drives failing
>simultaneously has a probability of X*X

But if you have a filesystem logic failure with probability X, then the
chance of both drives being affected is X.

Also make sure that your iTunes catalog is on duplicated media. I don't
know where it's kept on the Mac. I'm running iTunes on Windows, and it is
impossible to persuade iTunes to keep it anywhere but on the C: drive. That
leaves a huge hole in any scheme that counts on a duplicated external drive
as the primary means of data protection.

I've recently been backing up using Jungle Disk, http://www.jungledisk.com.
It uses Amazon S3 storage, which is cheap enough to store tens of gigabytes
as offline backup storage. This makes backing up both music and catalog
straightforward. The Windows version of JD is currently a minor release
ahead of the Mac version, but usually they are in sync. The newer version
contains built-in backup capabilities, making it more friendly to
non-techie users, though marginally -- at this time its still very much a
techie tool. At long last, remote backup has become a significant part of
my backup strategy, though still only one part.

Edward
--
Art works by Melynda Reid: http://paleo.org

am 18.01.2007 11:12 Uhr schrieb tidbits-talktidbits.com unter
tidbits-talktidbits.com:

> I don't understand why backing up iTunes data (and iPhoto for that matter) has
> to be so complicated.

I don't understand that either... There's a program called "Synchronize!"
which just backs up the changed files to whatever HD you tell it to do so.
It even keeps folder views the same.

I use it about two years now for synchronizing the data folders of four Macs
and am very pleased with it.

Udo

I've been working with one of the SuperDuper! folks about this loss
of extended attributes, and through a bit of further testing, I've
determined that this only occurs if/when the file being duplicated is
locked (or has the BSD flag "uchg" -- which is the same thing). If
the file is unlocked, all extended attributes are duplicated just
like they're supposed to.

I just emailed this discovery to the tech, and I assume they'll fix
this minor bug in a future version.

I do apologize for spreading fear, and I assure you it wasn't out of
malice, but rather out of concern. For the time being, make sure your
files aren't locked before you sync 'em though. :)

--Nik

Another follow-up. According to Shirt-Pocket, the issue is even more rare than originally suspected (Nik and the company were finally able to exchange emails and nail it down):

The file in question must be marked as immutable. The file must remain unchanged. A custom extended attribute must be changed, but nothing else.

In this particular scenario, the custom extended attribute change won't be reflected on the clone. According to Shirt-Pocket, this looks to be a bug in OS X which they'll be reporting to Apple -- and for which they'll be finding a workaround in SuperDuper in the meantime.

On 19-Jan-2007, at 10:45, Edward Reid wrote:
> Also make sure that your iTunes catalog is on duplicated media. I
> don't
> know where it's kept on the Mac. I'm running iTunes on Windows, and
> it is
> impossible to persuade iTunes to keep it anywhere but on the C: drive.

That's simply untrue. It's exactly the same as on the Mac
(Preferences -> Advanced)

(It's set to I:\iTunes\ on my PC)


--
Generalizations are always inaccurate. -Mugsy

Since we are on the subject of iTunes backup, I want to get feedback on an app that does "real-time" iTunes backup to S3. It has not been released but it does answer some of the issues here regarding the saving of items metadata.

We have built an "ituneswatcher" (runs on a separate thread) that uses the kqueue lib that "watches out" for any change in the iTunes XML, since metadata is stored there.

So our application saves things like play count, ratings, lyrics, itunes preferences, etc., basically stuff that does not get "synced" or backed up in real-time once you have a backup to a DVD or even a rsync solution.

Would love to get the community's thought on such a service.

More info here: http://xacktunes.com/

For those interested in beta testing, email me at terencepua at gmail dot com.

Den 22/01/2007 kl. 8.33 skrev Google Kreme:

> On 19-Jan-2007, at 10:45, Edward Reid wrote:
>> Also make sure that your iTunes catalog is on duplicated media. I
>> don't know where it's kept on the Mac. I'm running iTunes on Windows, and
>> it is impossible to persuade iTunes to keep it anywhere but on the C:
>> drive.
>
> That's simply untrue. It's exactly the same as on the Mac
> (Preferences -> Advanced)
>
> (It's set to I:\iTunes\ on my PC)

One thing: remember to consolidate the library when you have set a
new desired location.

Kim

On Jan 22, 2007, at 2:33 AM, Google Kreme wrote:

> On 19-Jan-2007, at 10:45, Edward Reid wrote:
>> Also make sure that your iTunes catalog is on duplicated media. I
>> don't know where it's kept on the Mac. I'm running iTunes on Windows, and
>> it is impossible to persuade iTunes to keep it anywhere but on the C:
>> drive.
>
> That's simply untrue. It's exactly the same as on the Mac
> (Preferences -> Advanced)
>
> (It's set to I:\iTunes\ on my PC)

isn't he talking about the iTunes Library database file? while i have
my iTunes library on a 500GB external firewire drive the database
file remains in ~/Music/iTunes/ on my Mac.

On 2007-01-23, at 10:41, scruffy wrote:

> isn't he talking about the iTunes Library database file? while i have
> my iTunes library on a 500GB external firewire drive the database
> file remains in ~/Music/iTunes/ on my Mac.

Mine's not, though I am on a Mac.

In my ~/Music folder I've created an alias to an iTunes folder which
resides elsewhere.

My two catalogs take almost 100 MB, and it was seriously chewing up
my FileVault space rewriting them for every song played.

[Let's wrap up the "where do you keep iTunes" branch. -Andrew ]

On 22-Jan-2007, at 14:41, scruffy wrote:
> On Jan 22, 2007, at 2:33 AM, Google Kreme wrote:
>> On 19-Jan-2007, at 10:45, Edward Reid wrote:
>>> Also make sure that your iTunes catalog is on duplicated media. I
>>> don't know where it's kept on the Mac. I'm running iTunes on
>>> Windows, and
>>> it is impossible to persuade iTunes to keep it anywhere but on
>>> the C:
>>> drive.
>>
>> That's simply untrue. It's exactly the same as on the Mac
>> (Preferences -> Advanced)
>>
>> (It's set to I:\iTunes\ on my PC)
>
> isn't he talking about the iTunes Library database file? while i have
> my iTunes library on a 500GB external firewire drive the database
> file remains in ~/Music/iTunes/ on my Mac.

Perhaps. The iTunes library is stored inside the My Documents
folder. By default Windws puts this on the C: drive. It is pretty
trivial to change, however (yes, mine is I:\My Documents\)


--
Dinosaurs are attacking! Throw a barrel!

At 23:33 01/21/07 -0800, Google Kreme wrote:
>On 19-Jan-2007, at 10:45, Edward Reid wrote:
>>Also make sure that your iTunes catalog is on duplicated media. I don't
>>know where it's kept on the Mac. I'm running iTunes on Windows, and it is
>>impossible to persuade iTunes to keep it anywhere but on the C: drive.
>
>That's simply untrue. It's exactly the same as on the Mac (Preferences ->
>Advanced)
>
>(It's set to I:\iTunes\ on my PC)

No, that's the iTunes music folder -- where iTunes puts music files by default.

The catalog is the database that keeps track of what music files you have,
what they contain, etc. There's no preference for its location. I have the
iTunes music folder set to

      E:\music\iTunes

but the iTunes catalog remains resolutely at

      C:\Documents and Settings\Edward Reid\My Documents\My
Music\iTunes\iTunes Library.itl

with a neighboring XML version.

Since you are familiar with both Mac and PC versions, I suppose this
verifies that it's not configurable on the Mac either.

Edward
--
Art works by Melynda Reid: http://paleo.org

Dan Frakes wrote:
> Another follow-up. According to Shirt-Pocket, the issue is even more
> rare than originally suspected (Nik and the company were finally able
> to exchange emails and nail it down):
>
> The file in question must be marked as immutable. The file must
> remain unchanged. A custom extended attribute must be changed, but
> nothing else.
>
> In this particular scenario, the custom extended attribute change
> won't be reflected on the clone. According to Shirt-Pocket, this
> looks to be a bug in OS X which they'll be reporting to Apple -- and
> for which they'll be finding a workaround in SuperDuper in the
> meantime.

For simple cases of "just plain files" most things work as expected.
Once you get into forks, EAs, etc... there is a lot of missing parts to
almost every tool. Check out <iwiring.net>, go to Papers, then read the
one on doing Copies.

At 12:45 01/24/07 -0800, Google Kreme wrote:
>Perhaps. The iTunes library is stored inside the My Documents
>folder. By default Windws puts this on the C: drive. It is pretty
>trivial to change, however (yes, mine is I:\My Documents\)

Which moves the entire My Documents, when for many the point of the
external disk was to separate certain things (music, images) from the rest.
The fact remains that you cannot control the location of the catalog
independently, nor keep it with the iTunes music folder, and so must be
careful to back it up along with the music files.

Edward
--
Art works by Melynda Reid: http://paleo.org

On Fri, Jan 19, 2007 at 5:40 pm -0800, Dan Frakes wrote:

>(Just as impressive, IMO, is that Shirt Pocket regularly responds to email
>questions like this within a matter of hours [or even minutes] -- and based
>on feedback I've received from Macworld readers, it's not just because I'm a
>magazine editor ;-) This is apparently par for the course for the company.)

As an ordinary mortal, I can confirm this. :-)


[OK, I think we've strayed far enough... :-) Great as ShirtPocket is, I'm going to shut this thread down. -Adam]


TimH