Sparseimages Replacing FileVault

On Jul 6, 2006, at 2:35 PM, Jonathan Hogg wrote:

> On 4 Jul 2006, at 17:54, Nik wrote:
>
>> Is a sparseimage truly that reliable?
>
> I expect it's going to be slightly less reliable than FileVault. My
> reasoning being that FileVault *is* an encrypted sparse image, but
> one that is system managed. That means it gets correctly unmounted
> when you logout/restart/shutdown.
...
> It gets Spotlight indexed correctly, all applications work with it
> seamlessly, and most importantly I don't expend a single moment
> worrying about it. I mean, what's not to like about FileVault?

As I noted in my previous posting, I do occasionally have problems
with sparseimages getting corrupted when they don't get properly
closed, when the machine locks up or shuts down unexpectedly. When
logging out or shutting down normally, they get closed by the system,
as any open file does. But when the system is abnormally stopped, a
writable sparseimage of any flavor is likely to be corrupted. It's
because I've had this problem with FileVault that I've switched to
using a separate encrypted volume, since this allows me to log in to
do the necessary recovery when there is corruption. When FileVault
gets corrupted it is a nightmare to recover from. There aren't
adequate tools for replacing the sparseimage used by FileVault when
it gets corrupted. While my solution isn't perfect, it has been
working reasonably well for me, and it was remarkably easy copying
the encrypted sparseimage from my PowerBook to my new MacMini two
months ago. There was a little bit of trouble I mentioned in my
previous post about getting Address Book to recognize the data on
that volume, but otherwise there was no problem at all. The one
point that you brought up that I haven't protected against is that I
have left my caches on my unprotected volume, which clearly isn't the
wisest thing to do for high security, and I wish I had a good
solution for it, but I don't trust FileVault, because it's given me
considerably more trouble than other encrypted sparseimages.

Geoff
.

There's a timely ArsTechnica article discussing laptop security and
encryption. The author recommends a program called "Knox" which can
be integrated into PathFinder.

<https://arstechnica.com/news.ars/post/20060706-7205.html>

Anyone familiar with this solution or other encryption tools? I've
used GPG fairly successfully, but it requires a file to be fully
decrypted to disk, which leads to insecurity since the file can be
"recovered" from the disk if you don't securely delete it.

--Nik

On 2006-07-06, at 18:12, Tom Robinson wrote:
> If you do move them, remember there's caches of everything so that
> Spotlight can find each event/address in an individual file. There's
> probably another copy around for syncing too.
>
> ~/Library/Caches/com.apple.AddressBook/MetaData
> ~/Library/Application Support/Sync Services
> and an iCal one I can't find at the moment.

Thanks for this tip. I had forgotten about this when I moved a few
things to a secure disk image.

And the iCal Spotlight cache appears to be at:
~/Library/Caches/Metadata/iCal

There's also ~/Library/Caches/com.apple.iCal, but I'm not sure what
that does, though.

On 2006-07-08, at 10:35, Nik wrote:
> There's a timely ArsTechnica article discussing laptop security and
> encryption. The author recommends a program called "Knox" which can
> be integrated into PathFinder.
>
> <https://arstechnica.com/news.ars/post/20060706-7205.html>
>
> Anyone familiar with this solution or other encryption tools I've
> used GPG fairly successfully, but it requires a file to be fully
> decrypted to disk, which leads to insecurity since the file can be
> "recovered" from the disk if you don't securely delete it.

Knox is the secure disk image I use. As far as I understand, it's
just a standard Sparseimage file, with some management features
provided by the Knox program. A couple of nice things:

- Sparseimages can be set to automatically reopen on startup.
- You can enable Spotlight searching easily via the Knox application
(without resulting to CLI)
- It provides some useful image backup features

I haven't tried the "integration" into Pathfinder. It works well
enough for me with the standard finder, though.

Tomoharu

On 7 Jul 2006, at 15:40, Nik wrote:

> It is this ease of use which really makes me WANT to use Filevault,
> but the additional overhead of encrypting/decrypting, and the hassles
> with backups and file recovery (there is NONE of the latter if my
> drive gets corrupted somehow), scares me away from it. I suppose I'm
> more paranoid about random system failures than I am security
> breaches.

I wonder how much the fear of FileVault is due to a very small number
of problems. I guess data loss is an emotive subject: kind of like
how the chances of a child being kidnapped are almost insignificantly
low, but parents rate it as their number one fear.

I note that people here have talked of suffering data corruption, but
I do wonder how exactly? I mean, like I say, I'm not particularly
careful with my Mac. It travels with me everywhere thrown in a back
pack, I run a fair amount of beta software, I do development on it
and I don't think twice about hard rebooting it in the event of a
freeze. I just haven't seen any these corruptions that people talk
about.

The drive is journalled and the FileVault image is journalled. There
just shouldn't be any reason for it to become corrupted unless you
have something rudely writing across the disk surface. Even if there
was a corruption, Disk Utility is just as happy to repair an
encrypted sparseimage as it is a regular disk. And really, if you
backup every night would you have need to worry about the
astronomically low chance of a corruption so bad that you lost
today's work?

I guess I can see that if you have a desktop machine and work daily
with high bandwidth data like video, then FileVault is overkill and a
pain, but for a laptop I think it's a great peace of mind. I've now
had two stolen (one from a locked and alarmed city-centre office).

> I think the optimal Filevault configuration could be set to ignore
> specific folders in a user's home directory. For example, the iTunes
> library could be easily eliminated, thus freeing up tens of gigabytes
> on many users' filevaults! (I have a 35 GB music library!)

Now here is where I skimmed over the full detail of my setup for
dramatic effect ;-) In fact I have my iTunes library in /Users/shared
as you say. That removes 40GB from my home directory and makes it a
manageable backup. My home directory is 15GB at the moment, which is
the largest it's ever been. I now backup to a disk on another machine
rather than to DVD.

> But if you're doing all that, then probably an encrypted directory
> structure is the way to go...

I must say that I was quite excited about the news that Apple has
been chatting to the ZFS developers. One of the most interesting
features of Sun's ZFS is that filesystems are cheap: new directory
kind of cost. That means one can have arbitrary parts of your
directory structure have different properties. Thus each user's home
directory can be a different filesystem and different directories
within your home directory can be the same.

So you could mark your home directory as encrypted, but mark the
Music folder/filesystem within it as not. You could mark your "Old
Stuff" folder/filesystem as transparently compressed and give another
user's home directory a disk usage limit. And all of this is done at
the filesystem level with full database-style integrity. (Other
interesting features include cheap snapshotting for backup and easy
on-line extension of the storage pool.)

ZFS is one of my "here's hoping" for Leopard :-)

Jonathan

You need to use the commandline utility mdutil to turn on indexing for such volumes. There is an article at this URL: <http://www.macintouch.com/ tigerreview/spotlight.html> explaining the procedure to follow.

It worked! Spotlight is chugging away at my encrypted volume, even as I type this message. I can't wait to try the other bit of advice you gave about storing Address Book and iCal data on the encrypted volume.

One question, though, is whether or not the index data that Spotlight creates is secure when I unmount the encrypted volume. Is the data stored on the volume itself, or in some invisible place where a hacker could access it? How difficult would it be to get at the data if my Mac was stolen?

Incidently, in the year and a half that I have been using this encrypted volume, I had one crash (while using iMovie) which toasted the volume. Fortunately, my data loss was not that extensive as I had backed up the previous week, and was not using Apple's FileVault!

With all the discussion about potential data loss and so forth, I
decided to do a bit of testing to see how likely this sort of data
loss is.

I created both a sparse image and a regular fixed sized image, both
with encryption. I mounted both new images, saved a text file to
each, then hard restarted my TiBook. Upon reboot, not only did the
two new images work, each text file was sitting happily on the image,
perfectly intact. The fact that the image had not been properly
unmounted did not lead to data loss.

But this is an easy (if likely) scenario. So I tried something more
complex. I created a 100 MB sparse image and saved a text file to it,
as above. I then started a 90 MB file copy to the image. Before the
copy could complete, I restarted my Mac.

After the reboot, the image fails to mount, claiming "no mountable
filesystems." Corruption at last!

I repeated this last test with an encrypted read/write disk image
(non-resizable), and this time the image was still usable after the
restart. The same is true of a sparseimage that had already been
"stretched" to a sufficient size to contain the large file copy. This
makes sense, since the filesystem would be in the process of resizing
during the file copy, so an unexpected halt could render the
filesystem useless. (Same as if you restart a computer while
formatting a hard drive or other disk.)

So, despite the changes since 10.3.1 which were supposed to increase
the reliability of sparse images (and has in fact done so, as the
images only appear to fail under very specific circumstances), they
remain somewhat unreliable, and probably inadequate for very
important data. Given that the sort of data a person would want to
protect with encryption is usually extremely important, I'd be quite
leery of handing it over to a sparse disk image.

On the other hand, if you're securing a fairly limited number of
files of moderate size, creating a smallish fixed-size image with
encryption may be just the ticket. The image format appears to be
more robust, and you get the same ease of use as you would with a
sparse image. The only downside is file size, but for a list of
passwords and a few spreadsheets of assets, a 10 MB image would be
fine (and easy to back up, to boot!).

--Nik

Apparently it's stored on the encrypted volume.  I looked in my Library/Caches directory for a spotlight directory and found none, and I also looked on my encrypted volume, and found a hidden directory .Spotlight-V100 containing a variety of files.  I didn't examine their contents, but I'm fairly confident that they contain the index.  Here's a listing:

On 2006-07-06, at 05:09, Geoffrey Odhner wrote:

> I have had problems occasionally with a sparseimage getting something
> wrong with it such that it can't be ejected, and the system always
> says it's in use, even when all the processes that use it have been
> stopped. Then I have to copy its contents to a new sparseimage
> volume and replace the old one with the new one.

Sounds like the long way around!

Have you tried:

1. Using lsof at the command line to look for open files?

2. Logging out or restarting which should force a dismount.

On Jul 11, 2006, at 3:09 AM, Tom Robinson wrote:

> On 2006-07-06, at 05:09, Geoffrey Odhner wrote:
>
>> I have had problems occasionally with a sparseimage getting something
>> wrong with it such that it can't be ejected, and the system always
>> says it's in use, even when all the processes that use it have been
>> stopped. Then I have to copy its contents to a new sparseimage
>> volume and replace the old one with the new one.
>
> Sounds like the long way around!
>
> Have you tried:
>
> 1. Using lsof at the command line to look for open files?
>
> 2. Logging out or restarting which should force a dismount.

I have not tried lsof. Restarting does force a dismount, yes, but as
soon as I mount it again, the volume would again refuse to dismount
without logging out or restarting. Once I had the image corrupted in
this way I would restart to dismount it, then if I mount it, and
immediately try to dismount it, the dismount would fail, without
opening any files on the volume. The only method I could find to get
around this problem was to copy the volume to a new encrypted volume,
as I described.

Geoff
.

On Jul 11, 2006, at 12:09 AM, Nik wrote:

> So, despite the changes since 10.3.1 which were supposed to increase
> the reliability of sparse images (and has in fact done so, as the
> images only appear to fail under very specific circumstances), they
> remain somewhat unreliable, and probably inadequate for very
> important data. Given that the sort of data a person would want to
> protect with encryption is usually extremely important, I'd be quite
> leery of handing it over to a sparse disk image.

If the data are important (John is being old fashioned here: "data"
is a plural noun*), one copy is not enough. If the data are very
important, two copies are not enough. Two copies both backed up one
way are not enough.

*sigh--that usage is beginning to look wrong even to me.

   --John

[Please send responses directly to Robert so he can compile them and post back. -Adam]

This seems like a reasonable time to ask for people's direct
experiences:

1. Have you used FileVault?
2. Have you experienced any problems with it? Please detail
3. Do you use encrypted sparse images?
4. Have you experienced any problems with them? Please detail
5. Have you used any third-party encryption tools, and what are your
experiences?

Laptop encryption is a pretty hot issue these days and most people
(myself included) are making decisions based mostly on press reviews
and anecdotal evidence. Considering the size of the TidBITS
distribution list we might be able to get some more statistically
relevant numbers.

-rm

On Jul 11, 2006, at 2:52 PM, Robert Movin wrote:

> [Please send responses directly to Robert so he can compile them
> and post back. -Adam]
>
> This seems like a reasonable time to ask for people's direct
> experiences:
>
> 1. Have you used FileVault?

Yes, but it was when it was first introduced and I quickly stopped
using it once I heard horror stories.

> 2. Have you experienced any problems with it? Please detail

No problems, but I didn't use it long enough to be significant. It
was never really tested.

> 3. Do you use encrypted sparse images?

Yes.

> 4. Have you experienced any problems with them? Please detail

Yes. I have suffered random corruption which forced me to recover
from backup.

I have also run into problems where I will have an encrypted data
file, but it will be written to a cache file or temp space by the
application which opens it, thus negating any security benefits.

> 5. Have you used any third-party encryption tools, and what are your
> experiences?

I have used GPG extensively (along with GPG Tools and the GPG Mail
plug-in). It's a serviceable encryption package, but is not entirely
transparent or easy to use (as FileVault is). Additionally, it forces
you to encrypt and decrypt documents as you go, along with securely
deleting the decrypted copies. This can be arduous and can easily
lead to problems.

I think this solution is better suited for encrypting files for
archival purposes or for transferring files in a secure fashion,
rather than for day-to-day securing of sensitive information.

--Nik

On 2006-07-12, at 05:50, johnbaxterlistsmac.com wrote:

> If the data are important (John is being old fashioned here: "data"
> is a plural noun*), one copy is not enough. If the data are very
> important

 From one pedant to another: it may be a plural, but I'd never use
it in that sense (prob. because it sounds wrong). If I have two lots
of data, I'd call it one big lot of data, hence would always use 'is'
instead of 'are' above.

[OK, but let's not let this spin off into a grammar discussion, eh? -Joe]

On 2006-07-11, at 19:09, Nik wrote:

> But this is an easy (if likely) scenario. So I tried something more
> complex. I created a 100 MB sparse image and saved a text file to it,
> as above. I then started a 90 MB file copy to the image. Before the
> copy could complete, I restarted my Mac.
>
> After the reboot, the image fails to mount, claiming "no mountable
> filesystems." Corruption at last!

Would be interested to know if Disk Util and/or DiskWarrior could fix
this? I would assume not if it can't be mounted...

I use FileVault, but also take very regular backups just in case...

On Jul 12, 2006, at 5:28 PM, Tom Robinson wrote:

> On 2006-07-11, at 19:09, Nik wrote:
>
>> I then started a 90 MB file copy to the image. Before the
>> copy could complete, I restarted my Mac.
>>
>> After the reboot, the image fails to mount, claiming "no mountable
>> filesystems." Corruption at last!
>
> Would be interested to know if Disk Util and/or DiskWarrior could fix
> this? I would assume not if it can't be mounted...
>
> I use FileVault, but also take very regular backups just in case...

No. The image was unrecoverable by Disk Utility (could not mount/
repair/convert the image), and DiskWarrior doesn't treat it as a disk
until it mounts. Even the diskutil tricks to mount an image in read-
only mode for data recovery did not work.

I did find some anecdotal reports of exactly this happening to some
Filevault users. That could be quite problematic...

--Nik

1. Have you used FileVault?

No, because I read TidBITS first!!!

2. Have you experienced any problems with it? Please detail

No, since I never used it.

3. Do you use encrypted sparse images?

Yes, my wife and I have each had one on our iBooks for about a year now. We keep all our sensitive files on the sparseimage, including our Apple Mail folder (which was a bit tricky to set up).

4. Have you experienced any problems with them? Please detail

I have never experienced the "unable to dismount" problem that others in this discussion have faced, and we read and write to it constantly, and dismount (or is it "unmount"?) frequently, usually several times a day. I only experienced one problem, and that was when the image was mounted and I tried to force quit iMovie during a lengthy video process. This caused the Mac OS to fully crash, and when I rebooted, the sparseimage was corrupted beyond repair. I had to restore from the last backup, which unfortunately was not as recent as I would have liked.

5. Have you used any third-party encryption tools, and what are your experiences?

I used to use PGP to encrypt individual files, then switched to the built-in encryption in Adobe Acrobat for files that we no longer needed to edit and could therefore convert them into PDFs. I was delighted to read a TidBITS article about a year and a half ago that alluded to the possibility of an encrypted sparseimage as an alternative to FileVault (can't find the original articel). Now I no longer need to think about encryption, it just happens.

At 10:50 on 11-07-2006, johnbaxterlistsmac.com wrote:
> If the data are important (John is being old fashioned here: "data"
> is a plural noun*)

You'll be pleased to hear that this is standard usage in the UK, in
both academic writing (non-computer related) and the normal press.
Datum is used (correctly, I believe) for the singular. So it doesn't
sound old fashioned to me. :-)

Cheers,
Jolin

_________________
=> Jolin Warren, Edinburgh, Scotland

Greetings,

We have a sparse image which after leaving mounted and experiencing a crash gives the error that it is a corrupted image and can't be mounted.

Is there any hope to recover this image?

Does anyone have any experience recovering in similar circumstances. Unfortunately we don't have a backup so this is problematic.

We read here: http://toxicsoftware.com/blog/filevault_corruption/ that he was able to copy the files from his sparse image to another. How do you do this? Our image is password protected (this is the whole point of doing it this way). So anyway we can copy the files from this corrupted image is ok.

Any suggestions how to do this? That is, save the files?

Thanks Venu vee308hotmail.com

1. Incremental backups across the network were problematic.

You may want to look at LBackup <http://www.lucidsystems.org/lbackup/>