Flash vs. iPod

At 5:19 PM -0800 2006/03/17, Jonathan Ploudre wrote:

Could you go into specifics about how you backup those files when you plug in your flash. Are you drag/dropping or do you have one of your fancy shell based bits. If it's complex or nonintuitive, I bet others would be interested to hear about it on Tidbits-Talk.

Jonathan,

I have mentioned it on TBT a few times (not recently). I use Synchronize Plus from Qdea, which can run automatically when both source and destination become available (when the iPod mounts on the desktop), and can run a script when it completes. Mine launches a .command script, which starts Eudora and iChat, and loads my ssh key(s).

pepperpepperbook:~$ cat bin/mylogin.command #!/bin/sh open /Applications/Eudora.app open /Applications/iChat.app keychain --host pepperbook --timeout 540 ~/.ssh/id_dsa.pepper.200510

I probably won't buy from Qdea again. I don't like their licensing or upgrade policies (or the chances to the license terms over time), although Synchronize works very well. I am not entitled to upgrades without paying again, so I will need a new scheme when I get an Intel system to keep.

I tried Synk, and trashed a bunch of Eudora TOCs. This may have been because Synk acquired a read lock on the TOCs to copy them. If Eudora can't get a write lock (prevented by an existing read lock) to the TOCs when it tries to open them, it apparently silently creates new ones -- not good. I can't take the risk of running Synk and Eudora at the same time, and trashing files, so I haven't gone back to test further.

There are lots of other synchronize programs (I downloaded several for an aborted attempt to migrate off Synchronize, which will now wait until I get an Intel Mac and want something UB). Many of them can automatically initiate a sync, and launch an application when done. Synk suggests a System Pref (kernel module?) named Do Something When, because it doesn't have this capability built in. DSW should be able to launch any sync tool, but I don't think it would do the post-action launches I want.

In 10.4.0, rsync gained the ability to handle resource forks (where Eudora stores TOCs), but it was broken. I don't know if it's been fixed as of 10.4.5, but "rsync -aE --delete source/ dest/" should do the trick (once it handles resource forks properly). Note that you need the trailing slashes for rsync to recurse through the directories.

I do a few related things to manage apps when I get to or depart from a Mac. I have cron jobs to quit iChat at 2am at home and 6pm at work for me, with "killall -HUP iChat > /dev/null 2>&1".

Briefly, I tried locking the Apple Keychain and flushing the Gentoo keychain (for ssh) via crontab, script, and the Keychain menu (which can also lock the screen). I stopped worrying about the Apple Keychain, and flush the Gentoo keychain from cron.

I have aliases to mount and unmount the iPod as well. If you attach a removable disk while Tiger is sleeping, it doesn't mount for security, although I think this is misguided. Note that the "dmc" alias varies from machine to machine, based on how many other drives are present.

pepperpepperbook:~$ alias dmc alias dmc='diskutil mount disk1s3' pepperpepperbook:~$ alias dec alias dec='diskutil eject /Volumes/cPod'

Chris -- Chris Pepper: <http://www.reppep.com/~pepper/> Rockefeller University: <http://www.rockefeller.edu/>

--On March 16, 2006 10:48:28 AM -0800 Chris Pepper <pepperreppep.com>
wrote:

> A 2gb thumb drive is a neat thing to have (my first computer,
> an Apple ][+, had 48k because I knew I wouldn't need 64k), but it's
> not going to help with Eudora. I am now switched over to IMAP, but
> not yet ready to give up my Eudora-on-iPod setup, where the same
> windows are open at home and work (and I'm carrying the iPod to
> listen to anyway...).

The concept of carrying your apps (and all the settings) around with you on
a thumb drive is becoming pretty popular in the Windows world. Several apps
are available in portable version -- suitable for installation on a flash
drive. Since you're on IMAP now you could throw Thunderbird for Windows and
PuTTY (SSH client for Windows) on your thumb drive and be able to get your
e-mail via SSH tunnel, even on a windows machine.

<http://portableapps.com/>

PuTTY isn't on PortableApps, but works fine from flash drive for me.

<http://www.chiark.greenend.org.uk/~sgtatham/putty/>

In the Mac world, I would assume anything drag & drop installable would
work from a thumb drive. Not sure about apps using the .Pkg installer.

--On March 21, 2006 8:17:51 AM -0800 Chris Pepper <pepperreppep.com> wrote:
> I have mentioned it on TBT a few times (not recently). I use
> Synchronize Plus from Qdea, which can run automatically when both
> source and destination become available (when the iPod mounts on the
> desktop), and can run a script when it completes. Mine launches a
> .command script, which starts Eudora and iChat, and loads my ssh
> key(s).

Hmmm, would it be possible to emulate this behavior by setting a Folder
Action on the /Volumes folder so something occurs when a new volume is
mounted? I'm assuming Folder Actions only work on the one directory, and
not when something happens in a sub-directory, otherwise you'd be
continually launching a script.

Kevin

At 11:20 AM -0600 2006/03/21, Kevin van Haaren wrote:
>--On March 21, 2006 8:17:51 AM -0800 Chris Pepper <pepperreppep.com> wrote:
>>I have mentioned it on TBT a few times (not recently). I use
>>Synchronize Plus from Qdea, which can run automatically when both
>>source and destination become available (when the iPod mounts on the
>>desktop), and can run a script when it completes. Mine launches a
>>.command script, which starts Eudora and iChat, and loads my ssh
>>key(s).
>
>Hmmm, would it be possible to emulate this behavior by setting a
>Folder Action on the /Volumes folder so something occurs when a new
>volume is mounted? I'm assuming Folder Actions only work on the one
>directory, and not when something happens in a sub-directory,
>otherwise you'd be continually launching a script.

        Probably, since every auto-mounted partition shows up there.


                                                Chris
--
Chris Pepper: <http://www.reppep.com/~pepper/>
Rockefeller University: <http://www.rockefeller.edu/>

At 8:17 AM -0800 3/21/06, Chris Pepper wrote:
>I probably won't buy from Qdea again. I don't like their
>licensing or upgrade policies (or the chances to the license terms
>over time), although Synchronize works very well. I am not entitled
>to upgrades without paying again, so I will need a new scheme when I
>get an Intel system to keep.

I was actually burned by this policy when I had a drive failure and
lost the program altogether. Though I politely pled my case and asked
for a copy of the version I had a license for, they tersely refused
quoting their licensing agreement. There were no copies of this on
the 'Net so it was pay up or hit the road. I replied that I thought I
should have access to the version I purchased. No dice. It would have
been a small thing to do but here I am now telling the tale to
thousands of people.

Fortunately I did find a copy and used my legit serial number.
Interestingly the updates to this program have been few and not very
significant. Makes me wonder if this is a case of churning product to
force customers to ante up. V2.0 from 11/04 still works great in
Tiger though.

John Mather
D. C. Smith Greenhouse
UW - Madison

At 1:51 PM -0800 2006/03/21, Kevin van Haaren wrote:
>--On March 16, 2006 10:48:28 AM -0800 Chris Pepper <pepperreppep.com>
>wrote:
>
>> A 2gb thumb drive is a neat thing to have (my first computer,
>>an Apple ][+, had 48k because I knew I wouldn't need 64k), but it's
>>not going to help with Eudora. I am now switched over to IMAP, but
>>not yet ready to give up my Eudora-on-iPod setup, where the same
>>windows are open at home and work (and I'm carrying the iPod to
>>listen to anyway...).
>
>The concept of carrying your apps (and all the settings) around with you on
>a thumb drive is becoming pretty popular in the Windows world. Several apps
>are available in portable version -- suitable for installation on a flash
>drive. Since you're on IMAP now you could throw Thunderbird for Windows and
>PuTTY (SSH client for Windows) on your thumb drive and be able to get your
>e-mail via SSH tunnel, even on a windows machine.
>
><http://portableapps.com/>
>
>PuTTY isn't on PortableApps, but works fine from flash drive for me.
>
><http://www.chiark.greenend.org.uk/~sgtatham/putty/>
>
>In the Mac world, I would assume anything drag & drop installable would
>work from a thumb drive. Not sure about apps using the .Pkg installer.

Kevin,

        You're assuming I'm using Windows. Gong! :-)

        Yes, PuTTY ran from a floppy before the U3 flash standard was
a gleam in Sandisk's & M-Systems' eyes. Any Mac should have ssh, so
shell access isn't a big deal. If I wanted, I could easily put a
ssh.command on the flash drive like this:

>#!/bin/sh
>ssh -i /Volumes/cFlash/.ssh/id_dsa peppersalt.rockefeller.edu
>ssh -i /Volumes/cFlash/.ssh/id_dsa peppersalt.rockefeller.edu
>ssh -i /Volumes/cFlash/.ssh/id_dsa peppersalt.rockefeller.edu
>ssh -i /Volumes/cFlash/.ssh/id_dsa peppersalt.rockefeller.edu

        That would pop 4 Terminal windows logged into salt (after 4
password dialogs), with nothing beyond the ssh commands logged
locally. Alternatively, I could use something like this as X.command:

>#!/bin/sh
>open /Applications/Utilities/X11.app/
>export DISPLAY=:0.0
>ssh -i /Volumes/cFlash/.ssh/id_dsa pepperwww.reppep.com -aTXY xterm

        Then I'd run x4 on www, containing the following (so I'd have
4 xterms, all running through ssh):

>#!/bin/sh
>xterm&
>xterm&
>xterm&


        Any of these are liable to keyboard sniffing, unless I used a
portable keyfile without password (very bad idea), but I could make
up a special password just for this private keyfile, if I was going
to use it on other people's machines.

        And of course I can use SquirrelMail (via HTTPS), which is
already available, but not as capable as Eudora.


--
Chris Pepper: <http://www.reppep.com/~pepper/>
Rockefeller University: <http://www.rockefeller.edu/>

At 10:48 AM -0800 3/16/06, Chris Pepper wrote:
> As I've stated here in the past, I run Eudora, with my Eudora
>Folder on an iPod (previously it was on Zip disks, then on an
>external 2.5" drive). This is convenient, because Eudora is exactly
>the same at home or at work, and the 60gb iPod (photo, not video) is
>much smaller and lighter to commute with than a laptop.
>Unfortunately, it's still a bit awkward to carry around the house.
>
> I got a 2gb thumb drive (once I realized I didn't really need
>an SD card) for $75 (Cruzer Mini), intending to mount an encrypted
>filesystem in case I lost it. I back up the important data folder
>every time I plug it into one of my Macs, so at least 10x/week. As a
>result, I'm protected against data loss, but having someone pick up a
>thumb drive with all my email on it would be bad. When the iPod
>shuffle came out, I got one for the same purpose, but eventually sent
>it back.
>
> Unfortunately, the experiment was a bust. I have a couple of
>large files totalling 232mb. Copying them to my iPod takes 24-26
>seconds. Copying them to the thumb drive (formatted as Journaled
>HFS+) takes 56-64 seconds. Copying them to an encrypted .dmg
>(AES-128) on the same thumb drive takes 95 to 120 seconds. Reading
>from the thumb drive is faster, but Eudora writes to a mailbox file
>every time a new message comes in, so the Cruzer Mini just isn't
>feasible.
>
> On the other hand, read performance from unencrypted flash
>was very similar to the iPod. Reading those two files from the iPod
>took 24 seconds the first time, then 3 seconds for the second and
>third (cached). Reading from the flash drive (unencrypted) took 21
>seconds initially, then 3 seconds for the second and third times,
>just like the iPod. I didn't test reading from an encrypted .dmg on
>flash.
>
> A 2gb thumb drive is a neat thing to have (my first computer,
>an Apple ][+, had 48k because I knew I wouldn't need 64k), but it's
>not going to help with Eudora. I am now switched over to IMAP, but
>not yet ready to give up my Eudora-on-iPod setup, where the same
>windows are open at home and work (and I'm carrying the iPod to
>listen to anyway...).

A few notes about this. Most thumb drives are formatted FAT. Although the Mac supports FAT, it is slow. If you're only going to use it on a Mac, reformat it as HFS+ (with journalling). It will mount faster, file access will be faster, and journalling will automatically repair any directory corruption.

If you're going to leave it FAT, at least make sure it's FAT32.

Writing to most flash drives is slow. This is because their memory is arranged in large physical blocks, often 128K, and the controller has to write to the whole block. Whereas FAT32 and HFS+ typically use 4K clusters (called allocation blocks on HFS).

Unfortunately, the host computer OS doesn't know this. So when writing a file to a thumb drive, the OS will write out several 4K clusters to the thumb. The thumb's controller will read in an entire physical block, change the parts you wrote to, and write the entire physical block back out. If you're writing to a bunch of mail files, you're writing the same 128K block over and over, changing a different pieces of it each time. Highly inefficient.

And writing to flash drives in general is slow, even if you write in nice neat 128K blocks, because of the nature of flash.

Reading is better, because the thumb controller reads the physical block once, and keeps it cached, passing out pieces of it as requested.

In the iPod nano we have an intelligent cache which tries to combine multiple writes to the same physical block into a single write. Of course the nano has 32M of RAM, and we can use a lot of it for cache. The thumb drive doesn't.

Whether it's FAT or HFS, for maximum performance you'd like the cluster size to be an even multiple of the physical block size, and the cluster boundaries to be aligned with the physical block boundaries. The iPod updater takes care of this when you restore an iPod nano. You can do this manually on a thumb drive by partitioning and formatting it from the command line, but it's not pretty.

--
David

At 8:07 PM -0800 2006/03/21, David Shayer wrote:
>At 10:48 AM -0800 3/16/06, Chris Pepper wrote:
>> As I've stated here in the past, I run Eudora, with my Eudora
>>Folder on an iPod (previously it was on Zip disks, then on an
>>external 2.5" drive). This is convenient, because Eudora is exactly
>>the same at home or at work, and the 60gb iPod (photo, not video) is
>>much smaller and lighter to commute with than a laptop.
>>Unfortunately, it's still a bit awkward to carry around the house.
>>
>> I got a 2gb thumb drive (once I realized I didn't really need
>>an SD card) for $75 (Cruzer Mini), intending to mount an encrypted
>>filesystem in case I lost it. I back up the important data folder
>>every time I plug it into one of my Macs, so at least 10x/week. As a
>>result, I'm protected against data loss, but having someone pick up a
>>thumb drive with all my email on it would be bad. When the iPod
>>shuffle came out, I got one for the same purpose, but eventually sent
>>it back.
>>
>> Unfortunately, the experiment was a bust. I have a couple of
>>large files totalling 232mb. Copying them to my iPod takes 24-26
>>seconds. Copying them to the thumb drive (formatted as Journaled
>>HFS+) takes 56-64 seconds. Copying them to an encrypted .dmg
>>(AES-128) on the same thumb drive takes 95 to 120 seconds. Reading
>>from the thumb drive is faster, but Eudora writes to a mailbox file
>>every time a new message comes in, so the Cruzer Mini just isn't
>>feasible.
>>
>> On the other hand, read performance from unencrypted flash
>>was very similar to the iPod. Reading those two files from the iPod
>>took 24 seconds the first time, then 3 seconds for the second and
>>third (cached). Reading from the flash drive (unencrypted) took 21
>>seconds initially, then 3 seconds for the second and third times,
>>just like the iPod. I didn't test reading from an encrypted .dmg on
>>flash.
>>
>> A 2gb thumb drive is a neat thing to have (my first computer,
>>an Apple ][+, had 48k because I knew I wouldn't need 64k), but it's
>>not going to help with Eudora. I am now switched over to IMAP, but
>>not yet ready to give up my Eudora-on-iPod setup, where the same
>>windows are open at home and work (and I'm carrying the iPod to
>>listen to anyway...).
>
>A few notes about this. Most thumb drives are formatted FAT.
>Although the Mac supports FAT, it is slow. If you're only going to
>use it on a Mac, reformat it as HFS+ (with journalling). It will
>mount faster, file access will be faster, and journalling will
>automatically repair any directory corruption.

David,

        Thanks. I did erase it and use Journaled HFS+. I had an
original shuffle, and learned that Apple's FAT implementation slows
the shuffle down. I wanted to use it as an MP3 player, and didn't
realize until after I'd returned it that the shuffle would apparently
play MP3s off an HFS+ file system (even if unsupported).

        I would have preferred a shuffle for this purchase, if only
for the earbuds (although I'd probably listen to it sometimes too),
but I don't expect to see a 2 GB shuffle, and my Eudora Folder is
currently 875 MB.

>Writing to most flash drives is slow. This is because their memory
>is arranged in large physical blocks, often 128K, and the controller
>has to write to the whole block. Whereas FAT32 and HFS+ typically
>use 4K clusters (called allocation blocks on HFS).
>
>Unfortunately, the host computer OS doesn't know this. So when
>writing a file to a thumb drive, the OS will write out several 4K
>clusters to the thumb. The thumb's controller will read in an entire
>physical block, change the parts you wrote to, and write the entire
>physical block back out. If you're writing to a bunch of mail files,
>you're writing the same 128K block over and over, changing a
>different pieces of it each time. Highly inefficient.

>In the iPod nano we have an intelligent cache which tries to combine
>multiple writes to the same physical block into a single write. Of
>course the nano has 32M of RAM, and we can use a lot of it for
>cache. The thumb drive doesn't.

        This is cool, but not relevant for me, since I have 43 GB of
music on my 60 GB iPod photo -- all of which I want with me. A nano
still needs a cable with each computer, and isn't so much smaller
than a full-size iPod that I'd want both, or to give up most of my
music.

>Whether it's FAT or HFS, for maximum performance you'd like the
>cluster size to be an even multiple of the physical block size, and
>the cluster boundaries to be aligned with the physical block
>boundaries. The iPod updater takes care of this when you restore an
>iPod nano. You can do this manually on a thumb drive by partitioning
>and formatting it from the command line, but it's not pretty.

        Cool -- the relevant incantation would be welcome, if you
know it off-hand.


        Separately, here's an update. Alex Hoffman bought the flash
drive, to replace a lost iPod shuffle (which is larger than the
Cruzer Mini, thus validating my fear of losing it).

        I see M-Systems and Verbatim are addressing my concerns as
well. Verbatim sells the Store n Go Pro USB flash drive, based on
M-Systems technology. Verbatim claims the drive provides 23mbyte/s
reads & 14mbyte/s writes, compared to a little worse than 10mbyte/s
(both ways) for the 60gb iPod photo. The flash drive includes strong
AES256 encryption in silicon. Hopefully the speeds are with
encryption (the whole point of the drive), although I wouldn't put
money on it. Although <http://www.d-silence.com/feature.php?id=1007>
claims Mac compatibility, I wonder if the encrypted area is really
accessible from a Mac. The V-Safe encryption engine is supposed to
securely delete the files after a configurable number of password
failures, which is very slick (I have good backups of this data).

        I've asked Verbatim about Mac access to the encrypted portion
and encrypted read/write speed. If they say Macs can access the
encrypted portion at full iPod speeds, it might be time to spend some
more money.

        I love technology. The 2gb drive was announced at $600 list,
and Amazon now offers the drive for $186, $238, and $258. No, I don't
know why Amazon offers 3 different pages & prices, and 2 different
photos, but I doubt Amazon does either.

--
Chris Pepper: <http://www.reppep.com/~pepper/>
Rockefeller University: <http://www.rockefeller.edu/>

On Mar 21, 2006, at 2:51 PM, Kevin van Haaren wrote:

> Hmmm, would it be possible to emulate this behavior by setting a
> Folder
> Action on the /Volumes folder so something occurs when a new volume is
> mounted?

Absolutely. I've got a script that writes out a mess of OmniOutliner
documents to text files on my iPod so that I can view them as notes.
It's triggered by a folder action on the Volumes folder.

The trick to doing this is to set up a folder action on the /Volumes
folder, you need to get that folder to appear in the open/save
dialog. The key there is command+shift+G from the choose folder
dialog, and then just type in /Volumes. From there, the syntax is
pretty straightforward:

on adding folder items to volumesFolder after receiving theVolume
        if (theVolume as text) contains "My Disk:" then -- Change "MyDisk:"
to the name of your actual drive.
                -- Whatever you want to do
        end if
end adding folder items

Obviously, if you don't have the if statement in there, this will run
every time you put in any disk, which I can't recommend.

--Nik

--On March 23, 2006 11:30:55 AM -0800 Nik <nikinik.net> wrote:

> The trick to doing this is to set up a folder action on the /Volumes
> folder, you need to get that folder to appear in the open/save
> dialog. The key there is command+shift+G from the choose folder
> dialog, and then just type in /Volumes. From there, the syntax is
> pretty straightforward:

I cheated on this part when I tried it on my own. I went to Finder, Go To
Folder and typed in /Volumes
Then I command+option dragged the Volumes icon in top of the window that
opened to create a shortcut in my home folder. I can never remember the
shortcut keys that work in the various dialogs. I can't remember the
command+option+drag for creating shortcuts either, I have to try various
key combinations until the icon changes from the green + to the shortcut
arrow.

Kevin