[ANN] The Entourage Help Blog: Entourage and Time Machine

On 01/04/2008 09:50 AM, "Dave Scocca" <davescocca.org> wrote:

>> Storing information in thousands of files is usually not a good idea.
>> Opening thousands of files can be very slow, and it can take up lots
>> of system resources. I even suspect that Apple Mail's change from
>> standard Unix format to individual files was done specifically for the
>> use of Spotlight and possibly Time Machine.
>
> Yes, but the addition of Spotlight means that Apple Mail doesn't have to
> "[open] thousands of files"--it stores information about the file contents
> when
> each file is created/modified, and--when searching--uses the Spotlight
> metadata
> files rather than opening the individual mail message files.

Right, but that doesn't mean you don't deal with thousands of files. It
means you deal with thousands of spotlight files. For spotlight to work,
there has to be a 1:1 match with the files it's cataloging. Entourage's
integration works the same way as Address Book's, or really, any database.
Spotlight replicates everything in there as single files.

--
John C. Welch

On 01/04/2008 09:50 AM, "Chris Devers" <cdeverspobox.com> wrote:

>> Storing information in thousands of files is usually not a good idea.
>
> The people migrating from mbox to maildir or mix, or from Panther's
> version of Mail to Tiger's version of Mail, might disagree here.

While computer speeds can compensate somewhat, the truth is, as you scale up
the number of files you have to deal with, you start losing efficiency over
a database. A database can handle ginormous amounts of data far more
efficiently than a standard filesystem can, and you can slice the data in
more ways far easier.

The backup problem is one that has always, and still plagues databases, and
always will, but if you have to relate a lot of similar kinds of data to
each other in random ways, a database is going to be your best option.

--
John C. Welch

On 01/04/2008 09:50 AM, "Dan Frakes" <frakesmac.com> wrote:

>>> Does anyone know if Entourage 2008 changes this behavior, or if it still
>>> uses the one big identity file?
>>
>> They know, but they can't say.
>> (At least not until 15 Jan 2008.)
>
> The press embargo was actually lifted yesterday (Jan. 2, 08). It's still a
> database.

And the E'rage team did a ton of work on the database, both the database
itself, and the rebuilding utility. Reliability is MUCH better, and the
Rebuild is no longer nearly as painful. It's almost transparent, other than
time.

--
John C. Welch

On 01/04/2008 09:50 AM, "LewisGmail" <gkremegmail.com> wrote:


>> Opening thousands of files can be very slow,
>
> It is generally much faster than seeking through a single database file.

Um, Oracle, Microsoft, and IBM would beg to differ. While hardware increases
can help, a database can search massive amounts of data far faster than you
can through an equal amount of individual files. Given enough RAM, that
difference increases as the DB can be kept in memory, so you get even more
speed. If nothing else, the delays involved in opening the files will cause
individual file searches to be slower than a database. A database isn't
always the right answer, but it is not "generally" slower than crawling a
file system.

>
>> and it can take up lots of system resources.
>
> No, that's not right at all. Basically, when the IMAP server looks at
> the maildir it sees three folders inside it, cur, old, and new (and a
> couple of tiny 'index' files of only a few bytes). The file names
> themselves, without even looking in the files at all, give the status
> of the mail; so for example, not a single file need be opened to see
> that this mailbox has 147 emails, 17 are unread, 11 have been replied
> to, 3 where forwarded, and 1 was redirected.

That assumes your search parameters only care about what the file names can
show you. As soon as you search based on terms that cannot be represented in
the file name, that trick no longer works.

--
John C. Welch

On 01/04/2008 09:50 AM, "Dan Frakes" <frakesmac.com> wrote:

> On 1/3/2008 4:30 AM, "Dan O'Donnell" wrote:
>>> Does anyone know if Entourage 2008 changes this behavior, or if it still
>>> uses the one big identity file?
>>
>> They know, but they can't say.
>> (At least not until 15 Jan 2008.)
>
> The press embargo was actually lifted yesterday (Jan. 2, 08). It's still a
> database.

With that in mind, (and since I'm technocally press):

There is an important new addition to E'rage's AppleScript Dictionary: Export Archive.

Export Archive allows you to export the contents of the e’rage database into an Entourage Archive, which is just a package containing the contents of the database, (sans attachments, but saving them out via AppleScript is a well – solved problem) in ‘standard’ files, i.e. mbox, ics, etc.

In my tests, a 1.23GB database, exporting everything created a 705.4MB export package.

Now, that package is easily backed up via Time Machine. The problem here is incremementals, however, that’s not impossible by any means, but it would be rather tedious.  
--
John C. Welch

On Jan 4, 2008, at 7:50 AM, Nik Friedman TeBockhorst wrote:

> For backups, I have found myself simply using ChronoSync to back up
> my Microsoft User Data folder on a less frequent basis than my Time
> Machine backups. I keep five versions of the DB (so five weeks' of
> versions), and I also know that all my email is backed up on my
> company's Exchange server. So at most I lose tasks and categories.

I use Entourage so I can answer questions about it. I use it only for
company-related mailing lists. If the database gets blown away beyond
repair, it will be only a minor inconvenience. And it's "only" 189
meg (I keep messages only very selectively--and most of what I've kept
are no longer of interest), so it doesn't impact backup enough to
worry about.

I don't think I'll be moving on to Office 2008 (I'm slightly more
likely to put a Home and Student Office 2007 onto my little used Vista
laptop--that has no Outlook.

Note that there is no Exchange server anywhere in sight here. So my
view of Entourage and Outlook isn't the same as that of lots of other
folks.

   --John

At 7:50 AM -0800 1/4/08, LewisGmail wrote:
>>Storing information in thousands of files is usually not a good idea.
>
>This is at worst just plain wrong, and at best severely incomplete.
>Storing tens of thousands of files on a Mac OS 8 system was a bad
>idea, but is not at all an issue on OS X. On a Windows 98 system, it
>was a dreadful idea (I have no idea how XP deals with a folder with,
>say, 10,000 items in it). On a UNIX system it is standard procedure
>because the OS is written to accommodate a large number of small files.

The success or difficulty of storing many/hundreds/thousands of files
in a single directory largely depends on the file system being used.
The modern HFS+ (journaled) is 'okay', as is EXT3. UFS is very
problematic, as I recall. XFS and GFS (and probably ZFS and EXT4,
whenever it's finished) are, from what I've heard, quite nice.

-Andrew

On Fri, 4 Jan 2008, John C. Welch wrote:

> On 01/04/2008 09:50 AM, "Chris Devers" <cdeverspobox.com> wrote:
>
> >> Storing information in thousands of files is usually not a good idea.
> >
> > The people migrating from mbox to maildir or mix, or from Panther's
> > version of Mail to Tiger's version of Mail, might disagree here.
>
> While computer speeds can compensate somewhat, the truth is, as you
> scale up the number of files you have to deal with, you start losing
> efficiency over a database. A database can handle ginormous amounts of
> data far more efficiently than a standard filesystem can, and you can
> slice the data in more ways far easier.
>
> The backup problem is one that has always, and still plagues
> databases, and always will, but if you have to relate a lot of similar
> kinds of data to each other in random ways, a database is going to be
> your best option.

Sorry, I missed the part where I was advocating against databases.

Oh wait, I never said that :-)

I'm not opposed to database indexing of the data for quick retrieval.

But I do think single monolithic files for storing a lot of information
is an approach that has difficulty scaling well, or recovering well from
problems, as opposed to indexed hierarchies of structured files.

And that's one reason (among many, really) that I avoid Entourage.


--
Chris Devers

On 4-Jan-2008, at 12:01, John C. Welch wrote:
> On 01/04/2008 09:50 AM, "LewisGmail" <gkremegmail.com> wrote:
>>> Opening thousands of files can be very slow,
>>
>> It is generally much faster than seeking through a single database
>> file.
>
> Um, Oracle, Microsoft, and IBM would beg to differ. While hardware
> increases
> can help, a database can search massive amounts of data far faster
> than you
> can through an equal amount of individual files. Given enough RAM,
> that
> difference increases as the DB can be kept in memory, so you get
> even more
> speed. If nothing else, the delays involved in opening the files
> will cause
> individual file searches to be slower than a database. A database
> isn't
> always the right answer, but it is not "generally" slower than
> crawling a
> file system.

Depends on what you are doing. Simply searching a database and
searching a structure file system is, honestly, not much different.
The power of the database is the ability to correlate data, search
through multiple data stores, and format and present the data it
finds. Searching your mail.app mail for the word "googlewhack" is not
going to be any slower than searching for it in your E'rage database.
Searching for emails that are related to the meeting you had on 12
June last year that was attended by six cow-orkers and two clients?
Well, that's where the database approach will really help.

> Depends on what you are doing. Simply searching a database and
> searching a structure file system is, honestly, not much different.
> The power of the database is the ability to correlate data, search
> through multiple data stores, and format and present the data it
> finds. Searching your mail.app mail for the word "googlewhack" is not
> going to be any slower than searching for it in your E'rage database.
> Searching for emails that are related to the meeting you had on 12
> June last year that was attended by six cow-orkers and two clients?
> Well, that's where the database approach will really help.

As the number of items you have to search grows, iterating through
individual files becomes more than noticeably slower. If you are searching
for something inside a file, then the database will move faster earlier,
since it doesn't have to do many, many file opens, reads, and closes. As
hardware gets faster, the point where databases get faster takes longer to
get to, but it's still quite definitely there.

--
John C. Welch

--On January 4, 2008 7:50:45 AM -0800 "LewisGmail" <gkremegmail.com>
wrote:

>> Opening thousands of files can be very slow,
>
> It is generally much faster than seeking through a single database file.

Uh no. 1st, Most databases do not store data in a single file, especially
not enterprise level ones used in large databases.

in the document management system I admin, we run an index server day an
night whose only job is to open files, read the contents and index them in
a database. We can do full text searches of thousands of files in seconds.
Simpler searches, such as searching for a particular filename, can do tens
of thousands of files in seconds.

Databases have one huge advantage that makes searching through them fast.
They are indexed. For single-purpose databases such as one dedicated to
e-mail, the indexes can be tweaked for common searches making them even
faster.
>
> No, that's not right at all. Basically, when the IMAP server looks at
> the maildir it sees three folders inside it, cur, old, and new (and a
> couple of tiny 'index' files of only a few bytes). The file names
> themselves, without even looking in the files at all, give the status
> of the mail; so for example, not a single file need be opened to see
> that this mailbox has 147 emails, 17 are unread, 11 have been replied
> to, 3 where forwarded, and 1 was redirected.

Courier-IMAP uses Maildir. Cyrus uses it's own methodology. There is a hash
directory structure for the user name (i.e. folder for my mail is in the
k/user/kevin directory. If a 2 letter has were defined the folder would go
in k/e/user/kevin.) This allows systems with many users to not have 10,000
folder names in one directory. Inside the user folder the messages are
stored in a folder structure that mimics the folder structure in the
Client. So the 8900 tidbits-talk messages that I've yet to archive are in a
folder named Tidbits-Talk.

I can prevent directories from having too many messages simply by creating
a folder and archiving mail to it (I have almost 20,000 Tibits-Talk mail
archived since 2000)

Cyrus uses a database for storing indexs with pointers to the actual e-mail
files. This makes searches very fast without needing to scan through
thousands of files one at a time.

On 01/06/2008 05:49 AM, "Chris Devers" <cdeverspobox.com> wrote:

> But I do think single monolithic files for storing a lot of information
> is an approach that has difficulty scaling well, or recovering well from
> problems, as opposed to indexed hierarchies of structured files.

Single monolithic files are seen everywhere, some of truly gargantuan sizes,
they store a lot of information, and they scale fine.

However, again, even with indexing, if you are looking for something not in
the index, and you have to search hundreds of thousands of files, what do
you do? Open, read, close, open, read, close. Spotlight tries to solve this,
but ask anyone who's had their spotlight indexes go corrupt, and everything
has to be redone again, it will whack your machine, and not for an
insignificant amount of time.

As well, I have yet to see, on the Mac, any of the applications that "do it
right", give you the linking and cross-referencing abilities that Entourage
does. Databases are for far more than just quick retrieval, although they do
that well too, they also give you the ability to relate all kinds of
different information to each other, and that's what E'rage does. Mail
doesn't do it, Thunderbird doesn't, nor does, from what I've seen, any other
email/contact/calendaring application on the Mac.

--
John C. Welch

The discussion about whether to have a database or a single file for
each mail is slightly pointless. Nothing about the "single file"
solution prevents the application from *also* having a database with
indices and metadata. In fact, I think many or even most modern mail
clients use this solution: Store each mail as a single file, but also
maintain a kind of database so loading mail lists and searching
through them is fast. Hence, easy incremental backup and easy
Spotlight support. This has the additional advantage that the database
can easily be recreated from the single mail messages (minus some
metadata, perhaps) if it is corrupted.

lucas

On 01/09/2008 09:10 AM, "Lukas Mathis" <Lukas.Mthis.li> wrote:

> The discussion about whether to have a database or a single file for
> each mail is slightly pointless. Nothing about the "single file"
> solution prevents the application from *also* having a database with
> indices and metadata. In fact, I think many or even most modern mail
> clients use this solution: Store each mail as a single file, but also
> maintain a kind of database so loading mail lists and searching
> through them is fast. Hence, easy incremental backup and easy
> Spotlight support. This has the additional advantage that the database
> can easily be recreated from the single mail messages (minus some
> metadata, perhaps) if it is corrupted.

There aren't many other email clients where the metadata is as important. I
use IMAP and Sync Services. Because of that, my raw email is backed up in a
number of places. It is the metadata that I get through the database that
gives E'rage it's value. Links, etc.

--
John C. Welch Writer/Analyst
Bynkii.com Mac and other opinions
jwelchbynkii.com

On 05/01/2008 4:50 AM, "Chris Devers" <cdeverspobox.com> spake thus:

> My concern is that disk damage happens, and takes files down with it.
>
> One bad block has a higher risk of hitting a big file than a small one.

This has happened to me twice now with my Entourage DB. Both times I was
extremely fortunate enough to be able to rebuild the database and lose maybe
a single message (you're never quite sure, of course, but nothing critical
disappeared either time).


On 05/01/2008 8:01 AM, "John C. Welch" <jwelchbynkii.com> spake thus:

> While computer speeds can compensate somewhat, the truth is, as you scale up
> the number of files you have to deal with, you start losing efficiency over
> a database. A database can handle ginormous amounts of data far more
> efficiently than a standard filesystem can, and you can slice the data in
> more ways far easier.

In essence, most monolithic database files are created in an effort to
create a "private" file system customised for the purposes of the
application in question. This is essentially what Oracle does, for example:
files in Oracle are essentially a means of allocating disk space, which is
then taken over and "formatted" by Oracle itself. This makes it possible for
Oracle to sell machines that have no OS other than Oracle itself (i.e.,
Oracle running on bare metal), for those who need absolute performance.

One possible solution to this single/multiple file dichotomy could be to
keep the existing single-file database, but provide something like a FUSE
plugin to expose the contents as a file system to the OS. That way you get
the advantages of your own custom database format, and things like Spotlight
are able to access emails as individual "files" without having to resort to
jiggery-pokery. (Although that may be more or less what Spotlight does with
Entourage now --- I don't know enough about how metadata importers work to
be certain.)

That said, I don't see any particular reason, other than convenience, why
Entourage has to use a single file for everything. DBMS vendors long ago
recognised the benefits of physically splitting up a database for
performance reasons, although to be fair many of those benefits only really
accrue when you put the files on different disks. Hardly the norm for your
typical home user...

--
Nigel Stanger, Dunedin, NEW ZEALAND.
http://xri.net/=nigel.stanger

On Jan 15, 2008 7:01 AM, Nigel Stanger <nstangerinfoscience.otago.ac.nz> wrote:
> That said, I don't see any particular reason, other than convenience, why
> Entourage has to use a single file for everything. DBMS vendors long ago
> recognised the benefits of physically splitting up a database for
> performance reasons, although to be fair many of those benefits only really
> accrue when you put the files on different disks. Hardly the norm for your
> typical home user...

I can think of very few databases that go as far as to split up files
for each individual entry (and those "databases" are really little
more than text files with an indexed overlay).

Entourage's (and Outlook, for that matter) let you create links
between all your items, be they tasks, notes, emails, addresses, etc.
Keeping that together and fast requires some sort of underlying
database so that you can quickly pull "queries" to find related items.

Granted, Entourage is fairly lightweight in this regard, but still, I
can see why Microsoft took this approach.

If you're really serious about having last-ditch recovery of your
email, you can mine Entourage's spotlight store. All your email
messages are available as XML files (plists, actually) in
~\Library\Caches\Metadata\Microsoft\Entourage\ in a fairly complex
filesystem. If you back that folder up, you get a set of your messages
that's complete and parse-able (albeit with quite a bit of work).

--Nik

On 16/01/2008 11:46 PM, "Nik Friedman TeBockhorst" <nikinik.net> spake
thus:

> Entourage's (and Outlook, for that matter) let you create links
> between all your items, be they tasks, notes, emails, addresses, etc.

Sure, but that doesn't mean they can't be stored in separate files. This
sort of thing happens all the time in *really* large databases. Your
50-million row Customer table gets split across five different disks (this
is not the same thing as RAID, BTW) so that you can have up to five truly
simultaneous reads happening, while the even larger Order table (which is
linked to Customer), is split up in a similar way across a completely
separate collection of disks.

In fact, the performance of "links" should be better in this scenario,
because we can retrieve a customer and their orders at the same time. In a
single-file scenario, we would have to read first one, then the other. Of
course, as I said, this scenario is irrelevant for most home use as most
people have only a single disk. Even RAID is fairly uncommon.

Even in the single-disk case, however, smaller files can be an advantage,
simply because there's less data to wade through. Again, however, this is
probably not an issue for something like Entourage, because the kinds of
queries that can be posed on the underlying data are strictly limited by the
UI, and they'll have indexed for all the cases that they allow. This becomes
more of an issue in a more general database where queries can be ad hoc and
not everything is indexed (lots of indexing is a Very Bad Thing if the
database changes a lot).

OK, enough database geekery :)

--
Nigel Stanger, Dunedin, NEW ZEALAND.
http://xri.net/=nigel.stanger

On 1/16/08 2:46 AM, "Nik Friedman TeBockhorst" <nikinik.net> wrote:

> If you're really serious about having last-ditch recovery of your
> email, you can mine Entourage's spotlight store. All your email
> messages are available as XML files (plists, actually) in
> ~\Library\Caches\Metadata\Microsoft\Entourage\ in a fairly complex
> filesystem. If you back that folder up, you get a set of your messages
> that's complete and parse-able (albeit with quite a bit of work).

The contents are the first 75K of a message body. Don't rely on it for a
full message.

--
Diane Ross
Entourage Help Page <http://www.entourage.mvps.org/>
Entourage Help Blog <http://blog.entourage.mvps.org/>

On 01/17/2008 06:05 AM, "Nigel Stanger" <nstangerinfoscience.otago.ac.nz>
wrote:

>> Entourage's (and Outlook, for that matter) let you create links
>> between all your items, be they tasks, notes, emails, addresses, etc.
>
> Sure, but that doesn't mean they can't be stored in separate files. This
> sort of thing happens all the time in *really* large databases. Your
> 50-million row Customer table gets split across five different disks (this
> is not the same thing as RAID, BTW) so that you can have up to five truly
> simultaneous reads happening, while the even larger Order table (which is
> linked to Customer), is split up in a similar way across a completely
> separate collection of disks.
>
> In fact, the performance of "links" should be better in this scenario,
> because we can retrieve a customer and their orders at the same time. In a
> single-file scenario, we would have to read first one, then the other. Of
> course, as I said, this scenario is irrelevant for most home use as most
> people have only a single disk. Even RAID is fairly uncommon.
>
> Even in the single-disk case, however, smaller files can be an advantage,
> simply because there's less data to wade through. Again, however, this is
> probably not an issue for something like Entourage, because the kinds of
> queries that can be posed on the underlying data are strictly limited by the
> UI, and they'll have indexed for all the cases that they allow. This becomes
> more of an issue in a more general database where queries can be ad hoc and
> not everything is indexed (lots of indexing is a Very Bad Thing if the
> database changes a lot).

And are you willing to pay enterprise-class DB prices for a consumer-grade
DB? Those features are certainly an option, but how many people are going to
pay thousands of dollars for a PIM/Exchange client?

--
John C. Welch Writer/Analyst
Bynkii.com Mac and other opinions
jwelchbynkii.com

On 18/01/2008 1:38 PM, "John C. Welch" <jwelchbynkii.com> spake thus:

> And are you willing to pay enterprise-class DB prices for a consumer-grade
> DB? Those features are certainly an option, but how many people are going to
> pay thousands of dollars for a PIM/Exchange client?

Yes, I freely admit that most of what I was talking about is mainly
applicable to enterprise-scale applications. I just don't often get the
opportunity to talk about these things in a more general context :)

--
Nigel Stanger, Dunedin, NEW ZEALAND.
http://xri.net/=nigel.stanger