On Aug 16, 2006, at 9:30 PM, Sander Tekelenburg wrote:
> At 19:52 -0700 UTC, on 2006-08-16, John W Baxter wrote:
>
>> Moderator: Message Cc'd directly to Sander in case you want to
>> drop it.
>
> Agreed, this is probably getting too off-topic. Still CC-ing
> TidBITS though,
> in case the moderators are interested in more goodies from John :)
>
>> On Aug 16, 2006, at 5:29 PM, Sander Tekelenburg wrote:
>>
>>> Are you saying Eudora can know that the ">" isn't there on purpose,
>>> inserted by the sender?
>>
>> That could be why Eudora doesn't try. Probably the better assumption
>> would be that the sender never meant it
>
> I don't know. I have this feeling it's bad enough that one party
> messes with
> the contents of mail. If other parties do guesswork to unmess it,
> there's a
> risk of messing things up even more.
>
> Man, I never realised mail was this messy... It must be great fun
> to run
> mailservers ;)
And I hadn't even gotten into the SMTP side of things (transmission
among mail servers). There, the sequence
CR LF . CR LF
is the signal that the message data has ended. Meaning a lone "." on
a line would end a message prematurely. So the sending SMTP server
inserts a . before the real data on a line starting with ., and the
receiving one removes the . from lines with more than one . between
the CR LF sequences. (Messages are--normally--stored on a machine
using the machine's line ending convention, but in transmission the
CR LF sequence is the line ender.)
>
> [...]
>
>> telnet server.example.com 110
>
> Ah! I needed to specify the port of course :) Thanks. With the
> commands you
> listed I managed to look at mail on my POP server just fine.
> Definitely
> useful to know how to do this. And confirmed: in this way too do I
> see an
> inserted angle bracket on my POP server in my test messages.
>
>> The 110 specifies the port. Unless your ISP requires the use of SSL,
>> they do allow that, or POP wouldn't work. It's not just the
>> occasional person doing POP by hand that uses telnet, it is also the
>> mail client programs.
>
> Ha! I never realised that.
>
> [...]
>
> [$telnet mail.euronet.nl 110]
>
>> I'm not sure that that initial banner means they use Qualcomm's (ie,
>> Steve's) qpopper, but I think so. If so, the latest version is
>> 4.05.
>
> Oh wow, so with 2.53 they're apparently using an ancient version then.
On the other hand, early versions of 3.x had a security flaw. I'm
not sure how much real progress is being made in qpopper these days
(or how much is needed).
>
> But, I'm still confused about exactly what part of mail is
> inserting that
> angle bracket. Are you saying qpopper does? Because another part of
> my ISP's
> mail servers seems to be Postfix, so up until now I was under the
> impression
> that Postfix inserts those angle brackets. (I know that mail
> servers exist of
> different parts, MTA's and something else, or even multiples of
> else, but I'm
> unaware of the details.)
It is Postfix which is inserting the >. On the Euronet servers, the
> is present. The POP server is merely passing it along untouched
(and also using it to count messages. (The RETR, DELE, etc commands
have a message number argument, and when the storage is mbox the
messages are counted based on the "From blah" lines. You can RETR
the last message first if you want to, or just the 17th out of 37.
>
> [...]
>
>> A pair of messages at the start an mbox file looks something like
>> this [...]
>>
>>
>> From <some text showing the return path and the date*>
>> SomeHeader: stuff
>> From: John <example

example.com>
>> SomeOtherHeader: stuff
>>
>> Message body stuff (which may itself start with a headerish line)
>>
>> From <here starts the next message>
>> SomeHeader:
>> From: ...
>> More Headers:
>
> Now I'm confused again :) Why are there both a "From " and a "From:
> " header
> here? I don't think I've ever ran across anything but a "From: "
> header in
> email. Not even when I use that telnet approach. That suggests it
> is not a
> header at all, but 'just' the delimiter in the POP server's mbox,
> and not
> made visble to (or by?) MUAs? Is it generated by the MTA that
> writes the
> mbox? Or by the sending MUA? Or by the smtp server?
>
> (Sorry, but that's what you get for giving such great information:
> more
> questions ;))
I did that deliberately to show that the "From " line is not the
From: header in the message. The "From " line does not travel over
the wire between SMTP servers; it is generated when the message is
stored in mbox format. Other storage formats exist in which there is
no "From " line, in at least some of those there is no messing about
with added ">" characters.
The ISP you found with your test messages, which didn't "helpfully
provide" the ">", is using one of those, probably "maildir" in which
each message is a separate file (with things like having been seen
(S), replied to (R I think), trashed (T), and flagged (F I think)
tacked onto the end of the file name on the server.
With mbox, the entire mailbox content is one file (meaning if you
delete a message, the file has to be copied without that message.
That becomes really slow if people let their mailbox files grow (and
really hard to fix by hand on the server when that happens--speaking
from experience). In the extreme case, the mbox file has grown so
large that the mail client times out while the POP server is doing
its copy. (qpopper, at least, copies the mbox file at login time,
then deals with another--complex--copy at logout to take care of both
shrinkage due to deletions and growth due to new messages arriving
between login and logout.)
We use maildir, which is why it's hard for me to test mbox.
There are a couple of reasons I said a few messages ago that Eudora
uses an "mbox-like" format, rather than saying mbox. One is that the
messages aren't deleted from the middle when you delete them--Eudora
keeps track of things using its Table of Contents. So if you open
one of Eudora's mailbox files in a text editor, you see deleted
messages along with current ones. When Eudora--either automatically
or manually--"compacts" a mailbox, it is getting rid of the stale
bytes from deleted messages.
While we're at it, Eudora, on the one hand and Windows Outlook
Express, on the other hand, use different logic in sending DELE
commands. Eudora sends a DELE for a message as soon as it knows it
wants the message off the server. Right after a successful RETR
command if it isn't leaving messages on the server, for instance.
Outlook Express keeps track of what messages it wants to issue DELE
for, and does so when everything else is done. That means if OE
crashes on the 10th message, it doesn't delete the first 9. That
also makes manual cleanup on the server harder. There was an overly-
popular Christmas message sometime around 1996 with crashed OE every
time. So not only was OE crashing, it was giving users multiple
copies of everything before the problem message. Aarrgh. I spent
hours cleaning up those mbox files. (OE may have reformed since
then; I haven't cared since we ditched mbox. And in Vista, there
will be no "Outlook Express"--it will be called Microsoft Mail.)
Glossary:
POP: Post Office Protocol (which is now--and has been for more than
25 years--at POP3); server to mail client.
SMTP: Simplified Mail Transport Protocol. Server to Server. Also
decades old (and broken by the spammers, unfortunately).
"over the wire": the "wire" is mostly fiber these days, but "over
the wire" is still used to describe what is transmitted from machine
to machine.
--John