Spam

On 26-Mar-2009, at 03:20, Marshall Clow wrote:
> [ And yes, I know that this implies that server-based, site-wide spam
> filters cannot, in the general case, work correctly. I'm not
> particularly pleased with that, either. ]

Site-wide filtering based on content (Bayes) is usually a pretty bad
idea unless everyone in the company/organization is getting pretty
much the same types of mail. Also, the general recommendation is to
exclude list mail from spam filters.

To be perfectly honest, if we could simply eliminate the forged and
zombie-sourced spam messages the entire 'spam problem" would be quite
manageable. On my server the outright rejection rate (servers that are
confirmed to be spam sources or servers on dynamic IPs or servers
without FQDNs) is between 75-95%. For the last week the percentages
have been; 80%, 77%, 84%, 93%, 89%, 81%, and 77%.

Those are severs that are reject before I even see the text of their
messages. Of the messages that get through, approximately 20% of those
are spam with forged headers. Servers that haven't made it into the
RBL as yet. Generally, if I test those messages a day or two later,
they would be rejected at the same rate as general connections.

There are some spams that look very legitimate. They have FQDNs,
their from addresses match their sender domain, they have DomainKeys
that check out and yet they are still clearly spam. These are a tiny
percentage of the overall flood, and it these were all we had to deal
with we would not have much of a spam problem.

That said, good Bayes filtering is a marvelous tool and saves me from
a lot of spam:

Here's the SpamAssasin header from a recent example:
Content analysis details: (5.4 points, 5.0 required)

pts rule name description
---- ----------------------
--------------------------------------------------
3.5 BAYES_99 BODY: Bayesian spam probability is 99 to 100%
                            [score: 1.0000]
-1.3 DKIM_VERIFIED Domain Keys Identified Mail: signature
passes
                            verification
-1.0 DKIM_SIGNED Domain Keys Identified Mail: message has a
signature
0.0 HTML_MESSAGE BODY: HTML included in message
1.5 MIME_HTML_ONLY BODY: Message only has text/html MIME parts
1.1 SARE_SUB_GRANT Spammer subject - credit or money
1.7 SARE_MSGID_DDDASH Message-ID has ratware pattern (9-, 9$, 99-)

This spam had two very strong indicators against it being spam, the
DKIM flags give this message a starting value of -2.3, well under the
spam threshold. Even with the lack of a plain text portion, the
spammish subject "Government Grant Scams" and the 'ratware' message ID
this message would still be well under the spam threshold of 5.0 with
a total score of only 1.9. Fortunately I've been running bayes along
with SA for years, so Bayes says this message has a 99% probability of
being spam and that is enough to properly push it into the SPAM folder
where I don't have to see it.

--
<http://www.pvponline.com/2004/01/14/wed-jan-14/>

There has been quite a bit of action in Tidbits about spam, etc. In today's Future Tense a link to a listing of rogue software was given. It is a link to a Wikipedia article and is:

At 2:43 PM -0700 3/26/09, Kirk McElhearn wrote:
>On Mar 26, 2009, at 10:20 AM, Marshall Clow wrote:
>
>>However, none of this email is spam - because I signed up for the
>>mailing list. Deliberately and on purpose. Any spam-filtering
>>software that marks any of these messages as spam is behaving
>>incorrectly.
>
>So it's up to you to set a rule in your e-mail program to filter it. I
>have my spam filtering rule (with Intego's Personal Antispam) set to
>filter messages after a number of rules that sort mailing lists.

I don't have a problem with that - and you've (implicitly) agreed
with my point (that you snipped), that server-based filtering is (in
general) impossible to do correctly.
--
-- Marshall

On Mar 26, 2009, at 5:43 PM, Kirk McElhearn wrote:

> Has anyone noticed an increase in spam in the past few days? My stats
> are alarming; from fewer than 200 spams a day, I'm up to 250 on
> Monday, 300 on Tuesday, and 350 on Wednesday.

I do on one of my e-mail accounts (not this one). Apple Mail has
caught them all; I don't use any other spam filters.

Marilyn

On 27 Mar 2009, at 21:15, Marshall Clow wrote:

> I don't have a problem with that - and you've (implicitly) agreed
> with my point (that you snipped), that server-based filtering is (in
> general) impossible to do correctly.

I'm not sure I'm following this. Why do you believe that server-based
filtering is impossible to do correctly? I can understand that non-
user-specific filtering won't work well, but if you're doing per-user
filtering, I don't see that it matters whether this happens on the
server or in the client.

I do spam filtering on the server for a small set of domains with
SpamAssassin. A lot of SpamAssassin's rules are non-user-specific -
such as header validity checks, distributed hash checks, realtime URL
blacklists and scans for common spam patterns. However, bayesian
filtering and black/white-listing is done with per-user databases.

The advantage of not doing it in the client is being able to have a
simpler client. As I noted before, I read much of my email on my
iPhone now, and that would be impossible without server-side spam
filtering.

Jonathan

On 3/28/09 at 2:11 AM, jonathanonegoodidea.com (Jonathan Hogg)
wrote:

>On 27 Mar 2009, at 21:15, Marshall Clow wrote:

>>I don't have a problem with that - and you've (implicitly) agreed
>>with my point (that you snipped), that server-based filtering is
>>(in general) impossible to do correctly.

>I'm not sure I'm following this. Why do you believe that
>server-based filtering is impossible to do correctly? I can
>understand that non- user-specific filtering won't work well, but if
>you're doing per-user filtering, I don't see that it matters whether
>this happens on the server or in the client.

As a user, there is a significant difference. If I enable the
spam filters my ISP provides, I have to use a web interface to
check emails trapped by those filters. That is far less
convenient than scanning a folder on my machine to check those
emails. Additionally, I can easily modify filters on my machine
as I deem appropriate. I have no such ability to modify the spam
filters my ISP uses.

So, until there is a set of filters which never identify email I
want as spam, server side filtering is worse than useless to me.

--On March 28, 2009 4:50:03 PM -0700 Bill Rowe <readlistssbcglobal.net>
wrote:

>> I'm not sure I'm following this. Why do you believe that
>> server-based filtering is impossible to do correctly? I can
>> understand that non- user-specific filtering won't work well, but if
>> you're doing per-user filtering, I don't see that it matters whether
>> this happens on the server or in the client.
>
> As a user, there is a significant difference. If I enable the
> spam filters my ISP provides, I have to use a web interface to
> check emails trapped by those filters. That is far less
> convenient than scanning a folder on my machine to check those
> emails. Additionally, I can easily modify filters on my machine
> as I deem appropriate. I have no such ability to modify the spam
> filters my ISP uses.
>
> So, until there is a set of filters which never identify email I
> want as spam, server side filtering is worse than useless to me.

For the mail server I run for my family if the server identifies spam it
tosses it. For all accounts except my own the spam threshold settings are
pretty light (headers are added to suspected spam so they can filter mail
on the receiving side if they desire.) I've had my e-mail account
considerably longer than they have so i have my settings set very
aggressively. Users can go tweak their settings if they wish.

There are a couple of solutions to your issue:

1) ISP uses IMAP then your spam could be kept in a server side folder that
you would access through your e-mail program like a normal folder.

2) I believe a similar setup could be done with POP but you might want to
configure your e-mail program to download headers only.

3) At my office we're e-mailed a daily report of what has been captured as
spam. date, sender, subject is listed for each captured e-mail and you can
click an e-mail to be taken to the page to release it. We can tweak some
filter settings (whitelist addresses or domains, but not blacklist).

To be honest a combination of server and client side is needed. Your ISP
is probably rejecting a ton of spam you're probably not even aware of (via
DNS blacklists or several other methods.) You do not want to deal with the
deluge of spam a completely unfiltered mailbox would have to handle.

On 28-Mar-09, at 7:50 PM, Bill Rowe wrote:
>
> As a user, there is a significant difference. If I enable the
> spam filters my ISP provides, I have to use a web interface to
> check emails trapped by those filters. That is far less
> convenient than scanning a folder on my machine to check those
> emails. Additionally, I can easily modify filters on my machine
> as I deem appropriate. I have no such ability to modify the spam
> filters my ISP uses.
>
> So, until there is a set of filters which never identify email I
> want as spam, server side filtering is worse than useless to me.

While it's true of most ISPs, there are other options. Tuffmail for
one. It's spam folders appear right in your email client on your
computer just like any other folder (they are an IMAP provider). No
need to use a web interface to check for false positives. They also
give you a lot of control over the configuration of the spam filtering
on a per email address basis. It's not free, but they do offer free 30
day trials so you can see for yourself if it's what you need.

I don't have any connection to Tuffmail except as a very satisfied
customer.


Bill

On 28-Mar-2009, at 17:50, Bill Rowe wrote:
> So, until there is a set of filters which never identify email I
> want as spam, server side filtering is worse than useless to me.

That is a failing of your ISP, not of server side filtering. Here's
how it works on my server.

Messages are scanned by SpamAssassin. If they are suspected to be
spam, they are subject-tagged, enclosed into an attachment
(report_safe 1), and put into the user's .SPAM maildir. This means the
messages are available from the webmail interface or whatever IMAP
client the user decides to use.

Because the messages are attachments, it is trivial to pull a false
positive out and then put it in the right folder. All the SA tagging
goes away and the original untouched message is preserved. Once a
message is accepted the server never deletes it, regardless of how
high it scores. Even if the user has setup procmail to autodelete spam
at a certain score-level, that deletion doesn't happen until after the
message is archived and is recoverable for at least a week.

Of course, on my own mail accounts I toss out anything over 9.0 on all
accounts but my admin account. that account never deletes mail
either. I scan the .SPAM maildir every week or so on the off chance
there is something important that got miss-tagged.

Most user, honestly, don't even subscribe to the .SPAM maildir in
their clients (except the Mail.app users who have no choice :/ )

--On March 28, 2009 2:11:05 AM -0700 Jonathan Hogg
<jonathanonegoodidea.com> wrote:

> The advantage of not doing it in the client is being able to have a
> simpler client. As I noted before, I read much of my email on my
> iPhone now, and that would be impossible without server-side spam
> filtering.

There are bandwidth savings as well. I'm noticing a lot of rejected spam
have a .png image file attached them. Downloading thousands of those would
really suck down your bandwidth.

On 29 Mar 2009, at 09:41, William Poupore wrote:

> On 28-Mar-09, at 7:50 PM, Bill Rowe wrote:
>>
>> As a user, there is a significant difference. If I enable the
>> spam filters my ISP provides, I have to use a web interface to
>> check emails trapped by those filters. That is far less
>> convenient than scanning a folder on my machine to check those
>> emails. Additionally, I can easily modify filters on my machine
>> as I deem appropriate. I have no such ability to modify the spam
>> filters my ISP uses.
>>
>> So, until there is a set of filters which never identify email I
>> want as spam, server side filtering is worse than useless to me.
>
> While it's true of most ISPs, there are other options. Tuffmail for
> one. It's spam folders appear right in your email client on your
> computer just like any other folder (they are an IMAP provider). No
> need to use a web interface to check for false positives. They also
> give you a lot of control over the configuration of the spam filtering
> on a per email address basis. It's not free, but they do offer free 30
> day trials so you can see for yourself if it's what you need.

Yes, what you (Bill) appear to be describing here is not a failing of
server-side filtering, but a failure of your ISP's server side
filtering.

As William says, the best way to do server-side filtering is to filter
suspect email into folders that can be checked with a regular IMAP
client. That is what I do on my server for my users. Further to that I
also have the system set up to re-train the bayesian filter with
messages that are moved from one folder to another. So one need only
drag a missed piece of spam to the junk folder or drag a false
positive back to the inbox.

I'm not suggesting server-side filtering is for everyone, just that it
is not necessarily impossible or unusable. It helps to have a
motivated server admin: since I receive by far the most spam on my
server, I'm pretty motivated to sort it out ;-)

Jonathan

On 3/30/09 at 2:08 AM, kremelskreme.com (LuKreme) wrote:

>On 28-Mar-2009, at 17:50, Bill Rowe wrote:
>>So, until there is a set of filters which never identify email I
>>want as spam, server side filtering is worse than useless to me.

>That is a failing of your ISP, not of server side filtering. Here's
>how it works on my server.

>Messages are scanned by SpamAssassin. If they are suspected to be
>spam, they are subject-tagged, enclosed into an attachment
>(report_safe 1), and put into the user's .SPAM maildir. This means
>the messages are available from the webmail interface or whatever
>IMAP client the user decides to use.

That is essentially how my ISP does things and is precisely what
I don't like.

What you have described seems inherent in any server side
solution. That is the separation between spam and wanted email
happens on the server and the spam is not forwarded to a POP
client. That means I have to use something other than my POP
client to check if any of the emails identified as spam or in
fact emails I want. This involves additional effort on my part
which is what I want to avoid and why I don't prefer a server
side solution.

The extra effort needed on my part for a server side solution
only goes away if the server never identified email I want as
spam or if I use something other than a POP client to access my
email. It seems unlikely any server side solution will be good
enough to eliminate the need for verification and I have no
desire to move to a different email client.

On 3/30/09 at 2:08 AM, jonathanonegoodidea.com (Jonathan Hogg)
wrote:

>Yes, what you (Bill) appear to be describing here is not a failing
>of server-side filtering, but a failure of your ISP's server side
>filtering.

>As William says, the best way to do server-side filtering is to
>filter suspect email into folders that can be checked with a regular
>IMAP client.

You assume usage of a IMAP client. I use a POP client. In fact,
I don't even know nor care if my ISP supports IMAP.

>I'm not suggesting server-side filtering is for everyone, just that
>it is not necessarily impossible or unusable.

I am not saying it is either impossible or unusable. I am saying
leaving email on the server in a spam folder means I have to
access it via software other than my POP client. That means
additional effort on my part to deal with it.

On 3/31/09 10:49 AM, "Bill Rowe" <readlistssbcglobal.net> wrote:

> What you have described seems inherent in any server side
> solution. That is the separation between spam and wanted email
> happens on the server and the spam is not forwarded to a POP
> client. That means I have to use something other than my POP
> client to check if any of the emails identified as spam or in
> fact emails I want. This involves additional effort on my part
> which is what I want to avoid and why I don't prefer a server
> side solution.

However, for your ISP, or for any company with its own email servers and
internet connection the advantage to an external service is clear: If a
postini handles all of that for you, it never hits your router or your mail
server, resulting in a MUCH smaller load, less vulnerability to mail bombs,
etc.

--
John C. Welch

I simply don't use my ISP's email service and I never did. I use either the IMAP server provided with my web hosting account or Gmail (which I currently prefer). I don't like ISP-specific email addresses and I like being able to change ISPs freely based on speed & reliability without changing my email address. When my DSL service was down for a few days and BellSouth couldn't schedule repairs for almost a week, I simply called Comcast and was online in an hour, since I already used them for cable TV. I didn't have to change my email address or anything else, and the switch was no big deal.

On 31 Mar 2009, at 15:49, Bill Rowe wrote:

> You assume usage of a IMAP client. I use a POP client. In fact,
> I don't even know nor care if my ISP supports IMAP.

I don't want to start any kind of argument over your choice, but,
given that IMAP has been around for over 20 years now, I'd be
surprised if either your ISP or your email client didn't support it.

Jonathan

On 31-Mar-2009, at 08:49, Bill Rowe wrote:
> the spam is not forwarded to a POP client.

No it's not, but then again, we no longer offer POP3 access at all
(and had only had very limited POP3 access for the last 8 years). IMAP-
SSL only now, and IMAP-SSL only for most users for near-on a decade.
We decided on EITHER webmail OR POP3. Webmail won out in a landslide,
especially after we had several issues with the insecure nature of POP3.

> That means I have to use something other than my POP
> client to check if any of the emails identified as spam or in
> fact emails I want. This involves additional effort on my part
> which is what I want to avoid and why I don't prefer a server
> side solution.

POP3 is a hideous mail standard. There are very few mail clients that
don't support IMAP (in fact, I can only think of one off the top of my
head) and IMAP allows us to very efficiently do things like spam
filtering that allows users to still catch false-positives. The amount
of data being transferred to the user alone is significantly less
because, as I pointed out, most users who have the choice do not
subscribe to their SPAM mailbox (they login to the webmail if they
want to check it, generally). Also, users really like being able to
login to their webmail and see exactly the same mail, with exactly the
same status, as they last saw in their regular client. This makes
checking your email on someone else's computer convenient, and trivial.

I also recently made a significant change in the way the SPAM mailbox
was handled, automatically expiring mail out of it after 30 days (and
the Trash as well). The comments I got on that where so positive that
I reduced the SPAM auto-delete to 7 days.

Of course, ALL mail is still backed-up with the rest of the system for
a minimum of 32 days so if something important gets lost we can
recover it. The fact of the matter is, SpamAssassin does an excellent
job and nearly no mail gets incorrectly tagged. Some spam does get
through, but that is not really an issue as it's very little and it's
better to err on the side of to much mail than too little.

On Apr 1, 2009, at 4:56 AM, Jonathan Hogg wrote:

> On 31 Mar 2009, at 15:49, Bill Rowe wrote:
>
>> You assume usage of a IMAP client. I use a POP client. In fact,
>> I don't even know nor care if my ISP supports IMAP.
>
> I don't want to start any kind of argument over your choice, but,
> given that IMAP has been around for over 20 years now, I'd be
> surprised if either your ISP or your email client didn't support it.
>

Most ISPs don't support IMAP because they don't want everyone storing
their email on their server and using up too much space. Yet another
reason I never use my ISP's mail service.

On 4/1/09 at 1:56 AM, jonathanonegoodidea.com (Jonathan Hogg) wrote:

>On 31 Mar 2009, at 15:49, Bill Rowe wrote:
>
>>You assume usage of a IMAP client. I use a POP client. In fact, I
>>don't even know nor care if my ISP supports IMAP.

>I don't want to start any kind of argument over your choice, but,
>given that IMAP has been around for over 20 years now, I'd be
>surprised if either your ISP or your email client didn't support it.

My email client is Mailsmith which definitely doesn't support
IMAP even though it around for significantly less time than the
period you've indicated for IMAP. I am running the latest beta
version of Mailsmith. Barebones has made clear support for IMAP
is not in Mailsmith's immediate future. And since Mailsmith
doesn't support IMAP, I've never bothered to determine whether
my ISP supports IMAP or not.

On 1-Apr-2009, at 02:58, LuKreme wrote:
> No it's not, but then again, we no longer offer POP3 access at all


I realised after reading this that I'd left out one of the key points
I wanted to make.

IMAP offers several significant advantages over POP3 whilst POP3
offers absolutely no advantages over IMAP. In fact, one can configure
ones IMAP account to behave exactly like a POP3 account if one really
wants.