Remember Watson?

I never used Sherlock for disk searches because it was a PITA to open
Sherlock to do the search. By the time I opened Sherlock up, I could
usually find what I was looking for anyway.

However, Sherlock 3 with its various plugins were helpful. When people
said they copied Watson, I took a look at Watson and fell in love. I
used it for searching for movies, finding what's on TV, references,
weather, and tracking packages.

You're right that Google did a lot of damage. One of the big features
of Watson/Sherlock was its ability to search multiple Internet search
engines and present them in an easy to use list. With Google becoming
the dominate search technology, there's no reason to search dozens of
engines. Google will find it all by itself. Today, I received an email
that a package I ordered had shipped. I simply highlighted the tracking
number in Apple's Mail, went to the Services Menu and selected "Search
with Google". Google returned the tracking information I needed. Who
needs to open up Watson or Sherlock when Google will do it for you?

However, I still do miss Watson. Maybe Dashboard will be a good
replacement.

--
David Weintraub
davidweintraubworld.net
davidweintraub.name

That technology is being replaced, or so I presume; since Jobs pre-announced this and demoed the capabilities back in May and again, this month, developers are working on widgets, rather than plug-ins. The advantages are several. With Sherlock or Watson (which I loved) the user must 'switch' among the various tools. You could not have two tools available "to view" simultaneously without opening two windows, wasting more screen space than two individual widgets. With Dashboard, presumably the distraction of screen clutter can be offset by the users' needs for simultaneous input from more than a single widget at a time while 1) reducing required screen real estate and 2) allowing for "best" layout methodology for a given tool.

Watson and Sherlock, in the attempt to improve on the browser window, basically suffered from the same problem: A semi-static or rigid presentation of information in a tabular view. Widgets, on the other hand, allow for creative display of information in various ways that may be more suitable than a typical table view within a rectilinear window.

I recently was assigned an internal tech support case in which an
internal client suddenly was unable to get Sherlock results. I tested
and discovered that my four machines all failed to return Sherlock
results.

Too many to be a coincidence.

Checked the ports used, and the result is (mostly) port 80. Hm.

Checked with the firewall admin and he discovered that the firewall -
called SmartDesign or something like that - is blocking Sherlock
because it sends/receives a URL with /../ and this is interpreted as
a malformed URL string and an attack on Microsoft IIS.

I don't know IIS well enough to know the attacks, but a) Sherlock is
now useless behind our firewall and b) there is no way the admin can
or will allow it.

Your mileage may vary.

At 1:23 PM -0800 1/25/05, David Weintraub wrote:
>Today, I received an email
>that a package I ordered had shipped. I simply highlighted the tracking
>number in Apple's Mail, went to the Services Menu and selected "Search
>with Google". Google returned the tracking information I needed. Who
>needs to open up Watson or Sherlock when Google will do it for you?

Perhaps I'm overly picky. I find Google's approach to non-web
searches (i.e. UPS numbers, reverse lookups, etc..) helpful, but,
well slow. If I hand Google a UPS tracking number, there isn't much
point in telling me there was no found searches, except a possible
UPS tracking number. It should either, (1) get the tracking
information and display it or (2) redirect me to UPS's site.

It might be a bit of a UNIX geek feature, but that little Google box
isn't as strong as I'd like it. I've been working on whipping up a
"search director" which recognizes tracking numbers and other
information and sends your search to the right site. I think
something like this will be the next set of innovations in search.
Google attempts to be the "the closest thing the Web has to an
ultimate answer machine." It might be good at finding possible
answers, but if I hand it a tracking number or the like, I want _the_
answer not a possible answer.

Nick

>Checked with the firewall admin and he discovered that the firewall -
>called SmartDesign or something like that - is blocking Sherlock
>because it sends/receives a URL with /../ and this is interpreted as
>a malformed URL string and an attack on Microsoft IIS.
>
>I don't know IIS well enough to know the attacks,

HTTP URLs conform to UNIX path conventions.

In UNIX the string ".." represents "go up a
directory" in a path. ("." being "this
directory", ".." being the parent directory, and
"/" being the directory delimiter.)

IIS was discovered (v4 I think, a long time ago,
now since corrected BTW) to be subject to a
"directory traversal" attack... meaning you could
craft a URL that would allow you to escape the
web root directory, and then find your way to a
known binary and execute a buffer overflow. Like
this (a simplified example):

"/../../../../../../Win32/Programs/cmd.exe?#######<insert evil? code here>"

Basically: "Go up as far as you can, then back
down to this application, then run it with these
options."

Windows would allow the code execution, and
presto, the machine was compromised.

CodeRed exploited this flaw to replicate itself all over the Net within a day.

I have no idea why Sherlock, or any other
application, would need to perform upward
directory paths in a URL. While the Windows/IIS
exploits have long since been blocked, your
netsec guy is correct in denying that sort of
behavior. There should be no legitimate reason
that I know of for using that string in a URL.

Regards,
--

Chuck Goolsbee V.P. Technical Operations
_________________________________________________________________
digital.forest Phone: +1-877-720-0483, x2001
where Internet solutions grow Int'l: +1-425-483-0483
**** celebrating ten years of service 7/12/1994 - 7/12/2004 ****
12101 Tukwila International Blvd Fax: +1-425-482-6871
Suite 410 http://www.forest.net
Seattle, WA 98168 email: cgforest.net

Quoting dano <danowell.com>:

> Checked with the firewall admin and he discovered that the firewall -
> called SmartDesign or something like that - is blocking Sherlock
> because it sends/receives a URL with /../ and this is interpreted as
> a malformed URL string and an attack on Microsoft IIS.

It can be an attack on more than IIS, it can be an attack on Apache too
(although I believe it's been blocked in the default config of Apache
for quite
some time.) It's called a directory transversal attack and it's a way
of gaining
access to directories outside of the web server's immediate web page
directories.

In windows, and unix (and OS X) a .. means the directory above. On Mac
if you go
to terminal you start in your home directory, if you type:
cd ..
you'll move one directory up to the /Users directory.

Typing:
cd ../../..

would move you back 3 directories.

On some web servers, if you connect to http://www.tidbits.com/../ (that
doesn't
work btw) it would show you the directories above the directory where the web
pages are stored.

The attack on most IIS boxes consists of going all the way back to the root
directory by passing hundreds of ../../ then /Windows/System32/cmd.exe
then the
command line you want to run on the server. Most IIS installations run as the
Local System account (almost root level privs) so being able to run a command
via the web server is very, very dangerous.

Most of these vulnerabilities have been patched on IIS servers, but
many admins
don't patch as frequently as they should so there are still vulnerable
machines
around.

I'm surprised your outbound firewall blocks these. It would prevent locally
infected machines from infecting others outside the firewall, but it's also a
fairly common method of transversing directories inside a web site. It isn't
dangerous until you're allowed to transverse a directory outside the
normal web
directories.

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

Quoting chuck goolsbee <chucklistforest.net>:

> I have no idea why Sherlock, or any other
> application, would need to perform upward
> directory paths in a URL. While the Windows/IIS
> exploits have long since been blocked, your
> netsec guy is correct in denying that sort of
> behavior. There should be no legitimate reason
> that I know of for using that string in a URL.

I've used it with relative URL's to avoid hard-coding urls to a particular
domain name. For example, if all my images are in an images directory off the
root of the web server, and my pages are in another directory, I might use
"../images/image.jpg" for the IMG SRC URL. If I change my domain name
(something I actually do frequently on my personal web site) I don't have to
worry about finding all the references to the old domain as long as the
directory structure is constant.

Kevin

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

At 5:03 PM -0600 1/26/05, kevinvanhaaren.net wrote:
>Quoting chuck goolsbee <chucklistforest.net>:
>
>>I have no idea why Sherlock, or any other
>>application, would need to perform upward
>>directory paths in a URL. While the Windows/IIS
>>exploits have long since been blocked, your
>>netsec guy is correct in denying that sort of
>>behavior. There should be no legitimate reason
>>that I know of for using that string in a URL.
>
>I've used it with relative URL's to avoid hard-coding urls to a particular
>domain name. For example, if all my images are in an images directory off the
>root of the web server, and my pages are in another directory, I might use
>"../images/image.jpg" for the IMG SRC URL. If I change my domain name
>(something I actually do frequently on my personal web site) I don't have to
>worry about finding all the references to the old domain as long as the
>directory structure is constant.

ah... Murphy's Law. Or in this case Douglas Adams' Law.

The assumption is that web roots are hard chroot'ed and inescapable.
If you look at it from the perspective of a shared hosting provider,
or a server owner, allowing an anonymous web user to navigate above
and out of a chrooted directory has HUGE security implications.

In your example you are not really chrooted, but just serving
different sites out of a subdirectory. That of course could be
remedied at the URL level by using a full path, rather than a
relative one. The relative path would have to have the /../, whereras
the full one would/could not.

If you do this, it is like serving off an odd port, you have to live
with some percentage of the Internet's users to NOT be able to view
your site.

And Adam, ever figure out why some of us have our posts to your lists
get that weird line wrap?
--

Chuck Goolsbee V.P. Technical Operations
_________________________________________________________________
digital.forest Phone: +1-877-720-0483, x2001
where Internet solutions grow Int'l: +1-425-483-0483
**** celebrating ten years of service 7/12/1994 - 7/12/2004 ****
12101 Tukwila International Blvd Fax: +1-425-482-6871
Suite 410 http://www.forest.net
Seattle, WA 98168 email: cgforest.net

At 10:39 -0800 1/26/05, chuck goolsbee (speaking about directory traversal in Watson/Sherlock URLs) opined:
>your
>netsec guy is correct in denying that sort of
>behavior. There should be no legitimate reason
>that I know of for using that string in a URL.

One legitimate reason is that the standard requires it:

http://www.ietf.org/rfc/rfc1738.txt

Another is that it is a useful shortcut to creating web documents that use navigation elements that are inherited from a parent directory, even though there might be multiple paths for navigating to a given page. I use this a lot at my site, though anymore I think I have Apache rewriting those URLs before the client browser sees them.

A third is that HTTP is a legitimate protocol for browsing directories and downloading files, both functions that traditionally employ directory traversal paths.

--Ron
--
Ron Risley
www.risley.net

Good evening,

On 26/1/05 at 10:39 AM -0800, chuck goolsbee <chucklistforest.net> wrote:

>I have no idea why Sherlock, or any other
>application, would need to perform upward
>directory paths in a URL. While the Windows/IIS
>exploits have long since been blocked, your
>netsec guy is correct in denying that sort of
>behavior. There should be no legitimate reason
>that I know of for using that string in a URL.

Plenty of web sites use the parent "../" directory traversal to reference
content on pages such as images and style sheets. So there are legitimate
reasons for using the "../" string in URLs.

And I was not aware that some firewalls blocked such requests. (Something for
me to ponder for all my existing websites.) But hopefully that only applies to
firewalls protecting Windows servers rather than firewalls blocking legitimate
outgoing connections.

It's a shame that Microsoft's sloppy programming methods have caused sys
admins to block requests for URLs that (rightly) use a long-standing and
useful feature.

Charlie

--
   Charlie Garrison <garrisonzeta.org.au>
   PO Box 141, Windsor, NSW 2756, Australia

That's good HTML coding practice. However, I think the relative path
is actually translated long before it leaves your machine. I believe
that even though the HTML reads "../images/mylogo.gif", your browser
will translate it and send a file request for the absolute path based
on the path of the page containing the URL. So no "../" components
are sent in the HTTP request in these cases.

I'm not sure why Sherlock is sending them. However, how the rest of
the URL after the domain specifier is translated is up to the server
software. They may be encoding some other information in there for
their own purposes. (Normally that's handled differently though.)
It would be interesting to know the true story!

Dave

>I've used it with relative URL's to avoid hard-coding urls to a particular
>domain name. For example, if all my images are in an images directory off the
>root of the web server, and my pages are in another directory, I might use
>"../images/image.jpg" for the IMG SRC URL. If I change my domain name
>(something I actually do frequently on my personal web site) I don't have to
>worry about finding all the references to the old domain as long as the
>directory structure is constant.

> -----Original Message-----
> From: chuck goolsbee [mailto:chucklistforest.net]
> Sent: Thursday, January 27, 2005 12:38 PM
> To: tidbits-talktidbits.com
> Cc: tidbits-talktidbits.com; dano
> Subject: Re: Remember Watson?
>
> [snip]
>
> If you do this, it is like serving off an odd port, you have to live
> with some percentage of the Internet's users to NOT be able to view
> your site.

Chuck, are you saying "../" should be avoided entirely in URLs because of
the assumptions being made by some security software? Kevin could take care
of his every-changing domain name problem and avoid "../" by using
"/parent/images/image.jpg" but there would still be a problem I experience
when editing web pages.

I don't know about commercial editors like Dreamweaver but in Mozilla
Composer (and it's successor, Nvu) if you use a full path without the domain
name, it can't find the image file (or other file, like .css) referenced. If
I try to use "/style.css" as the location of a stylesheet, the editor can't
find it but if I use "../style.css" (assuming I'm editing a page one level
down from the top), it can.

At 9:37 AM -0800 1/27/05, kevinvanhaaren.net wrote:
>Quoting chuck goolsbee <chucklistforest.net>:
>>There should be no legitimate reason
>> that I know of for using that string in a URL.
>
>I've used it with relative URL's to avoid hard-coding urls to a particular
>domain name. For example, if all my images are in an images directory off the
>root of the web server, and my pages are in another directory, I might use
>"../images/image.jpg" for the IMG SRC URL. If I change my domain name
>(something I actually do frequently on my personal web site)

A prefer just to build the pages out of the root for the server (i.e.
"/images/image.jpg", skipping the "../") Which means I can set
templates up for pages (I'm an SSI addict of the 1st degree.) without
having to worry about where the actual page is located. This
solution still is portable between domains. (except if you get stuck
in http://www.somedomain.com/~someusername and have moved from
http://www.yourdomain.com or vice versa.)

Nick

At 6:31 PM -0800 1/26/05, Ron Risley wrote:
>One legitimate reason is that the standard requires it:
>
>http://www.ietf.org/rfc/rfc1738.txt

Wow... I hadn't read that one in at least 10 years, but a quick
cruise though it today did NOT show any reference to upward directory
traversal with /../ in HTTP, FTP, or even the FILE section.

It was however, an entertaining walk down memory lane... the largest
section by far is for gopher. =)

The "security considerations" section does obliquely suggest the
possibility of harmful URLs:

" A URL-related security threat is that it is sometimes possible to
    construct a URL such that an attempt to perform a harmless idempotent (sic)
    operation such as the retrieval of the object will in fact cause a
    possibly damaging remote operation to occur."

I would suggest that UPWARD directory traversal, especially outside
of the supposedly isolated web root directory would constitute such a
security threat.

>A third is that HTTP is a legitimate protocol for browsing
>directories and downloading files, both functions that traditionally
>employ directory traversal paths.

Agreed... in an ALREADY established session. The issue we are
misunderstanding between us is the time-frame of the request. An
inbound packet going to a server with an upward directory traversal
request in the INITIAL request is probably up to no good.

At 5:17 PM -0500 1/27/05, Wilcox, Curtis wrote:
>Chuck, are you saying "../" should be avoided entirely in URLs because of
>the assumptions being made by some security software?

Given the fact that this URL string (*/../*) is a pattern match for
at least two of the largest Windows Worms (I hate it when the press
calls them "Internet Worms"... The Internet was just the transit,
Windows was both the source and destination) I'm saying that your
likelihood for being blocked at a firewall is quite high. That is all.

--

Chuck Goolsbee V.P. Technical Operations
_________________________________________________________________
digital.forest Phone: +1-877-720-0483, x2001
where Internet solutions grow Int'l: +1-425-483-0483
**** celebrating ten years of service 7/12/1994 - 7/12/2004 ****
12101 Tukwila International Blvd Fax: +1-425-482-6871
Suite 410 http://www.forest.net
Seattle, WA 98168 email: cgforest.net

At 9:37 AM -0800 2005/01/27, Charlie Garrison wrote:
>Plenty of web sites use the parent "../" directory traversal to reference
>content on pages such as images and style sheets. So there are legitimate
>reasons for using the "../" string in URLs.
>
>And I was not aware that some firewalls blocked such requests. (Something for
>me to ponder for all my existing websites.) But hopefully that only applies to
>firewalls protecting Windows servers rather than firewalls blocking legitimate
>outgoing connections.

        I think that most web broswers (or is it the server that does it?)
take the relative URL encoded in an HTML document, and then request the
full URL when the link is clicked, rather than just sending the relative
URL part, so a firewall should never see the "../" string as part of a URL
and thus block it.

        I usually put a default index.html file in every directory of a
website with a redirect to "../", so that if someone accidentally gets to
the wrong place (image directories for example) they just get bounced up
and up through the structure until they hit a level that has an index.html
with real content. Not really robust security, but useful none-the-less.

At 12:27 PM -0800 1/28/05, Johann Beda wrote:
> I think that most web broswers (or is it the server that does it?)
>take the relative URL encoded in an HTML document, and then request the
>full URL when the link is clicked, rather than just sending the relative

It has to be done in the browser; HTTP requires an absolute path in
the request line. (I worked on a simple web server in a recent
project, and I hat reject requests that did not start with "/".)

Using "../" is OK in URLs on web pages, but the original comment was
about blocking this substring in an HTTP request. I did a quick test
in Safari with a URL containing a couple of instances of "../" and
the path traversal was resolved before the request was transmitted to
the server.

--
Larry Rosenstein
lsralum.mit.edu

Adam: I've wandered completely off topic. Feel free to hit [delete].

At 12:27 -0800 1/28/05, chuck goolsbee opined:
>At 6:31 PM -0800 1/26/05, Ron Risley wrote:
>>One legitimate reason is that the standard requires it:
>>
>>http://www.ietf.org/rfc/rfc1738.txt
>
>Wow... I hadn't read that one in at least 10 years, but a quick
>cruise though it today did NOT show any reference to upward directory
>traversal with /../ in HTTP, FTP, or even the FILE section.

Admittedly, it's buried in some assumptions, for example:

>3.2.2. FTP url-path
>
> The url-path of a FTP URL has the following syntax:
>
> <cwd1>/<cwd2>/.../<cwdN>/<name>;type=<typecode>
>
[deletia]
> The url-path is interpreted as a series of FTP commands as follows:
>
> Each of the <cwd> elements is to be supplied, sequentially, as the
> argument to a CWD (change working directory) command.

Since FTP:CWD would handle the /../ syntax correctly, presumably the intent was that interpretation of URLs would proceed along the same lines.

>An
>inbound packet going to a server with an upward directory traversal
>request in the INITIAL request is probably up to no good.

Agreed. But how many firewalls are stateful enough to know what is an initial request vs. an already established session?

In purely pragmatic terms, I think it probably represents a reasonable trade-off, and certainly did if it was all you had to block that malware when it was at its peak (though */../../* might work as well with a lot less collateral damage). It just strikes a nerve with me (and apparently a few others here) to be blocking a legitimate and potentially useful construct because one (big) player in the game fails to behave responsibly.

--Ron
--
Ron Risley
www.risley.net

At 9:02 PM -0800 1/28/05, Ron Risley wrote:
> > <cwd1>/<cwd2>/.../<cwdN>/<name>;type=<typecode>

I believe the ellipsis is there for "stuff left out" not a specific
reference to directory "dots."

>Since FTP:CWD would handle the /../ syntax correctly, presumably the
>intent was that interpretation of URLs would proceed along the same
>lines.

Well, given that in the above example (if it were not an ellipsis!)
"/<cwd2>/../" would basically be a recursive loop, I don't think so.
I guess the only *certain* way to clarify intent would be to ask the
author. Anyone have Tim Berners-Lee's current email address?

> >An
>>inbound packet going to a server with an upward directory traversal
>>request in the INITIAL request is probably up to no good.
>
>Agreed. But how many firewalls are stateful enough to know what is
>an initial request vs. an already established session?

Every single one of them.

If it is stateful enough to block based on the *content* of URL, then
it is by definition stateful enough to understand what packet is the
first in a conversation. That is how stateful firewalls work. That is
the exact definition of a "stateful" firewall.

>In purely pragmatic terms, I think it probably represents a
>reasonable trade-off, and certainly did if it was all you had to
>block that malware when it was at its peak (though */../../* might
>work as well with a lot less collateral damage). It just strikes a
>nerve with me (and apparently a few others here) to be blocking a
>legitimate and potentially useful construct because one (big) player
>in the game fails to behave responsibly.

I agree with the latter, but I'll stand by my first statement that
any INITIAL request asking to traverse upward in a directory tree has
NO valid use, and therefore should not be allowed.

Subsequent requests in an already permitted session? Fine. Provided
the chroot boundaries are respected and maintained.

Regards,
--

Chuck Goolsbee V.P. Technical Operations
_________________________________________________________________
digital.forest Phone: +1-877-720-0483, x2001
where Internet solutions grow Int'l: +1-425-483-0483
**** celebrating ten years of service 7/12/1994 - 7/12/2004 ****
12101 Tukwila International Blvd Fax: +1-425-482-6871
Suite 410 http://www.forest.net
Seattle, WA 98168 email: cgforest.net

[This thread has so completely gotten off-topic that I'm just going to leave it there, but let's try to wind it down. -Adam]

Quoting chuck goolsbee <chucklistforest.net>:

> Wow... I hadn't read that one in at least 10 years, but a quick
> cruise though it today did NOT show any reference to upward directory
> traversal with /../ in HTTP, FTP, or even the FILE section.
>

That RFC touches on relative links:
In some cases, those pointers are represented as relative links where the
expression of the location of the second resource is in terms of "in the same
place as this one except with the following relative path". Relative links are
not described in this document.

But actual relative linking is covered in RFC 1808.

<http://www.faqs.org/rfcs/rfc1808.html>

> I would suggest that UPWARD directory traversal, especially outside
> of the supposedly isolated web root directory would constitute such a
> security threat.

I agree, but an outbound firewall rule, which is how this dicussion started,
can't know when an upward transversal exceeds the bounds of the web root
directory. That is something an inward bound firewall MIGHT know (but usually
doesn't) and which the web server must know.

The relative linking RFC even points this out:
"Parsers must be careful in handling the case where there are more
relative path
".." segments than there are hierarchical levels in the base URL's path. Note
that the ".." syntax cannot be used to change the <net_loc> of a URL."

Using an outbound firewall rule to protect other's buggy servers will most
likely block access to legitimate servers too. Especially a rule based on a
single .. instance. Blocking URLs with 10 or 20 would be more understandable,
but not one or even 5.

Kevin

Quoting chuck goolsbee <chucklistforest.net>:
> Subsequent requests in an already permitted session? Fine. Provided
> the chroot boundaries are respected and maintained.

I don't quite understand this. My understanding of a stateful firewall
is that
the state is per TCP session. For web pages that session is terminated when
the currently requested page has been sent, and at that point a stateful
firewall would discard the state information. When a user clicks on another
URL within that site a new TCP session is built from scratch and the stateful
firewall remembers nothing of the previous TCP session (especially not the
entire URL previously used.)

Quoting Larry Rosenstein <lsralum.mit.edu>:
> Using "../" is OK in URLs on web pages, but the original comment was
> about blocking this substring in an HTTP request. I did a quick test
> in Safari with a URL containing a couple of instances of "../" and
> the path traversal was resolved before the request was transmitted to
> the server.

I expanded this test and FireFox 1 and Internet Explorer 6 on Windows, and w3m
and lynx on Linux all correctly do the resolution in the browser and resolve
the ..'s on the client side and request the full appropriate url from the
server.

I get the feeling that earlier browsers and tools (my lynx install is
the latest
version not one of the old versions), such as sherlock and watson, originally
just passed the ..'s to the server but due to abuse most now resolve on the
client side.

So a firewall that blocks URLs containing ".." would be appropriate as long as
all your clients are of a recent vintage.

BTW, checking my web server logs I see the standard for exploits is to now mix
URL encoding of characters, probably to bypass firewall rules and URL
protection tools like Microsoft's UrlScan (i believe the current version
resolves HTML encoded characters but older ones did not.)

<http://www.microsoft.com/technet/security/tools/urlscan.mspx>

Some of the URL's I'm seeing are:
/scripts/..%252f../
/scripts/..%c1%1c../

and lots more, including some that are surprising that they ever worked:
/c/winnt/system32/cmd.exe
/d/winnt/system32/cmd.exe