Filesystem metadata approaches

On Nov 9, 2006, at 2:17 PM, John C. Welch wrote:

>> UTIs exist in 10.4 but can not be set for a file. The recommended
>> method is for an application developer to set type information via
>> the filename extension, which the OS will then translate into a UTI
>> (which should be used when making typing decisions).
>
> depends on how you mean set.

I think it's pretty clear. There is no way, given a file, to change
its UTI to a particular value. UTIs are a layer on top of the other
types of information (extension, type code). They make dealing with
these much easier for developers, but they don't store any additional
information with the file. I'm not saying that's good or bad; that's
just how it is.

> Apple even has well defined functions for getting UTIs to and from
> OSTypes, (file Type/creator
> codes), namely UTCreateStringForOSType and UTGetOSTypeFromString.

Those simply encode OSTypes as strings for use with the UT APIs. They
don't do any mapping between type codes and UTIs. There are other
functions to do that.

> The ability to set UTI information for a file to a specific
> application is
> definitely possible as well.

I'm not sure what that means. The file has an extension and a type
code, and you can change those. Applications can define new UTIs and
associate them with extensions, type codes, and MIME types.
Applications can also declare which UTIs they know how to read. There
is an API to set which application will open files with a particular
UTI, and you can set whether creator codes should be ignored for
files with that UTI. But there is no way to set the UTI of a
particular file except indirectly by changing the file type or
extension, which will change the inferred UTI.

If you want a particular file to open in a particular application,
you can set the creator code or change the binding in Get Info, but
this doesn't really have anything to do with UTIs.

--Michael

At 11:17 on 09-11-2006, John C. Welch wrote:
> How is having 4 ways to correctly identify a file more limiting than 1?

I'm not talking about what is technically possible with OS X. I am
talking about the reality of the situation. It may be because of
application developers' implementations that we have such a reliance
on filename extensions, but these implementations are a direct result
of Apple's recommendations. No point in splitting hairs.

>> OS 9 did not restrict me in 'pretty much the same ways.' There was
>> one character that I couldn't use in my filenames -- the colon (':').
>
> and periods. The Finder was all kinds of unhappy if you started a name with
> a period, as that was a way to identify driver files.

Yes, sort of. I had filenames starting with a period and had no
problems. But you did have to be careful -- naming a file '.sony' was
not a good idea. :-)

_________________
=> Jolin Warren, Edinburgh, Scotland

On 11/13/06 19:48, "Jolin M Warren" <JolinWarrenOakAndApple.org> wrote:

>> How is having 4 ways to correctly identify a file more limiting than 1?
>
> I'm not talking about what is technically possible with OS X. I am
> talking about the reality of the situation. It may be because of
> application developers' implementations that we have such a reliance
> on filename extensions, but these implementations are a direct result
> of Apple's recommendations. No point in splitting hairs.

Well, in that case, there was only one way to identify a file in OS 9, and
you had no OS - driven way to change it, so if it was wrong or corrupt, you
were hosed or rather put out.

Sure, there were actually ways to deal with that, but they're all just
"splitting hairs", right? Just "technically possible".


Apple provides multiple ways to deal with this, but they *cannot* force an
ISV to do everything the right way. If the ISV refuses to do anything but
file name extensions to bind docs to their applications, Apple can't do much
about that.

However, the *reality* of the situation is, and this isn't splitting hairs,
if you want to ensure that your JPEG file is recognized as a JPEG on 99% of
platforms out there without readmes or metadata prayers, you're going to jam
a .jpg or a .jpeg on the end of that file name. Because while MIME types can
be set wrong or even changed by a server, and if it's not a mac, file types
and creator codes are useless, names of files are usually left alone.

Like it or not, Apple recommending the use of filename extensions is a smart
way to ensure that interoperability is smoother. You can stand there and
proclaim the innate superiority of File Types and creator codes, (which is
really amusing considering how unreadable many of them were, or how bizarre
they could get), but if the person you're sending the file to has to keep
asking you "what kind of file is this" because you refuse to use filename
extensions, that's not going to make your life easier. Or theirs

>
>>> OS 9 did not restrict me in 'pretty much the same ways.' There was
>>> one character that I couldn't use in my filenames -- the colon (':').
>>
>> and periods. The Finder was all kinds of unhappy if you started a name with
>> a period, as that was a way to identify driver files.
>
> Yes, sort of. I had filenames starting with a period and had no
> problems. But you did have to be careful -- naming a file '.sony' was
> not a good idea. :-)

That was never reliable. The point is, to insist that the OS 9 finder had no
restrictions on file names is incorrect. As well, I can't think of anyone
who misses 31 Character file names.

--
John C. Welch Writer/Analyst
Bynkii.com Mac and other opinions
jwelchbynkii.com

>At 11:17 on 09-11-2006, John C. Welch wrote:
>Yes, sort of. I had filenames starting with a period and had no
>problems. But you did have to be careful -- naming a file '.sony' was
>not a good idea. :-)

In grooming Macs I was recycling, Norton Disc Doctor frequently found
files beginning with a period and recommended renaming them. I
complied.

Richard

On Nov 13, 2006, at 5:48 PM, Jolin M Warren wrote:

> At 11:17 on 09-11-2006, John C. Welch wrote:
>
>>> OS 9 did not restrict me in 'pretty much the same ways.' There was
>>> one character that I couldn't use in my filenames -- the colon
>>> (':').
>>
>> and periods. The Finder was all kinds of unhappy if you started a
>> name with
>> a period, as that was a way to identify driver files.
>
> Yes, sort of. I had filenames starting with a period and had no
> problems. But you did have to be careful -- naming a file '.sony' was
> not a good idea. :-)

Had you started with the Mac earlier (eg, January 1984) than you seem
to have, you would have learned quickly to avoid leading "." in file
names. The system became more forgiving over time.

The day Apple announced A/UX, I stopped using / in file names (and
then never ran A/UX, and no one with whom I exchanged files did,
either). That is despite the fact that one CAN use / in a Unix file
name; it's just a great pain to do so.

   --John

On Nov 14, 2006, at 12:37 PM, John C. Welch wrote:

> That was never reliable. The point is, to insist that the OS 9
> finder had no
> restrictions on file names is incorrect. As well, I can't think of
> anyone who misses 31 Character file names.

That, of course, was the "new" Finder. 253 character file names were
"nice" (1984). (And only worked on volumes with 1-character names.*)

The same sort people miss "our" 31-character names as miss the
(visible) 8.3 filenames from Bellevue (Microsoft moved to Redmond
later).

The whole virtual folder thing in the 1984 filesystem was
"interesting."**

So was the ability to crash some versions of Finder by feeding Finder
floppies with filenames longer than 63 characters.
And the "Icon<cr>" special case was amusing, as well.

Meanwhile, Unix file naming is quite similar now to what it was well
before Macintosh was a gleam on Jobs' eye. ;-)

   --John

*The volume:filename string had to fit into a Pascal string, hence
max 253-characters for the file name.
**Some would say "bizarre" is a better word.

--On 11/14/2006 12:46 PM -0800 johnbaxterlistsmac.com wrote:

> Had you started with the Mac earlier (eg, January 1984) than you seem
> to have, you would have learned quickly to avoid leading "." in file
> names. The system became more forgiving over time.

In at least one instance of cross-platform software--TeXtures, the Mac OS
implementation of the TeX/LaTeX language--there were situations where the
software required dot-files for consistency. Norton always used to
complain about them, and I told it to ignore the "problem".

Dave Scocca

On 15/11/2006 9:46 AM, "johnbaxterlistsmac.com" <johnbaxterlistsmac.com>
spake thus:

> That is despite the fact that one CAN use / in a Unix file
> name; it's just a great pain to do so.

While Unix will allow you to use any character (and I mean *any* character)
in a file name, some can cause very strange problems under the right
circumstances. The weirdest problem I ever had with a Unix filename was one
that I somehow inadvertently named "-something", i.e., with a leading
hyphen. Man, the contortions I had to go through to fix that :)

I realise that some may not understand why this was such a problem. First,
this was on a GUI-less system, so I couldn't just rename the file that way.
Second, Unix generally treats anything in a command line string with a
leading hyphen as a command option. Thus, "rm -something" gives you
something like "rm: illegal option". Same problem with mv for renaming. The
usual tricks of pre-pending a backslash or wrapping it in quotes don't work
either. I know I eventually fixed it, but can't for the life of me remember
how I did it. For that matter, I'm not even sure how I managed to name it
that way in the first place...

--
Nigel Stanger, Dunedin, NEW ZEALAND.
http://xri.net/=nigel.stanger

On Nov 14, 2006, at 2:46 PM, johnbaxterlistsmac.com wrote:

> Had you started with the Mac earlier (eg, January 1984) than you seem
> to have, you would have learned quickly to avoid leading "." in file
> names. The system became more forgiving over time.
>
> The day Apple announced A/UX, I stopped using / in file names (and
> then never ran A/UX, and no one with whom I exchanged files did,
> either). That is despite the fact that one CAN use / in a Unix file
> name; it's just a great pain to do so.

When I was a NeXT user I learned not to use ">" in a file name, either.

See "cat foo > bar" for one of many reasons why not...

At 12:37 on 14-11-2006, John C. Welch wrote:
> Well, in that case, there was only one way to identify a file in OS 9, and
> you had no OS - driven way to change it, so if it was wrong or corrupt, you
> were hosed or rather put out.
>
> Sure, there were actually ways to deal with that, but they're all just
> "splitting hairs", right? Just "technically possible".

Yes, I agree. This was a big failing of the Classic Mac OS's metadata
system. (AppleScript was actually an OS-driven way to change
type/creator codes, but it is sufficiently obtuse for the average
user that using this as an example of allowing modification of the
metadata is just 'splitting hairs', as you say.) Type and creator
codes (and especially their implementation) were certainly far from
perfect. I'm just claiming that, on the whole, they are superior to
the current situation of filename extensions as the only 'required'
(by Apple) typing metadata.

> Like it or not, Apple recommending the use of filename extensions is a smart
> way to ensure that interoperability is smoother.

It is one way to ensure inter-operability, but I would not consider
it a 'smart' way. Under Classic Mac OS, all applications that
potentially handled moving files between OSes/filesystems were
supposed to map between types/creators and filename extensions/MIME.
So, for instance, the Finder would read filename extensions on a DOS
disk and add the appropriate type/creator codes. Email clients would
read type/creator codes of attachments and add appropriate filename
extensions/MIME types when the email was sent (and OS 9-era email
clients like Eudora and Entourage still do this). In this way,
recipients on other systems always had the metadata needed by their
system and they didn't have to ask what kind of file they'd been sent.

So a 'smart' way to ensure inter-operability is to make sure that all
entry and exit points for a file from the system do the appropriate
metadata translations. This is not too hard as there are relatively
few entry and exit points, and the experience of the Classic Mac OS
showed that when the operating system provided the mapping table and
Apple provided the appropriate recommendations, application
developers complied with this strategy. A 'smart' way to ensure
inter-operability is not to simply cater to the lowest common
denominator (though that might be effective by certain measures).

Now the tools to manage an effective metadata system are another
issue. As mentioned above, this was a big weakness in the Classic Mac
OS. The mapping tables were editable, but it was always quite clumsy.

> The point is, to insist that the OS 9 finder had no restrictions on
> file names is incorrect.

I'd be interested to see where anyone on this list stated that the OS
9 Finder had no restrictions on file names. I just object to being
forced to end my filenames with arbitrary (to my naming system)
strings of characters. The limitations on what characters I could or
couldn't use in filenames were less in OS 9. Note that colons and
starting with a full stop are still not possible in the OS X Finder
(well, the later is possible but the file will disappear, another
irritating example of embedding metadata in a filename).

> As well, I can't think of anyone who misses 31 Character file names.

I certainly don't miss the 31 character limit. But I don't see why an
improvement in one area justifies retrograde steps in another.

_________________
=> Jolin Warren, Edinburgh, Scotland

On 17.11.2006, at 15:42, Nigel Stanger wrote:
> While Unix will allow you to use any character (and I mean *any*
> character)
> in a file name, some can cause very strange problems under the right
> circumstances. The weirdest problem I ever had with a Unix filename
> was one
> that I somehow inadvertently named "-something", i.e., with a leading
> hyphen. Man, the contortions I had to go through to fix that :)

Like

$ rm -- -something

? :-)

> I realise that some may not understand why this was such a problem.
> First,
> this was on a GUI-less system, so I couldn't just rename the file
> that way.
> Second, Unix generally treats anything in a command line string with a
> leading hyphen as a command option. Thus, "rm -something" gives you
> something like "rm: illegal option". Same problem with mv for
> renaming. The
> usual tricks of pre-pending a backslash or wrapping it in quotes
> don't work
> either. I know I eventually fixed it, but can't for the life of me
> remember
> how I did it. For that matter, I'm not even sure how I managed to
> name it
> that way in the first place...

Like

$ touch -- -something

? B-)

Both work at least in the OS 10.4.8 built-in tcsh.

Cheers, Kei.

On Nov 17, 2006, at 9:42 AM, Nigel Stanger wrote:
> On 15/11/2006 9:46 AM, "johnbaxterlistsmac.com"
> <johnbaxterlistsmac.com>
> spake thus:
>
>> That is despite the fact that one CAN use / in a Unix file
>> name; it's just a great pain to do so.
>
> While Unix will allow you to use any character (and I mean *any*
> character)

Actually, this depends on the version of Unix. Not every Unix
implementation supports a slash in the filename. SunOS 4, for
instance, does not, since the slash is reserved for separating
components (directory names and file names) of a path. There may be
other specific characters which were also not allowed in filenames,
though I don't remember, and don't currently have access to a SunOS 4
system.

> in a file name, some can cause very strange problems under the right
> circumstances. The weirdest problem I ever had with a Unix filename
> was one
> that I somehow inadvertently named "-something", i.e., with a leading
> hyphen. Man, the contortions I had to go through to fix that :)

> I realise that some may not understand why this was such a problem.
> First,
> this was on a GUI-less system, so I couldn't just rename the file
> that way.
> Second, Unix generally treats anything in a command line string with a
> leading hyphen as a command option. Thus, "rm -something" gives you
> something like "rm: illegal option". Same problem with mv for
> renaming. The
> usual tricks of pre-pending a backslash or wrapping it in quotes
> don't work
> either. I know I eventually fixed it, but can't for the life of me
> remember
> how I did it. For that matter, I'm not even sure how I managed to
> name it
> that way in the first place...

Unlike the "rm" command in SunOS 4, the one we have in MacOS is the
GNU version, and takes the "--" flag, which indicates an end to the
options, so you can say "rm -- -something" to remove such a file.
Most modern implementations have this feature, but SunOS 4 is rather
old, and not compliant with modern standards. If you have a
metacharacter in the filename, then quoting it is necessary. If you
have an invisible character, such as a control character, in the
filename, then using a question mark as a wildcard can allow you to
delete it. In SunOS 4 there was either a "remove" or "unlink"
command (sorry, I don't remember which it was) which doesn't take
options, so it allowed me to remove the "-something" files, though
the easiest way was through Emacs (the editor), rather than the
command-line.

--
Geoff Odhner

.

>The weirdest problem I ever had with a Unix filename was one
>that I somehow inadvertently named "-something", i.e., with a leading
>hyphen. Man, the contortions I had to go through to fix that :)

$ mv ./-something somethingelse

or

$ rm ./-something

if you just want to remove it.

Brian

On 11/17/06 09:02, "Jolin M Warren" <JolinWarrenOakAndApple.org> wrote:

> It is one way to ensure inter-operability, but I would not consider
> it a 'smart' way. Under Classic Mac OS, all applications that
> potentially handled moving files between OSes/filesystems were
> supposed to map between types/creators and filename extensions/MIME.
> So, for instance, the Finder would read filename extensions on a DOS
> disk and add the appropriate type/creator codes. Email clients would
> read type/creator codes of attachments and add appropriate filename
> extensions/MIME types when the email was sent (and OS 9-era email
> clients like Eudora and Entourage still do this). In this way,
> recipients on other systems always had the metadata needed by their
> system and they didn't have to ask what kind of file they'd been sent.

The OS 9 Finder was less than reliable about this, and because of the
limitations in the FT/CC system, was less than stellar about getting
associations right based on extension if you had multiple applications that
could handle a given file type. Of course, in OS 9, fixing that problem
required *far* more work than in OS X. At least with OS X, fixing an
incorrect association is about ten seconds or so.

>
> So a 'smart' way to ensure inter-operability is to make sure that all
> entry and exit points for a file from the system do the appropriate
> metadata translations. This is not too hard as there are relatively
> few entry and exit points, and the experience of the Classic Mac OS
> showed that when the operating system provided the mapping table and
> Apple provided the appropriate recommendations, application
> developers complied with this strategy. A 'smart' way to ensure
> inter-operability is not to simply cater to the lowest common
> denominator (though that might be effective by certain measures).

Um...no. When the Finder starts adding metadata on its own that is tied to
the file, that is modifying the file. This is not generally a good thing.
Letting the Finder keep it's on associations, fine. Arbitrarily chaning
every file in a directory on a network share because I needed to access
*one* file? Yeah...no.

As well, the lowest common denominator is call that because it is also the
most *reliable* way of identifying a file's contents.

>
> Now the tools to manage an effective metadata system are another
> issue. As mentioned above, this was a big weakness in the Classic Mac
> OS. The mapping tables were editable, but it was always quite clumsy.

And without third party utilities, or some basic programming knowledge,
effectively impossible. That's hardly a "better" system.

>
>> The point is, to insist that the OS 9 finder had no restrictions on
>> file names is incorrect.
>
> I'd be interested to see where anyone on this list stated that the OS
> 9 Finder had no restrictions on file names. I just object to being
> forced to end my filenames with arbitrary (to my naming system)
> strings of characters.

I don't use extensions quite a lot, and don't really have a lot of problems.
Barely any actually, quite a bit less than I had on OS 9.

> The limitations on what characters I could or
> couldn't use in filenames were less in OS 9. Note that colons and
> starting with a full stop are still not possible in the OS X Finder
> (well, the later is possible but the file will disappear, another
> irritating example of embedding metadata in a filename).

OS 9 didn't have a tenth the scope of operational requirements as OS X.
WOuld you like to yank out all the features in OS X that OS 9 didn't have in
the way of networking and interoperability?

>
>> As well, I can't think of anyone who misses 31 Character file names.
>
> I certainly don't miss the 31 character limit. But I don't see why an
> improvement in one area justifies retrograde steps in another.

Sorry, we're doomed to disagree. I have yet to see proof of OS 9's fabled
superiority in handling file bindings in anything other than theoretical
discussion. In the real world, it wasn't nearly as good as people remember.

--
John C. Welch Writer/Analyst
Bynkii.com Mac and other opinions
jwelchbynkii.com

On 11/17/2006 7:02 AM, "Jolin M Warren" wrote:
> So, for instance, the Finder would read filename extensions on a DOS
> disk and add the appropriate type/creator codes.

I don't recall this being the case. Thanks to File Exchange, OS 9 could
often figure out which application should open which Windows filename
extension(s), but I don't recall the Finder actually *adding* type/creator
codes automatically. Am I mis-remembering? ;-)

On Nov 17, 2006, at 11:55 AM, John C. Welch wrote:

> The OS 9 Finder was less than reliable about this, and because of the
> limitations in the FT/CC system, was less than stellar about getting
> associations right based on extension if you had multiple
> applications that
> could handle a given file type. Of course, in OS 9, fixing that
> problem
> required *far* more work than in OS X. At least with OS X, fixing an
> incorrect association is about ten seconds or so.

One of the more interesting problems with the old Mac method exposed
itself each time I got a new version of MPW (Macintosh Programmers
Workshop) on CD. The obvious way to read the ReadMe file was to put
the CD into the drive, and double click the ReadMe.

Wrong.

The ReadMe was an MPW text file. The rules for deciding WHICH
instance of MPW would open the file (or any other case in which there
were multiple copies on an app in various places) decreed that if MPW
was not running at the moment of the double-click, then the MPW on
the CD would be launched. That--at CD speeds of the time--took a
long time, after which MPW would declare that a file it needed to
update couldn't be written (because it was on the same volume as the
running MPW). End of game.

Starting MPW before double-clicking the ReadMe solved the problem,
since the very first of the set of rules for picking the right
instance was "if the app is running, use it."

But I almost never remembered.

This situation was by no means limited to MPW--that's simply where I
ran into it most often.

Then there was the whole business of what removable volumes were
mounted at the moment of "rebuilding the desktop".

   --John

On Nov 17, 2006, at 12:55 PM, Kei Ishii wrote:
> On 17.11.2006, at 15:42, Nigel Stanger wrote:
>> While Unix will allow you to use any character (and I mean *any*
>> character)
>> in a file name, some can cause very strange problems under the right
>> circumstances. The weirdest problem I ever had with a Unix filename
>> was one
>> that I somehow inadvertently named "-something", i.e., with a leading
>> hyphen. Man, the contortions I had to go through to fix that :)
>
> Like
>
> $ rm -- -something

Yes, that would work NOW. that did not work in 1987 when I had the
same problem.

In fact, i have no idea how that file was ever (if ever) fixed, as I
had to have someone in operations fix it, and they hadn't gotten to
it by the time the quarter ended.

At 12:04 on 18-11-2006, Dan Frakes wrote:
> On 11/17/2006 7:02 AM, "Jolin M Warren" wrote:
>> So, for instance, the Finder would read filename extensions on a DOS
>> disk and add the appropriate type/creator codes.
>
> I don't recall this being the case. Thanks to File Exchange, OS 9 could
> often figure out which application should open which Windows filename
> extension(s), but I don't recall the Finder actually *adding* type/creator
> codes automatically. Am I mis-remembering? ;-)

My memory is that when you copied a file off a DOS floppy (for
instance), File Exchange would add the appropriate type/creator
codes. But I could be mis-remembering -- it's been a _long_ time
since I used a DOS disk with the OS 9 Finder. :-)

As John mentioned in a previous email, the Finder never performed its
full duties as an entry and exit point between filesystems. But that
says more about the implementation than the general strategy.

_________________
=> Jolin Warren, Edinburgh, Scotland

On 19/11/2006 9:04 AM, "Google Kreme" <gkremegmail.com> spake thus:

> Yes, that would work NOW. that did not work in 1987 when I had the
> same problem.

Early 90's for me IIRC.

> In fact, i have no idea how that file was ever (if ever) fixed, as I
> had to have someone in operations fix it, and they hadn't gotten to
> it by the time the quarter ended.

I think in my case I either used the "rm ./-something" trick, or more likely
is that I simply moved everything else out of the directory then rm -rf'd
the directory (aka deletion by tactical nuke).

--
Nigel Stanger, Dunedin, NEW ZEALAND.
http://xri.net/=nigel.stanger

On 11/17/2006 7:02 AM, "Jolin M Warren" wrote:
> So, for instance, the Finder would read filename extensions on a DOS
> disk and add the appropriate type/creator codes.

Dan Frakes disagreed with that; but Jolin is at least partly right,
as I verified just now (with MacOS 8.1). A text file of type .HTM
copied from a DOS floppy gets type code "TEXT" and creator code
"dosa". Or at least that's what my DataViz File Recognizer reports.
(That utility is also smart enough to tell from either the extension
or the file contents that "This is an HTML document", Kind "HTML",
which MacOS apparently could not.) Maybe MacOS 9.x was smarter, but
I skipped those versions in adding a new machine to my menagerie.

However, it's more complicated than that. If the file extension is
something that Finder doesn't recognize, then it displays an icon
which looks like an unidentified document with the latters "PC"
added. (In "icon" view, those two letters are composed of horizontal
line segments and spaces, much like the IBM logo. :-) If it does
recognize the extension, then it displays a more appropriate icon.
In either case, File Info will report Finder's association between
that extension and the document type (e.g., "PC document" or
"Microsoft Word 97-98 document"). These distinctions hold true
whether the file is on a read-only DOS floppy or has been copied to a
local disk, so obviously they do not depend on something created
during the copy process. Thus it appears that the Finder is adding
them on a temporary basis as part of the process of opening a folder
for display to the user.

Probably there are further complexities that are beyond my capability
to uncover. But at least it is very clear that the Finder does "add
... type/creator codes"

Carl