Finding rogue invisible files

On Apr 20, 2007, at 2:03 PM, Kiran Wagle wrote:

> Once upon a time, one could search for invisible files larger than
> some target size, or invisible files created after a certain date, or
> whatever, but I can't find any useful tools to do that, or to compare
> the entire drives and tell me what has changed. I can't even trust
> the Finder to help me narrow down what folder the rogue file might be
> in, since it doesn't seem to care about Unix-y files.

Try this:

http://www.id-design.com/software/whatsize/

It is good for spotting big files, but not for finding smaller ones.


Kirk

              Read my blog: Kirkville -- http://www.mcelhearn.com
           Musings, Opinion and Miscellanea, on Macs, iPods and more

Perhaps this thing done in the Terminal can help:

sudo find / -size +200000 -ls

Cut and paste it, push enter and input your password and then you go.
It should find files larger than 100 MB (200*512 KB, IIRC).

Kind regards,

Kim

Kiran Wagle wrote:

> Whatever occurred, it seems to have happened in the past 24 hours or
> so, because Wednesday's backup succeeded and Thursday's failed. I
> could restore from the backup, of course, but I'd rather figure out
> what ate 20G of hard drive space and fix the original drive. I
> assume some rogue process left a giant temp file behind somewhere...
> but how the heck do I find it?

With that much space, it sounds like your "rogue process" is the kernel
and the "temp files" are swap files. What's in /var/vm? If you're
running the backup in the background, doing the backup itself may be the
cause of the swapping.

Open a Finder window, press Shift-Cmd-G and type '/Volumes/' (no quotes). You should see an alias to your boot drive plus any other storage devices currently mounted on your system. If your backup drive was _not_ mounted when Thursday's backup ran, you'll see a folder stuffed with your missing 20gigs..

Sam

On Apr 20, 2007, at 5:03 AM, Kiran Wagle wrote:

>
>
> Once upon a time, one could search for invisible files larger than
> some target size, or invisible files created after a certain date, or
> whatever, but I can't find any useful tools to do that, or to compare
> the entire drives and tell me what has changed. I can't even trust
> the Finder to help me narrow down what folder the rogue file might be
> in, since it doesn't seem to care about Unix-y files.
>
> Ideas?
>


There are utilities that turn on the switch to display system and
hidden files. One that does this is the freeware Tinker Tool <http://
www.bresink.com/osx/TinkerTool.html>. The option is the first one in
the Finder section.

Alan Forkosh Oakland, CA
aforkoshmac.com

My favorite utility for finding large files is OmniDiskSweeper,
available from http://www.omnigroup.com/applications/omnidisksweeper/

It catalogs your entire disk, including all the normally invisible
UNIX files, and shows you the results in a Finder style column view,
along with the file and folder sizes. Then you can surf through the
view and look for things to remove.

Have you rebooted recently? I find that programs like iTunes will
move things to the Trash, and yet they don't get deleted until I
restart. If you're comfortable on the command line you can list
~/.Trash/ and /.Trashes/ to see what might be lurking there, and
remove it immediately if appropriate.

Hope that helps!

Dave

On or about 4/20/07 5:03 AM, thus spake "Kiran Wagle" <entropyio.com>:

> I'd rather figure out
> what ate 20G of hard drive space and fix the original drive. I
> assume some rogue process left a giant temp file behind somewhere...
> but how the heck do I find it?

I use DiskInventory X for this kind of thing. I use it a lot!

<http://www.derlien.com/>

It answers the question, "Hey, what happened to my disk space?"

m.

--
matt neuburg, phd = matttidbits.com, http://www.tidbits.com/matt/
pantes anthropoi tou eidenai oregontai phusei
AppleScript: the Definitive Guide - Second Edition!
http://www.amazon.com/gp/product/0596102119
Take Control of Word 2004, Tiger, and more -
http://www.takecontrolbooks.com/tiger-customizing.html
Subscribe to TidBITS! It's free and smart. http://www.tidbits.com/

On 4/20/2007 5:03 AM, "Kiran Wagle" wrote:
> Once upon a time, one could search for invisible files larger than
> some target size, or invisible files created after a certain date, or
> whatever, but I can't find any useful tools to do that, or to compare
> the entire drives and tell me what has changed. I can't even trust
> the Finder to help me narrow down what folder the rogue file might be
> in, since it doesn't seem to care about Unix-y files.

Check out WhatSize:

<http://www.id-design.com/software/whatsize/index.php>

It scans your drives and tells you what's taking up space, including
invisible files. Table View, available from the Tools menu, lets you filter
the results to show, for example, all files over 100MB in size.

On Apr 20, 2007, at 5:03 AM, Kiran Wagle wrote:

> So I came home to find my computer claiming that its disk was full.
> As of a few weeks ago, the Finder, which is slow to update itself,
> thought it had 18G free.

In some instances, users have experienced a situation that some
people refer to as an "Out-Of-Control Cache or Log File." This is
where you suddenly find that your hard drive has filled up and you
don't know what it has filled up with. This is the result of a cache
or log file that has gotten out of hand.

The first thing to try is simply restarting your Mac. If that
doesn't help...

WhatSize (free)
http://www.id-design.com/software/whatsize/

or

DiskSurveyor ($15)
http://www.twilightsw.com

or

OmniDiskSweeper ($15)
http://www.omnigroup.com/applications/omnidisksweeper/

will show you which files have balooned and are now taking up all of
your disk space.

Running MacJanitor and then Cache Out X should remedy the problem.

MacJanitor (Free)
http://personalpages.tds.net/~brian_hill/macjanitor.html

Cache Out X (Free. Free registration required to access and download.)
http://nonamescriptware.com/programs/CacheOutX.sitt


___________________________________________
Randy B. Singer
Co-author of The Macintosh Bible (3rd, 4th, and 5th editions)

Macintosh OS X Routine Maintenance
http://www.macattorney.com/ts.html
___________________________________________

On 20-Apr-2007, at 09:59, Alan Forkosh wrote:
> On Apr 20, 2007, at 5:03 AM, Kiran Wagle wrote:
>> Once upon a time, one could search for invisible files larger than
>> some target size, or invisible files created after a certain date, or
>> whatever, but I can't find any useful tools to do that, or to compare
>> the entire drives and tell me what has changed. I can't even trust
>> the Finder to help me narrow down what folder the rogue file might be
>> in, since it doesn't seem to care about Unix-y files.
>
> There are utilities that turn on the switch to display system and
> hidden files.

But that doesn't help with spotlight.

At 5:03 AM -0700 4/20/07, Kiran Wagle wrote:
>Once upon a time, one could search for invisible files larger than
>some target size, or invisible files created after a certain date, or
>whatever, but I can't find any useful tools to do that, or to compare
>the entire drives and tell me what has changed. I can't even trust
>the Finder to help me narrow down what folder the rogue file might be
>in, since it doesn't seem to care about Unix-y files.

I find GrandPerspective fun and useful in finding those hidden
beasts. It produces a graphical presentation of file sizes.

<http://grandperspectiv.sourceforge.net>

-Charles

On Apr 20, 2007, at 8:03 AM, Kiran Wagle wrote:
> Once upon a time, one could search for invisible files larger than
> some target size, or invisible files created after a certain date, or
> whatever, but I can't find any useful tools to do that, or to compare
> the entire drives and tell me what has changed. I can't even trust
> the Finder to help me narrow down what folder the rogue file might be
> in, since it doesn't seem to care about Unix-y files.

Unixy eh? I'll give you Unixy!

I use the Unix commands all the time to help trace down this type of
issue: What is taking up all the space on my hard drive? The nice
thing, as you state, is that files and directories that are normally
hidden by the Mighty Finder (http://myspace.com/themightyfinder.
Checkout the Jobs vs. Gates video), and not hidden to your Unix sub-
system. Try the "du" command like this:

$ sudo du -sk *

This command will list all the files in the current directory, and
calculate their sizes in Kilobytes. The "sudo" is there to allow you
permission to search directories where you may not have permission.
To sort the information, do this:

$ sudo du -sk * | sort -n

This will take the output of the du command, and sort it numerically
(without the -n, it will sort alphabetically, so, 1245 will come
before 200). However, that form of the command won't produce any
output until it has gone through all of the directories. For example,
to find out which user is hogging all the information, try this:

$ cd /Users
$ sudo -sk * | sort

20448 Shared
57344 Deleted Users
105944 dad
264496 mom
440640 carol
739784 bob
1870384 alice
2057816 ted

Looks like Ted is using 2Gb of storage.

I usually start with the root drive ("/") and work my way up. Last
time I did this, I discovered about 3 gigabytes worth of files inside
my Library/Preferences folder. I forget which program was doing it,
but it was apparently using my Library/Preferences folder to download
large files.

I find the "du" command not super fast, but it usually doesn't take
longer than 5 minutes to 10 minutes to go from the root directory,
and find which directory is taking up all the room. You can pretty
easily narrow down the problem with in 30 minutes.

======

Now, to do exactly what you asked, you can use the "find" command
like this:

$sudo find <directoryToSearch> -size +<sizeInBytes>c

For example, to search from the root directory and print out all
files bigger than 100Mb, do this:

$ sudo find / -size +100000000c

Note that the plus sign is extremely important. Without it, the find
command will list all the files around 100Mb and not the ones bigger
than 100Mb.

And finally, there's the "ls" command which can give you the details
of the files:

$ sudo ls -l

Will list all the files in the current directory with their size,
date, etc. Believe it or not, there's also a "-ls" switch on the find
command too:

$ sudo find / -size +100000000c -ls

This will produce a similar listing to the "ls -l" command for each
file found.

One of the nice things about Unix commands is that the entire online
manual for each command is stored on your disk. For more information
on these command simply type:

$ man <command>

Word o'Warning: Be very careful about deleting any files you don't
know -- especially "Unixy" files. Mac OS X is a Unixy OS and some of
those files may be important. On the other hand, you may want to look
in the /core directory and delete anything under that directory.
These are "core dumps" of command that have crashed over the years,
and are pretty safe to delete.

David Weintraub

--On April 21, 2007 11:30:56 PM -0700 David Weintraub
<davidweintraub.name> wrote:

> $ sudo du -sk *
>
> This command will list all the files in the current directory, and
> calculate their sizes in Kilobytes. The "sudo" is there to allow you
> permission to search directories where you may not have permission.

if you normally run as an non-admin user account you won't be able to use
sudo. If you think the large file may be in your home directory you can
run without the sudo. If you want to search you're whole drive you'll need
to login as an admin first.

Also, it defeats the sort command, but I like adding the -h option instead
of -k so the numbers are more readable to me.

du -sh ~/*

dumps the sizes of the folders in your home directory. If you think the
file is somewhere in your Library folder:

du -sh ~/Library/*

Kevin

On Apr 21, 2007, at 11:30 PM, David Weintraub wrote--as part of
excellent advice:

> $ sudo du -sk *

For those of us who routinely run as a non-administrative user, there
is a step to take before heading down this road.

$ su - youradminuser

You are prompted for youradminuser's password (and of course use the
real account (short) name.

That makes you a user who can use sudo (as the Mac shipped). A
normal user can't. As this is the second time this has come up in
this thread, I thought I would jump in.

Also:
> $ sudo ls -l
>
> Will list all the files in the current directory with their size,
> date, etc.

That (as the Mac shipped) will not list files that are hidden in the
Unix sense--those with names starting with "." .
(We could argue a long time about how to punctuate that one.)

$ sudo ls -lA

will include the files whose names begin with . except for the pesky
but important "." and ".." pseudo directories.

Oh, and both David and I are showing the commands with the shell's
prompt characters ("$ "). Don't type that part. (Other shells use
other prompts.)

   --John

At 11:30 PM -0700 4/21/07, bignoseduglyguy wrote:

>On 21/04/2007, at 11:33 AM, Charles Turner wrote:
> I find GrandPerspective fun and useful in finding those hidden
> beasts. It produces a graphical presentation of file sizes.
> <http://grandperspectiv.sourceforge.net>
> -Charles

>I have only reviewed the screenshot so am maybe missing an essential
>point but how do you make sense of the display?

Each square represents a file in a selected folder.
The size of the square is proportional to the file size.
The hierarchy of sub-folders is flattened so all files in sub-folders
are shown.
Files are grouped and colored by sub-folder.
You can zoom in and out for more or less fine scale visibility.
Mousing over a square provides file details.
Selected files can be revealed in the Finder

-Charles

JDiskReport <http://www.jgoodies.com/freeware/jdiskreport/>

Try AppleJack <http://applejack.sourceforge.net/>