Sitemaps

>What I find missing is a clear statement of why any Webmaster should
>expend the resources needed to implement and maintain sitemap files.
>In other words, what will a sitemap file enable a Web crawler to do
>that it doesn't already do?


First, if anyone is wondering what the sitemap is, see:

   http://www.google.com/support/webmasters/bin/answer.py?answer=40318

and in general, anyone with a website should read the documentation
on the Google Webmaster pages in great depth.

I heard Google's Search Evangelist address your very question at a
seminar two night ago. The above link answers you question but I will
add my impression from what he said the other night.

For the most part, the spider will crawl your site and the sitemap
will be identical. But for large sites with dynamic content, the
spider may miss some nooks and crannies. The sitemap tells Google
where you think valuable content is and the spider will make sure to
hit it all. I think it would also affect frequency. My guess that
heavily spidered sites will benefit less from the sitemap.

You also can have otherwise unlinked pages in the sitemap, something
there was previously now way for Google to find.

And Google is comparing their spider's activity to sitemaps to learn
how to improve their algorithms.

Regardless, it is important to know that the sitemap never hurts you,
it can only help.

Note, too, that the sitemap program is in beta and changing.

You might also check the Google webmaster discussion groups. This
question is right up their alley.

On Dec 1, 2006, at 8:28 AM, Carl S Zimmerman wrote:

>
>
> What I find missing is a clear statement of why any Webmaster should
> expend the resources needed to implement and maintain sitemap files.
> In other words, what will a sitemap file enable a Web crawler to do
> that it doesn't already do? The news releases don't answer this
> question, nor does the Sitemaps Website. Even the "official" blogs
> of Google, the developer of the sitemaps protocol, don't answer it.

There's information on the Google Webmaster Site:

http://www.google.com/support/webmasters/bin/topic.py?topic=8476

And also information on Mobile Sitemaps and why it is important to
organize content specifically for mobile phones:

http://www.google.com/support/webmasters/bin/answer.py?
answer=34627&hl=en

>
> Given that one part of the sitemap protocol is a frequency-of-update
> indicator, it might be reasonable to assume that a sitemeap can
> encourage Web crawlers to revisit some parts of the site more
> frequently in return for revisiting other parts less frequently. But
> I can find no confirmation of such an assumption.
>
> Other questions which I think important but which are not covered in
> the FAQ (and which I didn't find answered in Google's blogs):
>
> - Will Web crawlers find a sitemap file if it isn't explicitly
> submitted to the various search engines?

The chances are excellent that the crawler will, however, it will
only happen if there are incoming links from other sites that will
lead the spiders to your site. The more incoming links, esp. those
containing relevant copy, the better it is for your rankings.

By constructing a sitemap using keywords and relevant content, you
can improve your rankings. This will also help with Flash, Quicktime,
etc. content. And make sure there's links to the sitemap on you home
page and on internal pages.

Spider can also get lost if there are many levels to click on; i.e.,
the spider has crawled 5-6 levels down into a folder structure, and
there's no link back to a site map or the home page. It will probably
then leave the site before crawling any other pages.

>
> - Is it necessary (or at least important, or even just helpful) to
> update and/or resubmit an existing sitemap file when one of the files
> to which it points has been updated?

It's a good idea to think of a site map as commitment if you'll be
updating content - you'll need to add and remove appropriately, and
adjust the copy accordingly. Revisiting your site map might also help
you find new ways to structure information more effectively - like in
the example I mentioned above, it can serve as a guide to see if your
architecture is getting too complicated for visitors and spiders.

>
> - If a Website has a "what's new" page that is regularly updated to
> identify changed pages elsewhere on that Website, is it necessary to
> mention any page other than this one in the sitemap?

It isn't necessary, but it is good strategy. Keep in mind that the
engines constantly re-index sites, and that although you have great
rankings on Monday, on Tuesday things could look very different.
You'll need to carefully research and monitor keywords and links.

>
> - If a Website is well structured (e.g., according to Google's
> Guidelines for Webmasters), is it necessary to list all files on the
> Website in sitemap files? If not, how should a Webmaster decide
> which files to list and which to omit?

Think of it not just listing copy, but including descriptive
information that will help visitors that are looking for particular
information on your site. If the copy in the links is well
constructed and contains words that visitors will be searching for,
it will improve your rankings as well.

It's also a good idea for visitors to include a sitemap on a 404
error page, if you've got one.

Marilyn