Video Sitemaps: Is your video part of a gallery?

Friday, September 17, 2010 at 3:45 PM

Webmaster Level: All

Often a website which hosts videos will have a common top-level page that groups conceptually related videos together. Such a page may be of interest to a user searching on that subject. Sites with many videos about a single subject can group these videos together on a top-level page, often known as a gallery. This can make it easier for users to find exactly what they're looking for. In this case, you can use a Sitemap to tell Google the URL of the gallery page on which each video appears.

You can specify the URL of the gallery level page using the optional tag <video:gallery_loc> on a per-video basis. Note that only one gallery_loc is allowed per video.

For more information on Google Videos, including Sitemap specifications, please visit our Help Center. To post questions and search for answers, check out our Help Forum.

Posted by Nelson Lee, Product Manager, Video Search

To err is human, Video Sitemap feedback is divine!

Monday, August 16, 2010 at 10:06 PM

Webmaster Level: All

You can now check your Video Sitemap for even more errors right in Webmaster Tools! It’s a new Labs feature to signal issues in your Video Sitemap such as:

• URLs disallowed by robots.txt
• Thumbnail size errors (160x120px is ideal. Anything smaller than 90x50 will be rejected.)

Video Sitemaps help us to better crawl and extract information about your videos, so we can appropriately feature them in search results.

Totally new to Video Sitemaps? Check out the Video Sitemaps center for more information. Otherwise, take a look at this new Labs feature in Webmaster Tools.

Written by Jackie Lai, Video Search Team

Video Sitemaps: Understanding location tags

Sunday, August 15, 2010 at 10:41 PM

Webmaster Level: All

If you want to add video information to a Sitemap or mRSS feed you must specify the location of the video. This means you must include one of two tags, either the video:player_loc or video:content_loc. In the case of an mRSS feed, these equivalent tags are media:player or media:content, respectively. We need this information to verify that there is actually a live video on your landing page and to extract metadata and signals from the video bytes for ranking. If one of these tags is not included we will not be able to verify the video and your Sitemap/mRSS feed will not be crawled. To reduce confusion, here is some more detail about these elements.

Video Locations Defined

Player Location/URL: the player (e.g., .swf) URL with corresponding arguments that load and play the actual video.

Content Location/URL: the actual raw video bytes (e.g., .flv, .avi) containing the video content.

The Requirements

One of either the player video:player_loc or content video:content_loc location is required. However, we strongly suggest you provide both, as they each serve distinct purposes: player location is primarily used to help verify that a video exists on the page, and content location helps us extract more signals and metadata to accurately rank your videos.

URL extensions at a glance:

Sitemap:	mRSS:	Contents:
<loc>	<link>	The playpage URL
<video:player_loc>	<media:player> (url attribute)	The SWF URL
<video:content_loc>	<media:content> (url attribute)	The FLV or other raw video URL

NOTE: All URLs should be unique (every URL in your entire Video Sitemap and mRSS feed should be unique)

If you would like to better ensure that only Googlebot accesses your content, you can perform a reverse DNS lookup.

For more information on Google Videos please visit our Help Center, and to post questions and search for answers check out our Help Forum.

Posted by Nelson Lee, Product Manager, Video Search

Video Sitemaps 101: Making your videos searchable

Friday, July 16, 2010 at 9:20 AM

Webmaster Level: All

We know that some of you, or your clients or colleagues, may be new to online video publishing. To make it easier for everyone to understand video indexing and Video Sitemaps, we’ve created a video -- narrated by Nelson Lee, Video Search Product Manager -- that explains everything in basic terms:



Also, last month we wrote about some best practices for getting video content indexed on Google. Today, to help beginners better understand the whys and hows of implementing a Video Sitemap, we added a starting page to the information on Video Sitemaps in the Webmaster Help Center. Please take a look and share your thoughts.

Posted by Amy MacIsaac, Content Partnerships

Sitemaps: One file, many content types

Tuesday, June 29, 2010 at 4:48 PM

Webmaster Level: All

Have you ever wanted to submit your various content types (video, images, etc.) in one Sitemap? Now you can! If your site contains videos, images, mobile URLs, code or geo information, you can now create—and submit—a Sitemap with all the information.

Site owners have been leveraging Sitemaps to let Google know about their sites’ content since Sitemaps were first introduced in 2005. Since that time additional specialized Sitemap formats have been introduced to better accommodate video, images, mobile, code or geographic content. With the increasing number of specialized formats, we’d like to make it easier for you by supporting Sitemaps that can include multiple content types in the same file.

The structure of a Sitemap with multiple content types is similar to a standard Sitemap, with the additional ability to contain URLs referencing different content types. Here's an example of a Sitemap that contains a reference to a standard web page for Web search, image content for Image search and a video reference to be included in Video search:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:image="http://www.sitemaps.org/schemas/sitemap-image/1.1"
        xmlns:video="http://www.sitemaps.org/schemas/sitemap-video/1.1">
  <url> 
    <loc>http://www.example.com/foo.html</loc> 
    <image:image>
       <image:loc>http://example.com/image.jpg</image:loc> 
    </image:image>
    <video:video> 
    <video:content_loc>http://www.example.com/videoABC.flv</video:content_loc>    
      <video:title>Grilling tofu for summer</video:title> 
    </video:video>
  </url>
</urlset>

Here's an example of what you'll see in Webmaster Tools when a Sitemap containing multiple content types is submitted:



We hope the capability to include multiple content types in one Sitemap simplifies your Sitemap submission. The rest of the Sitemap rules, like 50,000 max URLs in one file and the 10MB uncompressed file size limit, still apply. If you have questions or other feedback, please visit the Webmaster Help Forum.

Written by Jonathan Simon, Webmaster Trends Analyst

Help Google index your videos

Monday, April 19, 2010 at 5:12 PM

Webmaster Level: All

The single best way to make Google aware of all your videos on your website is to create and maintain a Video Sitemap. Video Sitemaps provide Google with essential information about your videos, including the URLs for the pages where the videos can be found, the titles of the videos, keywords, thumbnail images, durations, and other information. The Sitemap also allows you to define the period of time for which each video will be available. This is particularly useful for content that has explicit viewing windows, so that we can remove the content from our index when it expires.

Once your Sitemap is created, you can can submit the URL of the Sitemap file in Google Webmaster Tools or through your robots.txt file.

Once we have indexed a video, it may appear in our web search results in what we call a Video Onebox (a cluster of videos related to the queried topic) and in our video search property, Google Videos. A video result is immediately recognizable by its thumbnail, duration, and a description.

As an example, this is what a video result from CNN.com looks like on Google:

We encourage those of you with videos to submit Video Sitemaps and to keep them updated with your new content. Please also visit our recently updated Video Sitemap Help Center, and utilize our Sitemap Help Forum. If you've submitted a Video Sitemap file via Webmaster Tools and want to share your experiences or problems, you can do so here.

Posted by Nelson Lee, Product Manager Video Search

Adding Images to your Sitemaps

Wednesday, April 07, 2010 at 1:30 PM

Webmaster Level: All

Sitemaps are an invaluable resource for search engines. They can highlight the important content on a site and allow crawlers to quickly discover it. Images are an important element of many sites and search engines could equally benefit from knowing which images you consider important. This is particularly true for images that are only accessible via JavaScript forms, or for pages that contain many images but only some of which are integral to the page content.

Now you can use a Sitemaps extension to provide Google with exactly this information. For each URL you list in your Sitemap, you can add additional information about important images that exist on that page. You don’t need to create a new Sitemap, you can just add information on images to the Sitemap you already use.

Adding images to your Sitemaps is easy. Simply follow the instructions in the Webmaster Tools Help Center or refer to the example below:

<?xml version="1.0" encoding="UTF-8"?>
  <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
   xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
  <url>
    <loc>http://example.com/sample.html</loc>
    <image:image>
        <image:loc>http://example.com/image.jpg</image:loc>
    </image:image>
  </url>
</urlset>

We index billions of images and see hundreds of millions of image-related queries each day. To take advantage of that traffic most effectively, take a moment to update your Sitemap file with information on the images from your site. Let us know in the Sitemaps forum if you have any questions.

Posted by Alkis Evlogimenos, Software Engineer

Tips for News Search

Tuesday, September 01, 2009 at 11:22 AM

Webmaster Level: All

During my stint on the "How Google Works Tour: Seattle", I heard plenty of questions regarding News Search from esteemed members of the press, such as The Stranger, The Seattle Times and Seattle Weekly. After careful note-taking throughout our conversations, the News team and I compiled this presentation to provide background and FAQs for all publishers interested in Google News Search.



Along with the FAQs about News Sitemaps and PageRank in the video above, here's additional Q&A to get you started:

Would adding a city name to my paper—for example, changing our name from "The Times" to "The San Francisco Bay Area Times"—help me target my local audience in News Search?

No, this won't help News rankings. We extract geography and location information from the article itself (see video). Changing your name to include relevant keywords or adding a local address in your footer won't help you target a specific audience in our News rankings.

What happens if I accidentally include URLs in my News Sitemap that are older than 72 hours?

We want only the most recently added URLs in your News Sitemap, as it directs Googlebot to your breaking information. If you include older URLs, no worries (there's no penalty unless you're perceived as maliciously spamming -- this case would be rare, so again, no worries); we just won't include those URLs in our next News crawl.

To get the full scoop, check out the video!

Written by Maile Ohye, Developer Programs Tech Lead
Filmed by Michael Wyszomierski, Search Quality Team

An Update on Sitemaps at Google

Thursday, June 11, 2009 at 4:26 AM

Did you know that the number of website hosts that have been submitting Sitemap files has almost tripled over the last year? It's no wonder: the secret is out - as a recent research study showed, Sitemaps helps search engines to find new and changed content faster. Using Sitemaps doesn't guarantee that your site will be crawled and indexed completely, but it certainly helps us understand your website better.

Together with the Webmaster Tools design update, we've been working on Sitemaps as well:

• Google and the other search engines which are a part of Sitemaps.org now support up to 50,000 child Sitemaps for Sitemap index files (instead of the previous 1,000). This allows large sites to submit a theoretical maximum of 2.5 billion URLs with a single Sitemap Index URL (oh, and if you need more, you can always submit multiple Sitemap index files). 


• The Webmaster Tools design update now shows you all Sitemap files that were submitted for your verified website. This is particularly useful if you have multiple owners verified in Webmaster Tools or if you are submitting some Sitemap files via HTTP ping or through your robots.txt file.


• The indexed URL count in Webmaster Tools for your Sitemap files is now even more precise.


• For the XML developers out there, we've updated the XSD schemas to allow Sitemap extensions. The new schema helps webmasters to create better Sitemaps by verifying more features. By validating Sitemap files with the new schema, you can be more confident that the Sitemap files are correct.


• Do I need to mention that Sitemap file processing is much faster than ever before? We've drastically reduced the average time from submitting a Sitemap file to processing it and showing some initial data in Webmaster Tools. 

For more information about using Sitemaps, make sure to check out our blog post about frequently asked questions on Sitemaps and our Help Center. If you have any questions that aren't covered here, don't forget to search our Help Forum and start a thread in the Sitemaps section for more help. 

Written by John Müller, Webmaster Trends Analyst, Google Switzerland

Research study of Sitemaps

Monday, April 27, 2009 at 1:20 PM

We've been tracking the growth of Sitemaps on the web. It's been just 2 years since Google, Yahoo and Microsoft co-announced the Sitemaps directive in robots.txt, and it is already supported in many millions of websites including educational and government websites! At the WWW'09 conference in Madrid, Uri Schonfeld presented his summer internship work studying Sitemaps from a coverage and freshness perspective. If you're interested in how some popular websites are using Sitemaps, and how Sitemaps complement "classic" webcrawling, take a look:

Sitemaps: Above and Beyond the Crawl of Duty

At Google, we care deeply about getting increased coverage and freshness of the content we index. We are excited about open standards that help webmasters open up their content automatically to search engines, so users can find relevant content for their searches.

Written by Narayanan Shivakumar, VP Engineering

Using stats from site: and Sitemap details

Wednesday, March 04, 2009 at 4:40 PM

Webmaster Level: Beginner to Intermediate

Every now and then in the webmaster blogosphere and forums, this issue comes up: when a webmaster performs a [site:example.com] query on their website, the number of indexed results differs from what is displayed in their Sitemaps report in Webmaster Tools. Such a discrepancy may smell like a bug, but it's actually by design. Your Sitemap report only reflects the URLs you've submitted in your Sitemap file. The site operator, on the other hand, takes into account whatever Google has crawled, which may include URLs not included in your Sitemap, such as newly added URLs or other URLs discovered via links.

Think of the site operator as a quick diagnosis of the general health of your site in Google's index. Site operator results can show you:

• a rough estimate of how many pages have been indexed
• one indication of if your site has been hacked
• if you have duplicate titles or snippets

Here is an example query using the site operator:



Your Sitemap report provides more granular statistics about the URLs you submitted, such as the number of indexed URLs vs. the number submitted for crawling, and Sitemap-specific warnings or errors that may have occurred when Google tried to access your URLs.



Feel free to check out our Help Center for more on the site: operator and Sitemaps. If you have further questions or issues, please post to our Webmaster Help Forum, where experienced webmasters and Googlers are happy to help.

Posted by Charlene Perez, Search Quality Team

A new Google Sitemap Generator for your website

Tuesday, January 13, 2009 at 5:12 AM

It's been well over three years since we initially announced the Python Sitemap generator in June 2005. In this time, we've seen lots of people create great third-party Sitemap generators to help webmasters create better Sitemap files. While most Sitemap generators either crawl websites or list the files on a server, we have created a different kind of Sitemap generator that uses several ways to find URLs on your website and then allows you to automatically create and maintain different kinds of Sitemap files.

About Google Sitemap Generator

Our new open-source Google Sitemap Generator finds new and modified URLs based on your webserver's traffic, its log files, or the files found on the server. By combining these methods, Google Sitemap Generator can be very fast in finding these URLs and calculating relevant metadata, thereby making your Sitemap files as effective as possible. Once Google Sitemap Generator has collected the URLs, it can create the following Sitemap files for you:

• XML Sitemaps for Web Search according to the sitemaps.org standard


• Mobile Sitemaps for mobile-friendly websites


• Code Search Sitemaps for source code that you make available to users

In addition, Google Sitemap Generator can send a ping to Google Blog Search for all of your new or modified URLs. You can optionally include the URLs of the Sitemap files in your robots.txt file as well as "ping" the other search engines that support the sitemaps.org standard.

Sending the URLs to the right Sitemap files is simple thanks to the web-based administration console. This console gives you access to various features that make administration a piece of cake while maintaining a high level of security by default.

Getting started

Google Sitemap Generator is a server plug-in that can be installed on both Linux/Apache and Microsoft IIS Windows-based servers. As with other server-side plug-ins, you will need to have administrative access to the server to install it. You can find detailed information for the installation in the Google Sitemap Generator documentation.

We're excited to release Google Sitemap Generator with the source code and hope that this will encourage more web hosters to include this or similar tools in their hosting packages!

Do you have any questions? Feel free to drop by our Help Group for Google Sitemap Generator or ask general Sitemaps question in our Webmaster Help Forum.

Posted by John Mueller, Webmaster Trends Analyst, Google Zürich

On-Demand Sitemaps for Custom Search

Thursday, November 13, 2008 at 10:31 AM

Since we launched enhanced indexing with the Custom Search platform earlier this year, webmasters who submit Sitemaps to Webmaster Tools get special treatment: Custom Search recognizes the submitted Sitemaps and indexes URLs from these Sitemaps into a separate index for higher quality Custom Search results. We analyze your Custom Search Engines (CSEs), pick up the appropriate Sitemaps, and figure out which URLs are relevant for your engines for enhanced indexing. You get the dual benefit of better discovery for Google.com and more comprehensive coverage in your own CSEs.

Today, we're taking another step towards improving your experience with Google webmaster services with the launch of On-Demand Indexing in Custom Search. With On-Demand Indexing, you can now tell us about the pages on your websites that are new, or that are important and have changed, and Custom Search will instantly schedule them for crawl, and index and serve them in your CSEs usually within 24 hours, often much faster.

How do you tell us about these URLs? You guessed it... provide a Sitemap to Webmaster Tools, like you always do, and tell Custom Search about it. Just go to the CSE control panel, click on the Indexing tab, select your On-Demand Sitemap, and hit the "Index Now" button. You can tell us which of these URLs are most important to you via the priority and lastmod attributes that you provide in your Sitemap. Each CSE has a number of pages allocated within the On-Demand Index, and with these attributes, you can us which are most important for indexing. If you need greater allocation in the On-Demand index, as well as more customization controls, Google Site Search provides a range of options.

Some important points to remember:

1. You only need to submit your Sitemaps once in Webmaster Tools. Custom Search will automatically list the Sitemaps submitted via Webmaster Tools and you can decide which Sitemap to select for On-Demand Indexing.
2. Your Sitemap needs to be for a website verified in Webmaster Tools, so that we can verify ownership of the right URLs.
3. In order for us to index these additional pages, our crawlers must be able to crawl them. You can use "Webmaster Tools > Crawl Errors > URLs restricted by robots.txt" or check your robots.txt file to ensure that you're not blocking us from crawling these pages.
4. Submitting pages for On-Demand Indexing will not make them appear any faster in the main Google index, or impact ranking on Google.com.

We hope you'll use this feature to inform us regularly of the most important changes on your sites, so we can respond quickly and get those pages indexed in your CSE. As always, we're always listening for your feedback on Custom Search.

Written by Rajat Mukherjee, Group Product Manager, Search

Video Tutorial: Google for Webmasters

Wednesday, October 15, 2008 at 12:45 PM

We're always looking for new ways to help educate our fellow webmasters. While you may already be familiar with Webmaster Tools, Webmaster Help Discussion Groups, this blog, and our Help Center, we've added another tutorial to help you understand how Google works. Hence we've made this video of a soon-to-come presentation titled "Google for Webmasters." This video will introduce how Google discovers, crawls, indexes your site's pages, and how Google displays them in search results. It also touches lightly upon challenges webmasters and search engines face, such as duplicate content, and the effective indexing of Flash and AJAX content. Lastly, it also talks about the benefits of offerings Webmaster Central and other useful Google products.


Take a look for yourself.

Discoverability:



Accessibility - Crawling and Indexing:


Ranking:


Webmaster Central Overview:


Other Resources:



Google Presentations Version:
http://docs.google.com/Presentation?id=dc5x7mrn_245gf8kjwfx

Important links from this presentation as they chronologically appear in the video:
Add your URL to Google
Help Center: Sitemaps
Sitemaps.org
Robots.txt
Meta tags
Best uses of Flash
Best uses of Ajax
Duplicate content
Google's Technology
Google's History
PigeonRank
Help Center: Link Schemes
Help Center: Cloaking
Webmaster Guidelines
Webmaster Central
Google Analytics
Google Website Optimizer
Google Trends
Google Reader
Google Alerts
More Google Products


Special thanks to Wysz, Chark, and Alissa for the voices.

Written by Evan Tang, Search Quality

Sitemaps offer better coverage for your Custom Search Engine

Tuesday, May 06, 2008 at 12:04 PM

Rajat Mukherjee, Group Product Manager, Search

If you're a webmaster or site owner, you realize the importance of providing high quality search on your site so that users easily find the right information.

We just announced today that AdSense for Search is now powered by Custom Search. Custom Search (a Google-powered search box that you can install on your website in minutes) helps your users quickly find what they're looking for. As a webmaster, Custom Search gives you advanced customization options to improve the accuracy of your site's search results. You can also choose to monetize your traffic with ads tuned to the topic of your site. If you don't want ads, you can use Custom Search Business Edition.

Now, we're also looking to index more of your site's content for inclusion in your Custom Search Engine (CSE) used for search on your site. We figure out what sites and URLs are included in your CSE, and -- if you've provided Sitemaps for the relevant sites -- we use that information to create a more comprehensive experience for your site's visitors. You don't have to do anything specific, besides submitting a Sitemap (via Webmaster Tools) for your site if you haven't already done so. Note that this change will not result in more pages indexed on Google.com and your search rankings on Google.com won't change. However, you will be able to get much better results coverage in your CSE.

Custom Search is built on top of the Google index. This means that all pages that are available on Google.com are also available to your search engine. We're now maintaining a CSE-specific index in addition to the Google.com index for enhancing the performance of search on your site. If you submit a Sitemap, it's likely that we will crawl those pages and include them in the additional index we build.

In order for us to index these additional pages, our crawlers must be able to crawl them. Your Sitemap will also help us identify the URLs that are important. Please ensure you are not blocking us from crawling any pages you want indexed. Improved index coverage is not instantaneous, as it takes some time for the pages to be crawled and indexed.

So what are you waiting for? Submit your Sitemap!

SES London Calling!

Tuesday, March 18, 2008 at 8:05 AM

Written by Luisella Mazza, Search Quality Team.

February is that time of the year: the Search Engine Strategies conference hits London! A few of us were there to meet webmasters and search engine representatives to talk about the latest trends and issues in the search engine world.

It was a three-day marathon full of interesting talks - and of course, we heard a lot of good questions in between the sessions! If you didn't get a chance to talk with us, fear not: we've pulled together some of the best questions we encountered. You can find a few of them below, and an additional set in our Webmaster Help Group. Please join the discussion!

Why should I upload a Sitemap to Google Webmaster Tools, if my site is crawled just fine?

All sites can benefit from submitting a Sitemap to Google Webmaster Tools. You may help us to do a better job of crawling and understanding your site, especially if it has dynamic content or a complicated architecture.

Besides, you will have access to more information about your site, for example the number of pages from your Sitemaps that are indexed by Google, any errors Google found with your Sitemap, as well as warnings about potential problems. Also, you can submit specialized Sitemaps for certain types of content including Video, Mobile, News and Code.
More information about the benefits of submitting a Sitemap to Google Webmaster Tools can be found here.

How do you detect paid links? If I want to stay on the safe side, should I use the "nofollow" attribute on all links?

We blogged about our position on paid links and the use of nofollow a few months ago. You may also find it interesting to read this thread in our Help Group about appropriate uses of the nofollow attribute.

How do I associate my site with a particular country/region using Google Webmaster Tools? Can I do this for a dynamic website?

The instructions in our Help Center explain that you can associate a country or region to an entire domain, individual subdomains or subdirectories. A quick tip: if, for instance, you are targeting the UK market, better ways of structuring your site would be example.co.uk, uk.example.com, or example.com/uk/. Google can geolocate all of those patterns.

If your domain name has no regional significance, such as www.example.com, you can still associate your website with a country or region. To do that you will need to verify the domain, or the subdomains and/or subdirectories one by one in your Webmaster Tools account and then associate each of them with a country/region. However, for the moment we don't support setting a geographical target for patterns that can't be verified such as, for example, www.example.com/?region=countrycode.

I have a news site and it is not entirely crawled. Why? Other crawlers had no problem crawling us...

First off, make sure that nothing prevents us from crawling your news site - the architecture of your site or the robots.txt file. Also, we suggest you sign up for Webmaster Tools and submit your content. We specifically have the News Sitemap protocol for sites offering this type of content. If you take advantage of this feature, we can give you more information on which URLs we had trouble with and why. It really rocks!

A quick note to conclude: the lively, international environment of SES is always incredible. I have had a lot of interesting conversations in English, as well as in Italian, French and Spanish. Fellow Googlers chatted with webmasters in English, Danish, Dutch, German and Hungarian. That's amazing - and a great opportunity to get to know each other better, in the language you speak! So next time you wonder how Google Universal Search works in English or you're concerned about Google News Search in German, don't hesitate; grab us for a chat or write to us!

Cross-submissions via robots.txt on Sitemaps.org

Wednesday, February 27, 2008 at 6:00 PM

Posted by Prashanth Koppula, Product Manager

Last spring, the Sitemaps protocol was expanded to include the autodiscovery of Sitemaps using robots.txt to let us and other search engines supporting the protocol know about your Sitemaps. We subsequently also announced support for Sitemap cross-submissions using Google Webmaster Tools, making it possible to submit Sitemaps for multiple hosts on a single dedicated host. So it was only time before we took the next logical step of marrying the two and allowing Sitemap cross-submissions using robots.txt. And today we're doing just that.

We're making it easier for webmasters to place Sitemaps for multiple hosts on a single host and then letting us know by including the location of these Sitemaps in the appropriate robots.txt.

How would this work? Say for example you want to submit a Sitemap for each of the two hosts you own, www.example.com and host2.google.com. For simplicity's sake, you may want to host the Sitemaps on one of the hosts, www.example.com. For example, if you have a Content Management System (CMS), it might be easier for you to change your robots.txt files than to change content in a directory.

You can now exercise the cross-submission support via robots.txt (by letting us know the location of the Sitemaps):

a) The robots.txt for www.example.com would include:
Sitemap: http://www.example.com/sitemap-www-example.xml

b) And similarly, the robots.txt for host2.google.com would include:
Sitemap: http://www.example.com/sitemap-host2-google.xml

By indicating in each individual host's robots.txt file where that host's Sitemap lives you are in essence proving that you own the host for which you are specifying the Sitemap. And by choosing to host all of the Sitemaps on a single host, it becomes simpler to manage your Sitemaps.

We are making this announcement today on Sitemaps.org as a joint effort. To see what our colleagues have to say, you can also check out the blog posts published by Yahoo! and Microsoft.

Sitemaps FAQs

Tuesday, January 15, 2008 at 10:35 AM

Written by Susan Moskwa & Trevor Foucher, Webmaster Tools Team

Last month, Trevor spoke on the Sitemaps: Oversold, Misused or On The Money? panel at Search Engine Strategies in Chicago. After receiving a lot of great questions at the conference in addition to all the feedback we receive in our Help Group, we've pulled together a FAQ:

Q: I submitted a Sitemap, but my URLs haven't been [crawled/indexed] yet. Isn't that what a Sitemap is for?
A: Submitting a Sitemap helps you make sure Google knows about the URLs on your site. It can be especially helpful if your content is not easily discoverable by our crawler (such as pages accessible only through a form). It is not, however, a guarantee that those URLs will be crawled or indexed. We use information from Sitemaps to augment our usual crawl and discovery processes. Learn more.

Q: If it doesn't get me automatically crawled and indexed, what does a Sitemap do?
A: Sitemaps give information to Google to help us better understand your site. This can include making sure we know about all your URLs, how often and when they're updated, and what their relative importance is. Also, if you submit your Sitemap via Webmaster Tools, we'll show you stats such as how many of your Sitemap's URLs are indexed. Learn more.

Q: Will a Sitemap help me rank better?
A: A Sitemap does not affect the actual ranking of your pages. However, if it helps get more of your site crawled (by notifying us of URLs we didn't previously didn't know about, and/or by helping us prioritize the URLs on your site), that can lead to increased presence and visibility of your site in our index. Learn more.

Q: If I set all of my pages to have priority 1.0, will that make them rank higher (or get crawled faster) than someone else's pages that have priority 0.8?
A: No. As stated in our Help Center, "priority only indicates the importance of a particular URL relative to other URLs on your site, and doesn't impact the ranking of your pages in search results." Indicating that all of your pages have the same priority is the same as not providing any priority information at all.

Q: Is there any point in submitting a Sitemap if all the metadata (<changefreq>, <priority>, etc.) is the same for each URL, or if I'm not sure it's accurate?
A: If the value of a particular tag is the same for 100% of the URLs in your Sitemap, you don't need to include that tag in your Sitemap. Including it won't hurt you, but it's essentially the same as not submitting any information, since it doesn't help distinguish between your URLs. If you're not sure whether your metadata is accurate (for example, you don't know when a particular URL was last modified), it's better to omit that tag for that particular URL than to just make up a value which may be inaccurate.

Q: I've heard about people who submitted a Sitemap and got penalized shortly afterward. Can a Sitemap hurt you?
A: Only if it falls on you from a great height. (Seriously, though: if it ever happened that someone was penalized after submitting a Sitemap, it would have been purely coincidental. Google does not penalize you for submitting a Sitemap.)

Q: Where can I put my Sitemap? Does it have to be at the root of my site?
A: We recently enabled Sitemap cross-submissions, which means that you can put your Sitemap just about anywhere as long as you have the following sites verified in your Webmaster Tools account:

• the site on which the Sitemap is located
• the site(s) whose URLs are referenced in the Sitemap
  

Note that cross-submissions may not work for search engines other than Google. Learn more about Sitemap cross-submissions.

Q: Can I just submit the site map that my webmaster made of my site? I don't get this whole XML thing.
A: There's a difference between a (usually HTML) site map built to help humans navigate around your site, and an XML Sitemap built for search engines. Both of them are useful, and it's great to have both. A site map on your domain can also help search engines find your content (since crawlers can follow the links on the page). However, if you submit an HTML site map in place of a Sitemap, Webmaster Tools will report an error because an HTML page isn't one of our recognized Sitemap formats. Also, if you create an XML Sitemap, you'll be able to give us more information than you can with an HTML site map (which is just a collection of links). Learn more about supported Sitemap formats.

Q: Which Sitemap format is the best?
A: We recommend the XML Sitemap protocol as defined by sitemaps.org. XML Sitemaps have the advantage of being upgradeable: you can start simple if you want (by just listing your URLs), but—unlike a text file Sitemap—you can easily upgrade an XML Sitemap later on to include more metadata. XML Sitemaps are also more comprehensive than an Atom or RSS feed submitted as a Sitemap, since feeds usually only list your most recent URLs (rather than all the URLs you want search engines to know about).

Q: If I have multiple URLs that point to the same content, can I use my Sitemap to indicate my preferred URL for that content?
A: Yes. While we can't guarantee that our algorithms will display that particular URL in search results, it's still helpful for you to indicate your preference by including that URL in your Sitemap. We take this into consideration, along with other signals, when deciding which URL to display in search results. Learn more about duplicate content.

Q: Does the placement of a URL within a Sitemap file matter? Will the URLs at the beginning of the file get better treatment than the URLs near the end?
A: No, and no.

Q: If my site has multiple sections (e.g. a blog, a forum, and a photo gallery), should I submit one Sitemap for the site, or multiple Sitemaps (one for each section)?
A: You may submit as few or as many Sitemaps as you like (up to these limits). Organize them in whatever way you find easiest to maintain. If you create multiple Sitemaps, you can use a Sitemap Index file to list them all. Learn more.

If your question isn't covered here, you can find even more questions and answers in our Sitemaps Help Group.

Introducing Video Sitemaps

Monday, December 17, 2007 at 11:36 AM

Written by John Fisher-Ogden, Software Engineer, and Amy Wu, Associate Product Manager

In our effort to help users search all the world's public videos, the Google Video team joined the Sitemaps folks to introduce Video Sitemaps—an extension of the Sitemap Protocol that helps make your videos more searchable via Google Video Search. By submitting this video-specific Sitemap in addition to your standard Sitemap, you can specify all the video files on your site, along with relevant metadata. Here's an example:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
     xmlns:video="http://www.google.com/schemas/sitemap-video/1.0">
<url>
<loc>http://www.example.com/videos/some_video_landing_page.html</loc>
<video:video>
   <video:content_loc>http://www.example.com/video123.flv</video:content_loc>
   <video:player_loc allow_embed="yes">http://www.example.com/videoplayer.swf?video=123</video:player_loc>
   <video:title>My funny video</video:title>
   <video:thumbnail_loc>http://www.example.com/thumbs/123.jpg</video:thumbnail_loc>
</video:video>
</url>
<url>
<loc>http://www.example.com/videos/some_other_video_landing_page.html</loc>
<video:video>
   <video:content_loc>http://www.example.com/videos/video1.mpg</video:content_loc>
   <video:description>A really awesome video</video:description>
</video:video>
</url>
</urlset>

To get started, create a Video Sitemap, sign into Google Webmaster Tools, and add the Video Sitemap to your account.

New: Content analysis and Sitemap details, plus more languages

Thursday, December 13, 2007 at 12:48 PM

Written by Jonathan Simon, Webmaster Trends Analyst

We're always striving to help webmasters build outstanding websites, and in our latest release we have two new features: Content analysis and Sitemap details. We hope these features help you to build a site you could compare to a fine wine -- getting better and better over time.

Content analysis

To help you improve the quality of your site, our new content analysis feature should be a helpful addition to the crawl error diagnostics already provided in Webmaster Tools. Content analysis contains feedback about issues that may impact the user experience or that may make it difficult for Google to crawl and index pages on your site. By reviewing the areas we've highlighted, you can help eliminate potential issues that could affect your site's ability to be crawled and indexed. This results in better indexing of your site by Google and other search engines.

The Content analysis summary page within the Diagnostics section of Webmaster Tools features three main categories. Click on a particular issue type for more details:

• Title tag issues
• Meta description issues
• Non-indexable content issues

Selecting "Duplicate title tags" displays a list of repeated page titles along with a count of how many pages contain that title. We currently present up to thirty duplicated page titles on the details page. If the duplicate title issues shown are corrected, we'll update the list to reflect any other pages that share duplicate titles the next time your website is crawled.

Also, in the Title tag issues category, we show "Long title tags" and "Short title tags." For these issue types we will identify title tags that are way too short (for example "IT" isn't generally a good title tag) or way too long (title tag was never intended to mean <insert epic novel here>). A similar algorithm identifies potentially problematic meta description tags. While these pointers won't directly help you rank better (i.e. pages with <title> length x aren't moved to the top of the search results), they may help your site display better titles and snippets in search results, and this can increase visitor traffic.

In the "Non-indexable content issues," we give you a heads-up of areas that aren't as friendly to our more text-based crawler. And be sure to check out our posts on Flash and images to learn how to make these items more search-engine friendly.

Sitemap details page

If you've submitted a Sitemap, you'll be happy when you see the additional information in Webmaster Tools revealing how your Sitemap was processed. You can find this information on the newly available Sitemap Details page which (along with information that was previously provided for each of your Sitemaps) shows you the number of the pages from your Sitemap that were indexed. Keep in mind the number of pages indexed from your Sitemap may not be 100% accurate because the indexed number is updated periodically, but it's more accurate than running a "site:example.com" query on Google.

The new Sitemap Details page also lists any errors or warnings that were encountered when specific pages from your Sitemap were crawled. So the time you might have previously spent on crafting custom Google queries to determine how many pages from your Sitemap were indexed, can now be spent on improving your site. If your site is already the crème de la crème, you might prefer to spend the extra free time mastering your ice-carving skills or blending the perfect eggnog.

Here's a view of the new Sitemap details page:

Sitemaps are an excellent way to tell Google about your site's most important pages, especially if you have new or updated content that we may not know about. If you haven't yet submitted a Sitemap or have questions about the process, visit our Webmaster Help Center to learn more.

Webmaster Tools now available in Czech & Hungarian

We love expanding our product to help more people and in their language of choice. We recently put in effort to expand the number of Webmaster Tools available languages to Czech and Hungarian, in addition to the 20 other languages we already support. We won't be stopping here. Our desire to support even more languages in the future means that if your language of choice isn't currently supported, stay tuned -- there'll be even more supported languages to come.

We always love to hear what you think. Please visit our Webmaster Help Group to share comments or ask questions.