Well the answer is simple really - merely getting indexed by the search engine doesn't guarantee rankings.

That means, you can submit your XML sitemap or urllist.txt to Google and Yahoo! and your site will likely get crawled. And if you perform a 'site:yourdomain.com' search you will likely see most or all of your pages indexed.

But as I said, if you try and search for an important keyword, if your site isn't properly optimized you won't likely find it in the search results.

Which leads to my point: If a search engine crawler is having problems indexing your site as it is, merely implementing an urllist.txt or sitemap.xml is not going to solve the root problem - that is your navigation architecture is not functional. This could be due to many reasons - poor link structure, dynamic URLs causing problems or improperly coded links.

In any case, my recommendation is to first identify the problems associated with crawling your site and fix them. Don't worry about submitting a sitemap or URL list, cause that's not going to solve the real problem.

It's kinda like saying "well your car's motor is having problems, you might as well replace it" when all you really need is an oil change. You aren't necessarily solving the real problem but circumventing the system.

Remember that the search engine crawlers are essentially web browsers on steroids. If they can't find pages on your site it's because there is likely a problem with the site. And there are many ways to identify problems.

First, review your server logs - see if there are large amounts of 404 errors being generated. Also review the spider paths - are they crawling deeply into your site? If not, where are they stopping? Perhaps identifying the spiders exit points can help you pinpoint problem areas of your site.

Another thing you can do is get a free tool like Xenu and run it against your site.

Xenu acts similar to a search engine crawler in that it will request all the URLs it can find and even output a report for you.

Using a software tool like Xenu you can see if there are barriers to indexing such as 404 errors caused by broken links, and then fix those problems.

Some other ways to help crawlers get through your site, especially if it is a large site, is to employ breadcrumbs and HTML sitemaps. Large sites can effectively employ multiple sitemap pages to help all pages get indexed.

For example, I have a client with over 4.5 million indexed pages and most of those pages were found through a series of sitemap pages which start broad and work through to specific pages. There are probably 2-3 levels of sitemaps, but ultimately the spider can get to deep content rich pages. And that is what they really want.

Similarly, breadcrumb navigation can link every deep page to its parent(s) and even the home page.

Using breadcrumbs a search engine crawler can easily move from any page to any other page in the site with only a few links to follow. Properly constructed breadcrumb navigation can allow a crawler to get from deep page to deep page with only 3 or 4 links to follow, even if the products are located in different categories.

As you may have guessed from this article, I'd prefer to tackle the non-indexing issue another way. So I will answer the question I posed below:

If your site isn't getting crawled by the engines chances are there is a technical issue to be solved to improve the site's indexability. Therefore I'd prefer to find and fix the problem then create and submit a sitemap.xml or urllist.txt.

So before you assume that you need to use such a URL list, consider what the problem may be and try to find a solution so that the crawler can find your pages on its own.


Rob Sullivan
Head Organic Search Strategist
Enquiro Full Service Search Engine Marketing

Copyright 2003 - 2005 - Searchengineposition Inc.