Helping Spiders Find Your Site |
Despite our best-laid plans, some of the web's most prized technology can be a major stumbling block for a search engine spider. In a quick fashion, we'll take a look at these spider stumbling blocks, and lay out some quick optimization solutions you can implement to overcome them.
|
 |
1. Frames |
Despite the unique design capabilities they present, can be a major problem for search engine spiders. Many spiders cannot read them. The quick solution is to utilize your No Frames content to optimize your page.
|
| 2. Password Protected Pages |
These pages you probably don't want indexed anyway. Just be aware that like a human, the spider cannot enter any area that is protected by a password.
|
| 3. PDF Files |
Also known as Adobe Acrobat Reader files, present a major stumbling block to most spiders. Some engines (specifically Google) are beginning to index these kinds of pages, but from an optimization perspective this is one format you want to avoid.
|
| 4. Dynamic Pages |
Any web address that contains a question mark (?), or some other like character, is a major stumbling block to most spiders. This is most often seen with dynamic pages that use Perl, ASP, Cold Fusion, etc. Some of the larger search engines are now beginning to index dynamic pages, (specifically Google), but this advanced capability is not widespread.
|
| 5. Meta Refresh |
This tag has been so abused by the XXX industry that it is now considered spam by the engines. Perhaps it is not really a stumbling block, but the spiders have been programmed to run from it.
|
| 6. Image Maps |
Can be read by some spiders but cannot be read by others. If you plan to use an image map, make sure there are other links on the page (perhaps on the bottom) those links to your other pages.
|
