The Search Engine Marketing Kit - Chapter 1

Crawler-Based (Organic) Listings

Most search portals feature crawler-based search results as the primary element of their SERPs. These are also referred to as editorial, free, natural, or organic listings. Throughout the rest of this kit, we will refer to crawler-based listings as organic listings.

Crawler-based search engines depend upon special programs called robots or spiders. These spiders crawl the Web, following one link after another, to build up a large database of Web pages. We will use the words spider, crawler, or robot to refer to these programs throughout this kit.

Each crawler-based search engine uses its own unique algorithm, or formula, to determine the order of the search results. The databases that drive organic search results primarily contain pages that are found by Web-crawling spiders. Some search engines offer paid inclusion and trusted feed programs that guarantee the inclusion of certain pages in the database.

Paid inclusion is one of many ways in which search engines have begun to blur the line between organic and paid results. Trusted feed programs allow the site owner to feed the search engine an optimized summary of the pages in question; the pages may be ranked on the basis of their content summaries rather than their actual content.

Although all the search engines claim that paid inclusion does not give their customers a ranking benefit, the use of paid inclusion does offer SEO consultants an opportunity to tweak and test copy on Web pages more frequently. We will learn more about this in Chapter 2, Search Engine Optimization Basics.

Organic search listings are certainly the primary focus for search engine marketers and consultants, but they're not the only concern. In many cases, the use of pay-per-click is essential to a well-rounded strategy.

Most of today's search portals do not operate their own crawler-based search engine; instead, they acquire results from one of the major organic search players. The major providers of organic search listings are Google and Yahoo! who, in addition to operating their own popular search portals, also provide search results to a variety of different portals.

Aside from Google and Yahoo!, only a few major players operate crawling search engines. Ask uses its own Teoma search engine, LookSmart owns Wisenut, Lycos, too, has its own crawler-based engine, and Microsoft's MSN search is also in the mix. That's a grand total of six crawler-based search engines accounting for nearly all of the organic search results available in the English language.

Note: In order to have a meaningful chance to gain traffic from organic search listings, a Web page must appear on the first or second page of search results. Different search portals show varying numbers of results on the first page: Google displays ten, Yahoo! shows 15, and MSN's search presents eight. Any changes a major search portal might make to the listing layout will affect the amount of traffic your search engine listings attract.

Sponsored (Pay-Per-Click) Listings

It costs a lot of money to run a search portal. Crawler-based search engines operate at tremendous expense—an expense that most portals can't afford. Portals that don't operate their own crawler-based search engines must pay to obtain crawler-based search results from someone who does.

Either way, the delivery of unbiased organic search results is expensive, and someone has to pay the bill. In the distant past, search portals lost money hand over fist, but today, even very small search portals can generate revenue through sponsored listings. Metasearch engines typically use sponsored listings as their primary search results.

In addition to helping search portals stay afloat, sponsored listings provide an excellent complement to organic search results by connecting searchers with advertisers whose sites might not otherwise appear in the search results.

Most portals do not operate their own pay-per-click (PPC) advertising service. Instead, they show sponsored results from one or more partners and earn a percentage of those advertisers' fees. The major PPC providers are Google AdWords and the Overture service offered by Yahoo! (Note that, at the time of this writing, Overture is expected to announce a rebranding in Q2 2005, under which it would adopt the name "Yahoo! Search Marketing Solutions.") Other PPC providers with a significant presence include Findwhat and LookSmart.

The PPC advertising model is simple. Advertisers place bids against specific search terms. When users search on those terms, the advertiser's ads are returned with the search results. And, each time a searcher clicks on one of those ads, the advertiser is charged the per-click amount he or she bid for that term. PPC providers have added a few twists to this model over the years, as we'll see in Chapter 4, Paying To Play: Pay-Per-Click And Paid Inclusion.

Different PPC providers use different methods to rank their sponsored listings. All methods start with advertisers bidding against one another to have their ads appear alongside the results returned for various search terms, but each method has its own broad matching options to allow a single bid to cover multiple search terms.

Note: The bidding for extremely popular search terms can be quite fierce: it's not unusual to see advertisers bidding $10 per click—or more—for the privilege of appearing at the top of the sponsored listings. Reviewing the amounts that bidders are willing to pay for clicks to sponsored listings can give SEO practitioners a very good idea of the popularity of particular search terms—terms that may also be suitable for organic optimization.

In addition, PPC ranking systems are no longer as simple as allocating the highest position to the highest bidder. Google's methodology, for example, combines the click-through rate of an advertiser's listing (the number of clicks divided by the number of times it's displayed) with that advertiser's bid in assessing where the PPC advertisement will be located. Google's method tends to optimize the revenue generated per search, which is one of the reasons why its AdWords service has gained significantly on Overture.

Note: In the example SERP shown above (Figure 1.1), Yahoo! displays the first two sponsored listings in a prominent position above the organic results. Understanding which sponsored results will be displayed most prominently will help you determine how much to bid for different search terms. For example, it may be worth bidding higher to get into the #1 or #2 position for the most targeted search terms, since those positions will gain the most traffic from Yahoo!.

Directory (Human-Edited) Listings

Directory listings come from human-edited Web directories like LookSmart, The Open Directory, and the Yahoo! Directory. Most search portals offer directory results as an optional search, requiring the user to click a link to see them.

Because directories usually only list a single page (the homepage) of a Website, it can be difficult for searchers to find specific information through a directory search. As the quality of organic search results has improved, search portals have gradually reduced their emphasis on directory listings.

Currently, only Lycos displays a significant number of directory listings (from LookSmart), and that's likely to change as LookSmart transitions from its old business model (paid directory) into a standard PPC service.

The decline of directory listings within search results does not diminish the importance of directory listings in obtaining organic search engine rankings. All crawler-based search engines take links into account in their rankings, and links from directories are still extremely important.

Note: The way that Yahoo! makes directory results available to users should be a significant factor in helping the site owner decide whether or not to pay for a listing in the directory. At $299 per year, a paid listing in the Yahoo! Directory is a considerable expense for small businesses. Yet, while there is value in any link, the directory itself no longer generates significant traffic.

In addition, it is by no means clear whether the display of a directory category link below a site's organic search result listing may contribute to the click-through rate for that listing. In fact, it's possible that users might click this directory link and arrive at the directory category page, where the given listing could be buried at the bottom of a long list of competing sites.

Compared to other advertising options, paying $299 for a link buried deep within the Yahoo! Website is not as appealing as it once was. In addition, sites listed in the Yahoo! Directory automatically have a title and description displayed alongside each of their listings in the organic search results. This style of listing can actually generate a lower click-through than an ordinary listing within the organic results.

Whether or not you currently have a Yahoo! Directory listing, you owe it to yourself to discuss other ways to make use of those funds. For example, at an average of 20 cents per click, you could bring in nearly 1500 visitors per year through PPC advertising.

Other Listings

In addition to the three main types of search results, most search portals now offer additional types of search listings. The most common among these are:

• Multimedia searches, which help users find images, sounds, music etc.
• Shopping searches to help those searching for specific products and services.
• Local searches to find local business and information resources.
• People searches, including white pages, yellow pages, reverse phone number lookups.
• Specialized searches, covering government information, universities, scientific papers, maps, and more.