Last month, we introduced our Advanced SEO series, discussing the importance of page rankings and ways to make sure your website performs well against the numerous ways these rankings are formulated. Now, we turn our attention to navigating the inner workings of different types of searches and spiders.

Behavioral and Intent Based Search
Google is now using increasingly customized results depending on a user's past search patterns, intent, and proximity search. This complicates SEO by several orders of magnitude and necessitates having separate strategies for each:

1. Behavioral: It's critical to understand how the history of a customer's searches affect the ranking of your pages for your specific desired query string. Google now tracks past search history (via sessions) and will change search results based on that. For example, if someone searched "java coffee" and then later "java programming," Google will weigh the results to show some "java coffee" results, depending on multiple variables such as frequency relative to the most recent typed term.

2. Intent-Based Search: Google also keeps track of conversion rates and shopping patterns. A keyword can be deemed either a research or shopping keyword, and the criteria for ranking sites are vastly different. If your goal is to climb the rankings for a research term, for example, your pages should obviously be largely text based, and should not contain purchase-focused language such as "buy now" or "shopping cart" or other calls to action.

   Research Terms vs. Shopping Terms
   One way to check whether a keyword has been deemed by Google to be a research term vs. a shopping term is to Google it to if Wikipedia ranks for it in the top three search results. Wikipedia is the ultimate reference research site and ranks for virtually all research keywords. If you type in "load balancing" into Google, for example, you'll see that [as of this release] Wikipedia is #1 and 2 for this search term, with www.webopedia.com coming in 3rd—it's clearly a research keyword. To continue with this example, if your company aims to improve rankings using the term "load balancing" could publish a position paper or related document online, to get it indexed and then passively inform the audience that you offer these solutions. This will build the longer term sales funnel and establish your company's position as an authority on the subject.

   A smart strategy when it comes to search rank is to create custom pages for individual keywords, determining whether they are shopping or research terms. Shopping terms will rank better for pages that have purchase-centric keywords, with numerous small images indicative of shopping carts, and bulleted lists; whereas research keywords will have three times the average amount of text, links to expert sites, whitepapers, and contain fewer images. Custom pages should also be created for Google vs. Yahoo as Google is research-oriented, while Yahoo is shopping-oriented.

   Intent is also based on proximity. For instance, if someone searches for "New York pizza" while in New York, Google will assume an intent to purchase, and therefore will deliver pages that are clearly purchase-oriented, and include localized results as well as a map of locations. However, if the same search is performed in California, it may produce a research-skewed Wikipedia article like this one: http://en.wikipedia.org/wiki/New_York-style_pizza

3. Proximity Search: As mentioned above and as the term implies, this refers to the user's geographic location being a factor in the types of search results Google will return. Bruce Clay used this humorous—and true—example to illustrate localized results that change the intent based on the customized definition of that keyword population. In most places in the US, if you type in "drug rehab" it will produce research results; however, if you type the same thing while visiting Los Angeles, CA it is considered a shopping term producing mostly expensive drug rehab center listings!

Spiders
Another aspect of search engine programs to become familiar with and use to your advantage are "spiders"—automated search agents that trail links on the Web to discover new pages. Spiders fetch the HTML of a site, parse the retrieved information, run it through a spam checker, and schedule it to be indexed in a database. This is the database from which results are pulled when you rank for a keyword. There is another database where the information is not parsed and the full content exists, called the data area. To access this area, queries are typed in with quotes so the exact search term is pulled from the data cache of the page, including stop words (and, for, the, to, and so forth).

Spiders are important since they regulate how often your site is visited and updated in the Google index based on how often the content is changed. If spiders see that the content of your site rarely changes, they will reduce the frequency of their visits since it would be inefficient for them to keep re-indexing pages that rarely change.

A word of caution if you're expecting to see an immediate surge of spidering by making large-scale changes to your site content. Spidering frequency is a learned behavior based on established patterns in how often you update your content.

For example, if you rarely make changes to your site, spiders may not visit it for six months even if you suddenly do make significant changes, based on the fact that you had not previously been making regular updates—it takes the spiders some time to "learn" that you are now in the habit of altering your content. However, if you are disciplined and keep up with making changes on your site, the spiders will "learn" to visit more frequently and might visit as often as every day. (This is why RSS feeds—which provide dynamic content to a page on a consistent and regular basis—are so effective. They automatically populate your site with fresh, relevant content, to keep spiders visiting.) Generally, the more consistent you have been with your updates, the more quickly you will see results.

The changes must also be significant (some industry-standard guidelines estimate that you need to change about 20% of your content), so don't expect to register with the spiders if you're only changing a few keywords.

Things to keep in mind when evaluating whether your site is spider-friendly:

• Javascripts still affect spiders, and should be contained in files outside the page, as with CSS files
• Flash is indexed —but separately
• All 404 errors need to be found and corrected, and 301 redirects placed to corresponding pages
• XML sitemaps are very important, but so are HTML sitemaps
• Server speed is exceedingly important as spiders "learn" of slow servers and will stop coming to them
• Directory submissions: if you cannot get into ODP, then apply at the city name level
• Have a robots.txt file and link to the sitemap.xml
• Have an HTML sitemap to pass page rank
• Being spidered is a privilege, not a right

In summary, the sphere of search and ranking is constantly evolving. Being aware of the many factors involved in its complex formulas is vital to ensuring your site's success. Stay tuned for more tips and advice on search, ranking, and a host of other topics key to helping your business grow. At Superb, we are committed to staying up-to-date with developments in this ever-changing landscape and sharing them with you, our valued customers, to help you stay Ahead of the Rest®.

Be sure to check out our next issue, where we'll demystify the ever-changing jumble of Google formulas for page rankings, searches, and more.

Questions? Want to know more? Contact Dale Bunten, Director of Marketing at Superb Internet (dbunten@superb.net). Dale has 11 years of experience in Search Engine Optimization and PPC marketing for highly competitive keyword terms. He is a frequent speaker on SEO as well as various other Internet marketing topics and issues.