Google Web Site Optimizer Tool: I Told You So

Now, to validate your JavaScript. This is one of the nicer aspects of the service. The biggest problem with most services like this is it's easy to make small cut-n-pasting error at this stage, and I'm sure many people get lost and give up when they can't figure out where the problem is.

Thankfully Google have implemented a nifty inline checker, which allows you add your code, upload your pages and test them without leaving the Website Optimizer tool. And if it does find problems, it points the issues out and offers reasonably helpful advice.

When Google gives your green ticks for all your scripts, the only thing left to do is wait for the users to roll in. The tool will automatically divvy up your incoming traffic between your test pages and track the behaviour of users as they exit the page.

The Results

SitePoint gets sufficient traffic to give us usable data in around twelve hours. After setting up the test pages on Thursday night, we had a clear cut result by Friday morning.

My main motivation in setting up the test was simply to prove that the underlined text version wouldn't perform significantly better than the clean version -- if they were performed roughly the same, picking the prettier version was a no-brainer.

In the end, even I was surprised with the actual results. After 4,000 tests, the pretty version was proving almost twice as effective as the uglier underlined version (85.7%). That is a remarkable difference for a seemingly small change!

It also makes you wonder what other things you could be testing...

The Downsides

There are a couple of downsides to this technique.

Firstly, the scripts snippets break HTML validation. Even more infuriating, when I tried to rewrite them so they would validate -- i.e. I changed <script> to <script type="text/javascript"> -- Google couldn't use them anymore.

It may be possible to work around this issue, but it's a sloppy piece of coding from Google.

Secondly, creating two versions of a page does mean that the traffic to the original page will be split. This means your web stats will show a dramatic drop off for that original page for the period of the experiment.

It's hard to prove either way, but there may be some effect on the PageRank of your original page if you left the experiment running for a long time.

For me, the slightly disappointing thing is that currently neither Google Analytics or PageRank appear to compensate or adjust their behaviour for their own tool. Hopefully this will be rectified in the future.

Of course, the idea is to determine which page works best as quickly as possible, and make that one the only page. How long this takes is obviously heavily dependent on your traffic levels. We were able to get more than enough data to make a decision in 12 hours, but it would be no good making important decision based on the behaviour of 10 or 20 users.

Despite a few niggling issues, I'd have to say this is a remarkable service that should be a huge benefit to almost anyone coding pages.

No matter how much you think you know, this tool will very likely teach you something new.