A Business Technology Place

A/B testing shows more than conversion rates

I recently tested two different versions of an updated landing page to a site that I manage. During the design process the usability analyst received feedback from people not associated with the site on several manual wireframe prototypes. With this input, she developed a couple of rough mockup designs using powerpoint and went through another round of user group feedback. Ultimately she narrowed down the designs to two leading options.

The two leading page designs were then developed and placed on the site as the new landing page in a 50/50 A/B split. Our initial thought is that we would compare each design against the baseline conversion rate established by the existing landing page (our control). What we found was the the conversion rate was flat to slightly higher for each design but that page B in our test had a higher instance of customers forgetting to select a required radio button choice.

Here’s a quick comparison of the three field design for options A and B. The page had other elements but the form layout varied by a horizontal and vertical stack of the required fields.

A/B testing form layout


So our conversion rate metric improved slightly but we learned that the usability of design A was much greater than design B. At a 2x rate users of the site would not choose a selection for the the radio button question on the form.

Usability is no stranger to A/B tests on eCommerce sites. It’s typical to see it with colors, button styles, form types, etc. In my case, I was so focused on the conversion rate metric prior to the release that I forgot about the usability test of the two design. The two options were setup perfectly for a usability test because their content was the same while we varied the layout pattern of the form on the page that has fields for user input.

Three golden nuggets for eCommerce financial metrics management

The ultimate goal of financial metrics in your eCommerce operation is to provide a measurement against revenue and profits required to sustain and grow your business. Choosing which financial metrics to measure can be a tricky endeavor. You want to make sure to capture relevant data that define your business and to use your time wisely. Common areas of measurement include revenue, average cart purchase, up-sell ratio, retail dollars, number of items in the cart, cost to operate, adword spend,and delivery dollars.

But what is financial metrics management really about and how should you structure this part of your eCommerce operation?

Use financial metrics that define the success of your business

Make sure the financial measurement points to a key performance indicator in your business. Remember too that areas that define the success of your business go beyond dollars spent. For example, it’s commonly documented that retail giant Costco makes the bulk of its profits on membership dues rather than merchandise sold. Costco retail margins are capped at a lower threshold than typical retailers which is the basis for it’s low-cost but high-quality reputation. While I don’t work for Costco, I can imagine that a critical financial metric for them is both memberships sold and renewed. Other possible metrics include:

  • Profit per order – Filling shopping carts is only beneficial if its done so at a profit.
  • Number of units sold – Important for businesses that need to move multiple units to turn a profit.
  • Average time of inventory before sell – Think about the implications of receiving your customer’s payment for a product before paying your supplier.
  • Percentage of inventory sold – Think about transportation providers and selling available seats for a specific trip
  • Revenue per order – If your cost structure is in good order and you are trying to show investors a growing revenue stream then you may be focused on the highest revenue possible
  • Revenue mix – You measure the revenue mix of eCommerce (Internet) against other available channels. This is important if you Internet channel has a higher profit per order than other channels so you may want to show the total contribution of profit of the channel.

It’s worthy to note here that it’s also important to focus on measurements of customer behaviors that lead to the dollars spent. This often gets into site measurement areas such as time on the site, delivery selection breakdown, and number of items in the cart. I consider these building blocks to the financial results, but that’s the topic for another session.

Use incremental financial metrics as a basis for enhancements justification

If you are a marketer or product manager then the financial metrics of your existing eCommerce property are the gold used to purchase new enhancements. In some cases you may have measurements on a pilot effort and are looking for the remainder of funding to complete the roll-out process of that feature. In other cases you may be able to show the existing reach and contribution of an eCommerce property to your business as a basis to request more funding for expansion of products and ideas. One fundamental concept to remember when creating new businesses cases is that you need to use the incremental financial contributions your change is forecasted to bring to the business. Your existing financial metrics baseline is already established. Don’t double count the financial contribution that would be present if your site remained as-is.

Increased financial metrics may trump decreased site metrics

After you release any new changes to an eCommerce property there are a number of measurements to gather to determine if the new release is successful. The ultimate scenario is that a new release increases both site metrics (conversion rates, number of visitors) and financial metrics. However, there are some situations where a decrease in site metrics may be off-set and even trumped by an increase in financial metrics. A simple example to consider is with a price increase or additive fee on a site. The resulting price increase may decrease the site’s overall conversion rate yet give it a stronger contribution towards financial goals. So remember to compare the resulting financial metrics against the established baseline after a release. Consider the long term implications of the results on both site metrics and financial metrics. This will guide your decision whether or not to keep the change.