“If you build it, they will come…”. That’s a classic line from Field of Dreams. While I love baseball, this isn’t a post about our great American sport and its glory. I’ve heard this quote it used on many occasions in casual conversation about things other than baseball. One example, is when building Internet sites. If you are planning to build a site, it’s an essential task to test your site for the type of volume you expect. After all, “If you build it, they will come…” A poor customer experience due to a slow web site, unavailable web site, or system error messages could mean lost business and a bad reputation.
There are three types of volume testing that are commonly used to assist with web site development:
Performance Testing
The primary purpose of performance testing is to find system bottlenecks. This type of testing typically involves software tools that can simulate concurrent users on the system. The response time of the system is compared against the goals for such things as page response, query response, etc. If there are discrepancies between the time recorded and the goal, then one should examine the performance of individual components in the system to determine the bottlenecks.
Another purpose of performance testing is to record a baseline of system performance for key transactions (login, order history lookup, order submit, etc.). After you establish the performance baseline, you can use it with future releases to determine if the release is improving, decreasing, or holding the existing performance bench mark. Certainly any release that decreases performance would be a concern and may require adjustments before releasing to the production environment.
Load Testing
Load testing is intended to simulate the highest expected load a system is expected to handle. Using automated testing tools, the highest expected load is setup in the test plan to run against the system. This could involve a certain number of concurrent users, a specific number of open database connections, or a set number of transaction calls. The goal of load testing is not to break the system, but to observe how the system operates under a constant maximum load. This type of testing could reveal slow memory leaks that result in gradual system performance degradation.
Stress Testing
Stress testing tries to break a system by giving it more or less inputs than it can process. There are two main goals with this type of testing:
1. Record at what level the system breaks. This gives you an idea of your maximum load and can be used for pro-active system sizing.
2. Make sure that the system has a graceful fail and recovery mechanism. This could be the result of adding too many inputs (say concurrent users) than the system can handle, or it could be the result of taking something away from the system such as its database. If you take away the database for a brief time how does the system notate this response to the users and does it recover automatically when the database is active again?
How does this relate to Customer focused eCommerce?
Have you ever been on a site that experienced one of these problems?
- Page does not respond
- HTTP 500 error
- Java.Lang…. error
- Error message that indicates the system has reached capacity
- Timeout waiting for response for server
Those are not good feelings and can often be very frustrating if you’ve invested time in a shopping, registering, or searching for information. In many cases, these types of errors could be found by following the volume testing techniques mentioned above. Focus on your customers, by making sure your site is tuned and available. They’ll be more likely thank you with their business and come back the next time they need your product or service.
Well done Bob! Good thoughts on volume testing.
As an e-Commerce Marketing executive, do you have any statistics or supporting data regarding the financial impact of site failure? For example, the wasted marketing dollars when the site goes down under too many concurrent users, especially when caused by a splendid marketing campaign that is wildly successful in driving traffic.
It is great to have your perspective because most of the people I interact with regarding load testing are developers or QA professionals. I put up a blog post pointing to this article. Feel free to check for any errors or misquotes. http://loadstorm.com/2009/performance-testing-ecommerce-perspective
Thank you for sharing.
Hi Scott. Site failure impact would be determined by several factors including:
* Amount of downtime
* Average orders received during that day of the week and time of day
* Average value of the orders received during that time
* Marketing campaigns that promote special options on the site during the time. This could add traffic above established baseline. Use an expected return rate on the number of promotional ads sent to people.
* The likelihood that customers will wait-out the failure and return instead of going to a competitor.
* Average number of new visitors to a site during that day of the week and time. The outage could have a residual effect because if prospective buyers have a bad experience on your site the first time they use it, then they won’t likely return.
The bottom two statistics may be harder to quantify and defend. However, I would expect with standard web analytics tools and financial reporting you could multiply out these statistics to show the value in minimizing risk through some form of volume testing.