Performance Testing Software Systems Workshop Logo

Core Performance Testing Principle:

Criteria

Overview

Performance acceptance criteria are typically derived from five perspectives: system users, business owners of the system, technologies, compliance expectations, and the project team.

Determining the desired performance characteristics for a system from particular perspectives is only the first part of determining the overall performance acceptance criteria for that system. After examining the desired performance characteristics from limited perspectives, you must resolve those characteristics against one another. For example, end-users may desire sub-second response time for every transaction; business stakeholders may want the system to support millions of users; and compliance criteria may mandate strict security policies.

Individually, any of these characteristics may be achievable, but collectively they may not be possible due to time, technology, and/or budget constraints. Finding a way to achieve these desired characteristics presents team-wide challenges that should be addressed proactively, not reactively after conflicts become apparent through testing.

Terminology

  • Performance Acceptance Criteria – Performance testing objectives refer to data that is collected through the process of performance testing and that is anticipated to have value in determining or improving the quality

  • Performance Testing Objectives – Performance testing objectives refer to data that is collected through the process of performance testing and that is anticipated to have value in determining or improving the quality of the product. However, these objectives are not necessarily quantitative or directly related to a performance requirement, goal, or stated quality of service (QoS) specification.

  • Performance Objectives – Performance objectives are usually specified in terms of response times, throughput (transactions per second), and resource utilization levels. Resource utilization levels include the amount of processor capacity, memory, disk I/O, and network I/O that your application consumes.

  • Performance Requirements – Performance requirements are those criteria which are absolutely non-negotiable due to contractual obligations, service level agreements, or non-negotiable business needs. Any performance criteria that will not unquestionably lead to a decision to delay a release until the criteria pass is not absolutely required, and therefore, not a requirement.

  • Performance Goal – Performance goals are those criteria that are desired for release, but may be negotiable under certain circumstances. For instance, if a response time goal of three seconds is set for a particular transaction, but the actual response time is determined to be 3.3 seconds, it is likely that the stakeholders will choose to release the application and defer performance tuning of that transaction for a future release.

  • Performance Targets – Performance targets are the desired values for a resource of interest under a particular set of conditions, usually specified in terms of response times, throughput, and resource utilization levels. Resource utilization levels include the amount of processor capacity, memory, disk I/O, and network I/O that your application consumes.

  • Performance Thresholds – Performance thresholds represent the maximum acceptable value for a resource of interest, usually specified in terms of response times, throughput (transactions per second), and resource utilization levels. Resource utilization levels include the amount of processor capacity, memory, disk input/output (I/O), and network I/O that your application consumes.

  • Performance Budgets or Allocations – Performance budgets are your constraints. Performance budgets specify the amount of resources that you can use for specific scenarios and operations and still be successful.

Sources of Performance Acceptance Criteria

When identifying performance acceptance criteria, consider the following:

  • Business requirements

  • User expectations

  • Contractual obligations

  • Regulatory compliance criteria and industry standards

  • Service level agreements

  • Resource utilization targets

  • Various and diverse, realistic workload models

  • The entire range of anticipated load conditions

  • Conditions of system stress

  • Entire scenarios and component activities

  • Key performance indicators

  • Previous releases of the application

  • Competing applications

  • Optimization objectives

  • Safety factors, headroom and scalability

  • Schedule, staffing, budget, resources and other priorities

Learn more through PerfTestPlus Training

This topic is covered in the following PTP training courses:

Content adapted from:

Performance Testing Guidance for Web Applications

 



by: J.D. Meier, Scott Barber, Carlos Farre, Prashant Bansode, and Dennis Rea
Copyright © 2005-12 PerfTestPlus, Inc. All rights reserved.
Site Design by: Scott Barber