Intro
Performance testing is not unlike other types of testing and should follow defined and manageable practices keeping pace with changing applications, infrastructure, and workload. A six-point model to maximize performance testing’s value is presented here as an overview of conceptual essentials.

1. Objectives
Like any task, knowing its objectives is the necessary first step and performance testing’s value likely will not be fully realized unless objectives are defined and against which results are gauged. Performance objectives are ultimately measurements and should begin with overarching, stakeholder-approved key performance indicator (“KPI”) thresholds and refined as circumstances warrant. Although some aspects may lean toward subjectivity, such as end-user satisfaction, it’s likely even these may be quantified to a meaningful degree.

2. Types & Roles
There are numerous types of performance tests but consider the following two categories:

• Continuous: Continuous performance testing is tantamount to continuous functional or security testing and consistent with test-driven cycles: a CI/DevOps effort measuring change early and often and catching and reporting problematic deviation.
• Periodic/Event-driven: Periodic testing is driven by specific actions, like release milestones or ad hoc events that, in any case, warrant formal load or benchmark testing to gauge performance readiness under deployed conditions.

Roles and responsibilities may overlap categories and types, but as a practical matter continuous testing is normally DevOps’ realm while periodic is reserved for performance engineers. Regardless, and in this post’s conceptual context, “who” is less relevant than “what” and “how”.

3. Tooling
Organizations satisfied with existing performance test tooling should retain them, especially when accompanied by established and thorough procedures and test cases/scripts.

If, however, tool selection is under consideration, off-the-shelf tools, while plentiful, differ greatly and the following lists a few (likely familiar) evaluation criteria:

• Capabilities: Do the tool’s features meet the application’s use cases and architecture both now and for what’s reasonably planned?
• Usability: How usable is the tool for test case maintenance and configuring and running tests?
• Manageability: What does the tool’s test case/script maintenance involve? Does creating a typical new test case take 20 minutes or a week? Is the tool’s language domain-specific or general?
• Portability: is the tool capable of both CI and periodic testing while relying on the same “write once” test case base?

4. Metric Data
Metric data includes “real time” and “post-test” with the following logical differences:

• Real time: Real time metric data is just that–a running test’s state and, regardless of CI/automated or periodic, drives determining if the test is running successfully or should be stopped. Real time metric sources should therefore be lightweight–enough gauges to signal stopping the test and no more. Actual performance results are necessarily post-test since only then do we have the full data set.
• Post-test: Post-test metric data is also “just that”– analyzed and reported data following test completion, but it obviously must be persisted, whether to simple, flat files like HTTP logs (advice: rely on delimited, time series flat files, not formats intended to represent objects, like JSON or XML), or a log ingestion or performance observability solution.Generally, as a test program iterates and matures, “less becomes more”: initial, widely-cast metric nets are narrowed as some metrics are deemed irrelevant to KPIs, others acquired from different sources are logically redundant, and so on. Ultimately, the ideal balance is one that gathers KPI data (including factors influencing KPIs) with the fewest sources thus facilitating analysis.

5. Validity
The importance of test validity cannot be overstated since an invalid test necessarily produces invalid results, is costly, and may result in detrimental decisions, production “surprises”, and poor user experiences. Generally, and especially for periodic testing, the closer the system and test configuration reflect deployed usage, the higher the confidence and, therefore, test validity.

Numerous factors affect test validity and, regardless of the test type, should be honed to maximize confidence commensurate with test objectives. Some variables affecting test validity include:

• Use cases/scenario coverage
• System configuration
• Test configuration (e.g., concurrency, use case weighting, think time, etc.)
• Platform and infrastructure (e.g., network latency)
• Skewing processes

Practices further promoting validity include recording the system’s configuration and state between tests, and especially for periodic testing, following pre-flight checklists before hitting the “start test” button (lest we risk hitting the “stop test” button too often…).

6. Analysis
All test metrics should be considered relevant, but KPIs should be analyzed first highlighting findings not meeting objectives. Similar to the test itself, analysis specifics vary according to the test type, test results, and the system, but include:

• Reliance on simple representations for KPIs; only a handful is needed to convey important behavior–anything from requests per second to round-trip-time to even network traffic anomalies–and within reach of most audiences. Advanced statistics or data science is rarely necessary for performance data analysis (or even applicable).
• Automation for efficiency, consistency, and, ultimately, accuracy.
• Presenting and reviewing results in aggregated (preferably percentiles), visual, and easily-interpreted formats (see, for example, Visualizing Performance with Cumulative Distribution):
  • Simple indicators for what met objectives.
  • Simple indicators for what did not meet objectives including potential bottlenecks and optimization options.
• For periodic tests not meeting objectives (and assuming CI objectives had been met), performance engineers should present anomalies to the appropriate team(s).

Final Thoughts
Echoing the introduction, this presented a conceptual outline highlighting key areas for maximizing application performance testing’s value. Separate posts could easily expand one area or another, namely those requiring more detailed consideration, such as tool selection, test validity, and discrete analysis, but are very much context-specific.

However, addressing within your context is a simple matter of contacting us.