Performance Engineering and Scaling¶
Vista Takes the Fast Lane
"Fellow architects, welcome to the chapter where we stop guessing about performance and start knowing! From up here I can see something beautiful: every team that rigorous about performance measurement ships systems that actually delight their users — and every team that skips it spends their weekends triaging production fires. ATAM's superpower is that it forces the performance conversation before the code is written. Let's weigh the tradeoffs!"
Summary¶
Performance is consistently one of the top quality attributes in ATAM evaluations, yet it is routinely misunderstood — confused with scalability, measured too late, or addressed with the wrong tactics. This chapter gives students a rigorous foundation in performance engineering: defining latency, throughput, and response time precisely; understanding Amdahl's law and its implications for scaling limits; and applying profiling, load testing, and stress testing to locate performance bottlenecks. The chapter then covers capacity planning and the scaling toolkit — vertical scaling, horizontal scaling, database sharding, connection pooling, and CDN — before introducing observability as the measurement infrastructure that makes performance engineering evidence-based.
The Performance Imperative in ATAM¶
Performance scenarios appear in virtually every ATAM evaluation, and for good reason. A system that fails its performance objectives fails its users — regardless of how elegant the architecture is or how clean the code. Amazon's famously cited research suggested that every 100ms of additional page latency cost them roughly 1% in sales. Google found that a half-second delay in search results caused traffic to drop by 20%. These are not abstract statistics; they represent the organizational pain that ATAM evaluations are designed to surface and address before deployment.
Yet performance engineering suffers from a peculiar professional blind spot: teams frequently treat it as an afterthought, addressed only when users complain or when load tests run the night before a launch. The ATAM perspective reframes this entirely. When an evaluation team examines architectural decisions through the lens of performance quality attribute scenarios, they are asking a fundamentally different question than "did the load test pass?" They are asking: "which architectural decisions create performance risk, and are those risks understood and accepted?" This distinction — from pass/fail testing to risk-informed decision-making — is the intellectual contribution that makes ATAM analysis so valuable.
This chapter builds the vocabulary and conceptual toolkit that performance ATAM scenarios require. Every concept from latency to observability will be examined not just as a technical definition but as a lens for architectural evaluation.
Latency, Throughput, and Response Time: The Performance Vocabulary¶
Before any meaningful performance analysis can occur, the team must agree on terminology. The most common source of confusion in performance discussions is the conflation of three distinct but related concepts: latency, throughput, and response time.
Latency is the time required for a single unit of work to pass through a specific system component or network segment. It is a one-way measurement: the time from when a packet leaves a sender to when it arrives at a receiver, or the time from when a database query reaches the database engine to when the engine begins returning results. Latency is inherently about the wire or the processing unit, not about the complete round-trip experience.
Response time (sometimes called end-to-end latency or elapsed time) is what users actually experience: the total elapsed time from when they initiated a request to when they received a complete, usable response. Response time is a composite measurement that includes every source of latency along the entire request path — client-side rendering, network transmission to the server, any queuing delay, actual processing time, database round-trips, network transmission back to the client, and final rendering. Response time is the number that appears in user stories ("the search results page shall load within 2 seconds at the 95th percentile under normal load") and in ATAM quality attribute scenarios.
Throughput is the rate at which a system processes work over a given time window, typically expressed as requests per second (RPS), transactions per second (TPS), or messages per second (MPS). Throughput is a capacity measure: it describes how much the system can do, not how long any individual operation takes. A system can have excellent latency on individual requests but poor throughput if it cannot handle concurrent users efficiently — and conversely, a high-throughput system might exhibit poor latency under sustained load due to queuing effects.
The Latency-Throughput Tension Is a Classic Tradeoff
From up here I can see one of architecture's most delightful paradoxes: improving throughput often hurts latency, and vice versa. Batching operations increases throughput by amortizing fixed costs, but the first item in a batch waits for the batch to fill. Caching improves latency but limits throughput for write-heavy workloads. This is exactly the kind of tension that ATAM tradeoff analysis lives for — "Let's weigh the tradeoffs!" is never more literal than in performance architecture.
Understanding the relationship between latency and throughput requires introducing Little's Law, one of the most powerful equations in systems performance analysis. Little's Law states that for any stable system in steady state:
Where L is the average number of requests in the system (including those waiting and those being processed), λ is the average arrival rate (throughput), and W is the average time a request spends in the system (response time). Little's Law is distribution-independent — it holds for any arrival pattern, any service time distribution, and any queuing discipline. Its implication for architecture is profound: if you want to reduce response time W, you must either reduce the number of in-flight requests L (by limiting concurrency) or increase system throughput λ (by scaling). These are the two fundamental levers.
A second critical concept is the percentile model of latency distribution. Performance requirements should never be specified as averages alone, because averages mask the tail behavior that actually determines user experience. A system with average response time of 50ms but a 99th percentile of 5 seconds is a system that produces angry users 1% of the time — which at scale means thousands of bad experiences per minute. ATAM quality attribute scenarios should always specify a percentile: "95th percentile response time shall not exceed 200ms under a load of 1,000 concurrent users."
Diagram: Performance Metric Relationships¶
Performance Metric Relationships Explorer
Type: Interactive simulation
sim-id: performance-metrics-explorer
Library: p5.js
Status: Specified
Purpose: Visualize the relationships between latency, throughput, response time, and concurrency using Little's Law. Users manipulate arrival rate and service time sliders to see how all metrics change.
Controls: - Arrival Rate slider (requests/second): 10–1000 - Service Time slider (ms): 1–500 - Concurrency Limit slider: 1–100
Display panels: - Live Little's Law calculation showing L, λ, W - Latency distribution histogram (p50/p95/p99 markers) - Throughput gauge - Queue depth indicator
Behavior: When arrival rate × service time > concurrency limit, the queue grows visually and p99 latency spikes dramatically — illustrating queue saturation.
Amdahl's Law and the Limits of Scaling¶
Every architect who has ever heard "just throw more servers at it" needs to understand Amdahl's Law. Formulated by Gene Amdahl in 1967, this deceptively simple equation has enormous practical consequences for scaling architecture decisions.
Amdahl's Law states that the maximum theoretical speedup S achievable by parallelizing a computation is bounded by the fraction of the work that cannot be parallelized:
Where p is the fraction of the task that can be parallelized, n is the number of parallel processors (or, by analogy, servers), and S(n) is the resulting speedup. As n approaches infinity, the speedup converges to 1/(1-p). If 80% of a workload can be parallelized (p=0.8), the maximum possible speedup — with infinite servers — is 5×. Adding servers beyond a certain point produces diminishing returns so severe that the cost-benefit ratio becomes absurd.
The practical architectural implications of Amdahl's Law are profound. Every shared resource that creates serialization — a global lock, a single-writer database, a centralized session store, a synchronous external API call — constitutes a non-parallelizable fraction that caps your scaling potential. This is why shared mutable state is so dangerous in distributed systems, why read replicas dramatically improve read scalability while write scaling remains constrained, and why "share-nothing" architectures (each node has its own data and state) achieve near-linear horizontal scaling while "share-everything" architectures hit Amdahl's wall quickly.
Gustafson's Law provides a complementary perspective that is equally important. Where Amdahl's Law fixes the problem size and increases processors, Gustafson's Law observes that in practice, when we get more computing power, we use it to solve larger problems. If the amount of work grows proportionally with the number of processors, the speedup can scale linearly. This is the regime in which truly horizontally scalable systems operate: adding more nodes doesn't just handle the same load faster, it enables the system to handle proportionally more load.
Amdahl's Law Is Your ATAM Early Warning System
When you are in an ATAM evaluation and a team proposes "horizontal scaling" as the tactic for meeting a performance scenario, ask one question: "What is your non-parallelizable fraction?" If they go quiet, you have found a sensitivity point. A shared relational database, a centralized message broker, or a distributed lock manager can all cap your scaling headroom at a fraction of what the team expects. Amdahl's Law makes this risk quantifiable.
Performance Profiling: Finding the Real Bottleneck¶
The first rule of performance engineering is "measure before you optimize." The second rule is "measure the right thing." Performance profiling is the discipline of systematically measuring where a system spends its time and resources, with enough granularity to identify the components that are responsible for performance problems.
A performance bottleneck is a component or resource that limits overall system throughput or degrades response time beyond acceptable levels. Bottlenecks are not always where developers expect them to be — a commonly cited industry observation is that 80-90% of a system's performance is determined by 10-20% of its code, and that the specific 10-20% is rarely intuitive without measurement data.
Profiling tools generally operate in three modes:
Sampling profilers periodically interrupt execution (typically thousands of times per second) and record the current call stack. Statistical analysis of the resulting samples produces a "hot path" — the call chain where the program spends the most time. Sampling profilers have low overhead (typically 1-5%) and are suitable for production use, but they miss short-duration calls that do not happen to be active during a sample.
Instrumentation profilers insert measurement code at every function entry and exit, capturing exact call counts and elapsed times. They provide complete coverage but carry substantial overhead (often 2-10× slowdown) that limits them to development and staging environments.
Distributed tracing systems (covered in depth in Chapter 16) propagate context headers through service boundaries, allowing end-to-end response time to be decomposed into per-service, per-database, and per-network contributions. For distributed systems, distributed tracing is the only way to identify which component in a multi-service call chain is the bottleneck.
Before a profiling session, effective performance engineers form a hypothesis. Based on understanding of the system architecture, they predict where time is being spent: "I expect database queries to account for 60% of this request's latency." Forming the hypothesis first prevents the analyst from accepting the first plausible explanation and stopping — the true bottleneck often sits one level deeper than the first anomaly the data reveals.
Load Testing and Stress Testing¶
With profiling identifying where time is spent in the system, load testing and stress testing answer the questions of how well the system holds up under realistic and extreme conditions.
Load testing subjects the system to a realistic production-like workload to verify that it meets its performance quality attribute scenarios. A well-designed load test specifies the arrival rate (requests per second), the workload mix (proportions of each request type), the test duration (sufficient for steady-state behavior to emerge — typically 30 minutes minimum for meaningful results), and the success criteria (response time percentiles, error rate thresholds).
The workload model is critical. Tests that use uniform request types in constant arrival rates are easy to implement but bear little resemblance to real traffic. Real workloads exhibit diurnal patterns (morning peaks, midday lulls, evening surges), bursty behavior around events, and heavy-tail distributions in request cost (most requests are fast, a few are extremely expensive). Sophisticated load testing tools such as k6, Gatling, and Apache JMeter support scripted scenarios that more faithfully replicate these patterns.
Stress testing deliberately pushes the system beyond its expected operating envelope to identify failure modes and degradation behavior. The goal is not to verify that the system meets its scenarios — it is to understand how the system fails when scenarios are violated. Does the system gracefully shed load (returning HTTP 503 responses with retry-after headers, or routing to a queue), or does it fail catastrophically (memory exhaustion, cascading database connection failures, OOM kills)? A system with good failure behavior degrades gracefully and recovers when load subsides; a system with poor failure behavior has a much narrower operational envelope.
The Load Test That Isn't
A common anti-pattern in ATAM evaluations is teams presenting load test results that used a single test client on the same machine as the server being tested. In this configuration, the test client and server compete for CPU, memory, and network bandwidth — so the test proves only that the system can survive running alongside a process that competes with it. Effective load tests use distributed load generators from separate machines or cloud regions, and should include network latency simulation representative of actual client geography.
Capacity Planning¶
Load and stress testing tell you how the system performs now, under controlled conditions. Capacity planning answers a different question: how long will the current architecture support growth, and when will investment in additional capacity or architectural changes become necessary?
Capacity planning begins with understanding the relationship between business metrics and system resource consumption. The key quantities to model are:
Resource utilization curves: For each key resource (CPU, memory, database connections, disk I/O, network bandwidth), how does utilization change as a function of load? Most resources exhibit approximately linear utilization up to some saturation point, followed by rapid degradation. Measuring the slope of the linear region (e.g., "2.5% CPU per 100 RPS") and the saturation point (e.g., "70% CPU at 2,800 RPS") gives a direct model for current capacity.
Growth projections: Using business forecasts, traffic modeling, or the historical trend of load metrics, project when current capacity will be exhausted. A simple model might be "traffic has been growing 15% month-over-month; at this rate, we saturate database CPU in approximately 5 months."
Scaling trigger thresholds: Rather than running to exhaustion, effective capacity planning defines triggers — resource utilization levels at which scaling actions should be initiated — that provide enough lead time for the scaling action to complete before capacity is exhausted. For cloud infrastructure, this might be "add nodes when CPU sustains above 65% for 5 minutes"; for procurement-constrained on-premise infrastructure, the trigger must account for weeks or months of procurement lead time.
The output of capacity planning is an architectural decision: which resources will become constrained first, when they will hit their limits, and what the options are for addressing each constraint. This is precisely the kind of analysis that feeds ATAM architectural risk identification.
The Scaling Toolkit¶
When capacity planning identifies a constraint, the architect must choose from a set of scaling tactics, each with distinct tradeoffs across performance, cost, complexity, and operational burden.
Vertical scaling (also called scaling up) replaces a constrained component with a larger instance: more CPU cores, more RAM, faster disks. Vertical scaling has the enormous advantage of simplicity — the application code does not need to change, no new architectural complexity is introduced, and the operational model remains the same. The disadvantages are its cost curve (larger instances are disproportionately expensive — doubling CPU often triples cost), its hard ceiling (there is a maximum instance size available at any price), and the operational risk of keeping all workload on a single machine (no fault isolation).
Horizontal scaling (also called scaling out) adds more instances of a component to distribute load. Horizontal scaling overcomes the cost and ceiling disadvantages of vertical scaling and provides fault isolation through redundancy. However, it introduces significant architectural complexity: requests must be routed across instances (load balancing), state must be externalized rather than kept in instance memory, and distributed coordination problems emerge wherever previously-local operations were assumed to be atomic.
The interaction between horizontal scaling and Amdahl's Law is particularly important. If the workload being scaled contains synchronization bottlenecks — a shared database writer, a global lock, a single-threaded coordinator — those bottlenecks cap horizontal scaling benefit just as Amdahl predicts. The prerequisite for effective horizontal scaling is a share-nothing or share-read-only architecture where each instance can process requests independently.
Diagram: Vertical vs. Horizontal Scaling Tradeoffs¶
Vertical vs. Horizontal Scaling Tradeoffs Explorer
Type: Interactive simulation
sim-id: scaling-tradeoff-explorer
Library: p5.js
Status: Specified
Purpose: Side-by-side comparison of vertical and horizontal scaling strategies showing cost curves, capacity ceilings, and fault isolation properties under different workload models.
Controls: - Workload type selector: stateless, stateful, database-write-heavy - Growth rate slider: 5%–50% monthly - Current load slider - Parallelizable fraction slider (Amdahl's p): 0.5–1.0
Display: - Left panel: vertical scaling — instance size vs. cost curve with saturation point - Right panel: horizontal scaling — nodes vs. throughput curve with Amdahl's wall - Bottom: time-to-saturation for each strategy at current growth rate - Recommendation panel: "given these parameters, horizontal scaling hits Amdahl's wall at N nodes"
Database Sharding¶
The database tier is frequently the first architectural component to become a scaling bottleneck, because traditional relational databases are designed around strong consistency guarantees that require centralized coordination. Database sharding is the technique of partitioning data horizontally across multiple database instances, each of which owns a subset of the total data, in order to distribute both storage and query load.
A shard is a horizontal partition of the database — a subset of rows or documents that is stored on a specific database instance (called a shard server). The shard key is the attribute used to determine which shard owns a given piece of data: all rows with the same shard key value reside on the same shard, and routing logic (implemented in the application layer or in a middleware proxy) directs queries to the appropriate shard based on the key in the query.
Shard key selection is one of the most consequential and irreversible architectural decisions in a sharded system. A good shard key distributes data evenly across shards (avoiding "hot shards" that receive disproportionate traffic), co-locates data that is frequently accessed together (avoiding cross-shard joins), and is stable enough that re-sharding (redistributing data when the shard key is changed) is rarely necessary.
Common shard key patterns include hash-based sharding (applying a hash function to a natural key like user ID, which distributes data uniformly but prevents range queries), range-based sharding (partitioning on an ordered key like creation timestamp, which supports range queries but risks hot shards for time-series data), and geographic sharding (assigning users in each region to a region-local shard, which reduces latency but may create uneven load).
The primary tradeoffs of sharding are significant. Cross-shard joins are expensive (requiring scatter-gather queries that fan out to all shards and aggregate results) or impossible (requiring application-level joins after fetching from each shard). Distributed transactions that span multiple shards require two-phase commit or Saga patterns (discussed in Chapter 12), which are complex and carry performance overhead. Schema changes must be applied to all shards consistently. These costs make sharding a tactic of last resort for relational data — teams should exhaust read replicas, caching, and query optimization before accepting the operational complexity of sharding.
Sharding Is an Architectural Commitment, Not a Tactic
Let me be direct from the high-level view: once you shard a production database, you have fundamentally changed the data model in a way that is extremely difficult to reverse. ATAM evaluations should examine sharding proposals carefully, because teams often underestimate the long-term operational cost. The question to ask is not "will sharding solve our current performance problem?" but "is the performance problem severe enough that accepting permanent sharding complexity is the right tradeoff?" Sometimes it is. Often a better shard key, a read replica, or an application-level caching layer eliminates the need entirely.
Connection Pooling¶
Establishing a database connection is expensive: it involves TCP handshaking, authentication negotiation, session state initialization, and allocation of server-side resources. For an application that processes thousands of requests per second, the overhead of establishing a new connection per request would be prohibitive. Connection pooling addresses this by maintaining a pool of pre-established, reusable connections.
A connection pool maintains a configurable number of database connections in an idle state, ready to be borrowed by application threads as needed. When a thread needs database access, it borrows a connection from the pool; when the operation completes, the connection is returned to the pool rather than closed. The pool maintains minimum (always-warm) and maximum (capacity ceiling) connection counts and a timeout for threads waiting for an available connection.
The critical performance parameter is pool size. A pool that is too small causes request threads to queue waiting for connections — creating artificial latency under moderate load. A pool that is too large exhausts database server resources, as each active connection consumes server memory, file descriptors, and sometimes locks.
The formula for optimal pool size is not "as many as possible." Database server performance research (notably the HikariCP documentation by Brett Wooldridge) demonstrates that the number of connections should be sized to the number of effective computing cores available to the database, not to the number of application threads. For a database with 8 CPU cores: pool_size = (core_count * 2) + effective_spindle_count. This formula accounts for the fact that database operations involve both CPU processing and I/O waits, and that having more connections than cores creates context-switching overhead that outweighs the benefit of additional concurrency.
CDN Architecture¶
Content Delivery Networks address a specific and ubiquitous performance problem: the speed of light. Network latency has a hard physical floor determined by the distance between client and server. A request from a user in Tokyo to a server in Virginia must travel approximately 10,000 kilometers, imposing a minimum round-trip latency of approximately 67 milliseconds regardless of how fast the server processes the request. CDNs solve this by deploying caching infrastructure at hundreds of Points of Presence (PoPs) distributed geographically close to end users.
When a user requests content served via a CDN, the DNS system routes their request to the nearest PoP rather than the origin server. If the PoP has a valid cache entry for the requested content (a cache hit), it serves the content directly from its local storage — potentially from a data center within 20 milliseconds of the user — without involving the origin server at all. If the PoP lacks a cache entry (a cache miss), it fetches the content from the origin server, caches it locally with a configured TTL (Time to Live), and serves it to the user.
CDNs are primarily valuable for static and semi-static content: images, CSS, JavaScript bundles, fonts, video streams, and API responses that are identical for all users. Dynamic, personalized content (user-specific data, real-time prices, authentication states) generally cannot be CDN-cached and must be served from origin. Modern CDNs have extended their capabilities to include edge computing — running application logic at the CDN PoP — which allows some dynamic computations to be performed close to the user rather than at the origin.
The CDN cache hit ratio is the key performance metric: what fraction of requests are served from the CDN cache without reaching origin? A hit ratio below 80% suggests that TTL configurations, cache key design (query parameters may be unnecessarily invalidating caches), or content characteristics (highly personalized content is being incorrectly routed through CDN) need review.
Diagram: CDN Request Routing and Cache Flow¶
CDN Architecture and Cache Flow Explorer
Type: Interactive diagram
sim-id: cdn-architecture-explorer
Library: vis-network
Status: Specified
Purpose: Interactive map showing CDN PoP locations, origin servers, and request routing. Users click on simulated user locations to see which PoP handles their request and the latency comparison (with CDN vs. direct to origin).
Nodes: - Origin server (central, labeled with data center location) - 8 CDN PoPs (distributed geographically — Americas, Europe, Asia-Pacific) - 5 simulated user locations
Interactions: - Click user node → animated request path to nearest PoP → cache hit/miss decision → response path back - Toggle "CDN Enabled/Disabled" to show direct origin routing with latency comparison - Cache hit rate display updates as different users/request types are simulated
Observability: The Foundation of Evidence-Based Performance Engineering¶
All of the preceding performance tactics — profiling, load testing, capacity planning, sharding, pooling, CDN — are only as effective as the measurement infrastructure that reveals whether they are working. Observability is the property of a system that determines how well its internal state can be inferred from its external outputs. A highly observable system provides rich, correlated, and queryable signals that allow engineers to ask and answer arbitrary questions about system behavior — including questions they did not anticipate when they designed the monitoring.
The three pillars of observability are metrics, logs, and traces (discussed in depth in Chapter 16). For performance engineering specifically, the most critical of these are metrics — time-series numeric measurements that quantify system behavior over time — and distributed traces — causal chains of events that show how a request flows through a distributed system and where time is spent.
The USE Method (Utilization, Saturation, Errors), developed by Brendan Gregg, provides a systematic framework for resource-level performance analysis. For each resource (CPU, memory, network interface, disk, database connection pool): - Utilization: what fraction of the resource's capacity is being used? (60% CPU) - Saturation: is there a queue forming for the resource? (request queue depth > 0) - Errors: are there error conditions associated with the resource? (OOM kills, connection pool timeouts)
A resource exhibiting high utilization, non-zero saturation, and increasing errors is a performance bottleneck by any definition.
The RED Method (Rate, Errors, Duration) provides the complementary service-level perspective. For each service or endpoint: - Rate: how many requests per second is this service processing? - Errors: what fraction of requests result in errors? - Duration: what is the p50/p95/p99 latency of requests to this service?
Together, USE and RED provide the dual instrumentation model that maps ATAM performance scenarios to observable system behavior. When a quality attribute scenario states "the search service shall respond at p95 ≤ 200ms under 1,000 RPS normal load," the RED method immediately tells you which metrics to instrument, which dashboards to build, and which alert thresholds to set.
Performance Engineering Feels Overwhelming at First
There is a lot of material in this chapter, and I see some of you thinking "where do I even start?" Here is the secret: start with one scenario. Pick the most important performance quality attribute scenario from your utility tree — the (H,H) priority item — and instrument exactly and only for that scenario. Get it measured, get it under control, get it into your CI pipeline as an automated fitness function. Then pick the next one. Performance engineering is not a grand unified project; it is an accumulation of well-instrumented scenarios.
Performance Scenarios in ATAM Evaluations¶
The full performance concept toolkit becomes most powerful when applied within the structured context of ATAM scenario analysis. Let's trace how an ATAM team would apply this chapter's concepts to a concrete evaluation.
Suppose the system under evaluation is a high-frequency trading (HFT) order execution platform. The evaluation team has elicited the following performance scenario:
Stimulus: A burst of 5,000 concurrent order submissions arrives simultaneously during peak market volatility Environment: Normal market hours, order book at 80% capacity Artifact: Order execution pipeline (submission → validation → matching → confirmation) Response: All valid orders are processed without loss Response measure: p99 order-to-confirmation latency ≤ 10ms; zero order loss; queue depth returns to zero within 500ms
Analyzing this scenario through the chapter's framework reveals multiple architectural concerns. First, the 10ms p99 requirement implies an extremely tight budget across the entire call chain — each hop (network, validation, matching engine, persistence) must be profiled to identify where within this budget the time is being spent. Second, the "5,000 concurrent" stimulus triggers Amdahl's Law analysis: if the matching engine serializes access to the order book through a lock, the parallelizable fraction p may be very low, capping throughput regardless of horizontal scaling. Third, the "queue depth returns to zero within 500ms" requirement implies a specific burst absorption capacity that depends on both throughput and the service time of each order.
The ATAM team would classify the matching engine lock as a sensitivity point (small changes in lock granularity dramatically affect both latency and throughput), the choice between lock-free data structures and mutex-based synchronization as a tradeoff point (lock-free structures improve throughput under contention but are far more complex to implement correctly), and the absence of p99 latency monitoring as an architectural risk (the team cannot verify the scenario requirement without it).
You Now Think Like a Performance Architect
Look at that — you have just analyzed a performance scenario using Amdahl's Law, Little's Law, the USE and RED methods, sensitivity point identification, and tradeoff analysis all in one pass. This is what performance engineering looks like when it grows up. You are not just tuning servers; you are doing rigorous architectural analysis. Let's weigh those tradeoffs, fellow architects! You've earned this view from the top.
Concepts Covered¶
This chapter covers the following 15 concepts from the learning graph:
- Latency — one-way transit time through a component or network segment; distinct from response time
- Throughput — the rate at which a system processes work; measured in RPS, TPS, or MPS
- Response Time — the complete end-to-end elapsed time experienced by a user or caller
- Performance Bottleneck — the component or resource that limits overall system performance
- Amdahl's Law — the mathematical limit on parallelization speedup due to non-parallelizable fractions
- Performance Profiling — systematic measurement of where a system spends time and resources
- Load Testing — verification testing against realistic production-like workloads
- Stress Testing — deliberate overload testing to identify failure modes and degradation behavior
- Capacity Planning — projecting resource consumption growth to determine when scaling is required
- Vertical Scaling — addressing capacity constraints by replacing components with larger instances
- Horizontal Scaling — addressing capacity constraints by adding more instances of a component
- Database Sharding — horizontal partitioning of data across multiple database instances
- Connection Pooling — reusing pre-established database connections to amortize connection overhead
- CDN Architecture — geographically distributed caching infrastructure for latency reduction
- Observability — the property that enables inference of internal system state from external signals
Prerequisites¶
This chapter builds on concepts from:
Self-Check Questions¶
Self-Check: Performance Fundamentals — Click to Reveal Answers
Q1: A web service has an average response time of 50ms and an average of 200 concurrent in-flight requests. What is its throughput according to Little's Law?
Answer: Little's Law: L = λ × W, so λ = L/W = 200 / 0.050s = 4,000 requests per second. This illustrates how Little's Law lets you compute throughput from response time and concurrency measurements alone.
Q2: A system's workload is 75% parallelizable (p = 0.75). What is the maximum speedup achievable by adding unlimited parallel processors? What speedup do you get with 4 processors?
Answer: Maximum speedup (n → ∞): S = 1/(1-0.75) = 1/0.25 = 4×. With 4 processors: S(4) = 1/((1-0.75) + 0.75/4) = 1/(0.25 + 0.1875) = 1/0.4375 ≈ 2.3×. Even with 4 processors you only get 2.3× of the theoretical 4× maximum, because the serial fraction creates a queuing bottleneck.
Q3: You are evaluating an architecture with a database connection pool sized at 200 connections for an application running on a server with 8 CPU cores. What concern would you raise?
Answer: This is significantly over-sized. The HikariCP formula suggests (8 × 2) + spindle_count connections, probably 16-20 for this server. With 200 connections, the database server is handling 200 concurrent sessions on 8 cores, which creates excessive context-switching overhead. The counter-intuitive result is that reducing the pool size would likely improve throughput by reducing context switching on the database server.
Q4: In an ATAM evaluation, a team proposes horizontal scaling as the tactic to meet a performance scenario requiring 10× throughput improvement. What follow-up analysis would you conduct?
Answer: Apply Amdahl's Law analysis: identify all serialized, non-parallelizable components in the request path. Key questions: Does the system have a single-writer database? Global session stores? Distributed locks? Synchronous external service calls? For each identified serialized component, estimate its fraction of total request time. Calculate whether the remaining parallelizable fraction is sufficient to achieve 10× speedup. If any single component accounts for more than 10% of request time, it caps horizontal scaling at 10× regardless of how many nodes are added.