Measuring Storage Performance: IOPS, Latency and Baselines

"The storage is slow" is one of the most common complaints in infrastructure and one of the least specific. Measuring it properly is what separates a hardware problem from a workload one, and the two have completely different remedies.

Three numbers, and they trade against each other

IOPS — operations per second. Matters for workloads doing many small reads and writes: databases, virtualisation hosts, anything with many concurrent users.

Throughput — data moved per second. Matters for large sequential work: backups, media, large file transfers.

Latency — how long an individual operation takes. This is what users actually experience, and it is the number most worth watching.

The relationship matters: pushing for maximum IOPS drives latency up. A system delivering high IOPS with terrible latency feels slow despite the impressive figure, which is why a single headline number tells you very little.

When someone says storage is slow, they almost always mean latency.

Vendor figures describe a different workload

Published performance numbers are measured under conditions chosen to produce a good number.

Typically that means a particular block size, a particular read-write mix, a queue depth chosen to saturate the device, and frequently sequential rather than random access.

Your workload is different. A virtualisation host produces heavily random I/O because many guests issue independent requests simultaneously — a completely different pattern from a single sequential stream. Our consolidation guide covers why that matters when sizing.

So a vendor figure is useful for comparing devices to each other and close to useless for predicting what you will see.

Measure the right thing

Four principles that make a measurement meaningful.

Match the pattern to your workload. Random or sequential, read or write mix, block size, and concurrency. Testing sequential reads on a system that does random writes tells you nothing about it.

Test at realistic concurrency. A single-threaded test on a system serving fifty users describes neither.

Run long enough to exhaust caches. Short tests measure cache rather than storage. On systems with substantial cache, a test that fits inside it produces excellent and meaningless numbers.

Record latency alongside IOPS, and preferably the distribution rather than the average. The occasional very slow operation is what users notice, and an average hides it.

Test on the actual configuration — same RAID level, same controller settings, same drives. A test on a bare drive does not describe the array it will live in.

Establish a baseline before you need it

The step that makes every later investigation faster.

Measure when the system is healthy and record it. When someone reports slowness later, you can compare against a known-good figure rather than arguing about whether the current number is bad.

Without a baseline, every performance discussion starts from opinion. With one, it starts from a delta.

The same principle our monitoring guide applies: trends are informative where single readings are not.

Common causes of a real slowdown

When performance has genuinely dropped, these account for most of it.

Failed cache battery. The classic. When cache protection fails, the controller falls back from write-back to write-through and write performance drops sharply — with no configuration change and no obvious error. Our controller guide covers this, and it should be the first thing checked on any sudden write slowdown.

A rebuild in progress. An array rebuilding after a drive failure is under sustained load and everything else suffers. Check array status before investigating anything else.

A degrading drive. A drive with rising reallocated sectors or retry counts slows the whole array because operations wait on it. Our lifespan guide covers the attributes that indicate this.

The array approaching full. Performance degrades before capacity actually runs out, which our capacity guide covers.

Workload change. Frequently the real answer — more users, a new application, a backup schedule that now overlaps the working day. Nothing is broken; the demand grew.

What the numbers should change

Measurement is only useful if it leads somewhere.

High latency with low utilisation suggests a specific component problem — a failing drive, a cache issue — rather than insufficient capability.

High latency with high utilisation means the storage is doing as much as it can. That is a sizing conversation: more spindles, faster media, or moving hot data onto flash. Our SSD versus HDD guide covers where flash genuinely pays.

Poor write performance specifically points at parity overhead or cache. On mechanical storage, RAID 10 behaves considerably better than parity for random writes, and rebuilds are far less punishing — our RAID guide covers the trade-offs.

Everything within baseline means the storage is not the problem, and the investigation should move elsewhere. That is a useful result rather than a wasted exercise.

Common questions

Measure and record a baseline while the system is healthy — it is what makes every later investigation start from a number rather than an argument.