Incident Report · Root Cause Analysis

Storage Array
Multi-Device Drop and Recovery

Incident WI-2026-0731-01 · 31 July 2026

On the evening of 31 July 2026, one of the storage arrays at the site stopped working. Four of its twelve hard drives went unreachable at exactly the same moment. That array is built to survive two drives failing, so four was more than it could absorb, and its 26.6 TB of storage went offline. Other storage at the site was unaffected and stayed available throughout, but the services that read from this particular array stopped working.

Service was back thirty-two minutes after Walter Infrastructure (WI) began work on it. Everything on the system was checked and found present and readable. No evidence of data loss.

The drives themselves turned out to be fine. The fault was in the electronics that connect the drives to the rest of the machine, which locked up and stopped answering for four of them. Switching the power off and back on cleared it, and all twelve drives have worked normally since, with no errors recorded.

Two points matter going forward. First, the storage came back running on ten of its twelve drives, so it currently has no spare capacity to absorb another failure. Putting the last two back requires several hours of heavy work, which is scheduled rather than rushed. Second, what made the electronics lock up in the first place, after thirty-two days of running without incident, is not yet established. Until it is, WI is treating the machine as unproven.

Because of those two points, WI began copying the highest-value data off that system the same evening, before doing anything else to it. Those copies are running now. The repair work happens once they finish, so that if anything goes wrong during the repair, nothing is lost.

Environment and Impact

NAS-01 is a sixteen-bay appliance on an ARM SoC platform. Bays are divided between a four-device flash pool and a twelve-device bulk pool. The two pools are independent arrays on independent controller paths, which is material to the analysis: the flash pool was unaffected throughout and remained mounted for the entire event.

The twelve bulk devices share one host adapter and are addressed as sequential SCSI targets behind a fan-out stage. The four flash devices each present on their own host. That asymmetry is the first structural clue: a fault confined to one shared stage can take multiple bulk devices without touching flash.

Root Cause Analysis

Bus address distribution

All twelve bulk devices attach to one SCSI host as targets 4 through 15. Mapping the failures against that address space produced the finding that drove the diagnosis:

host:target   state          host:target   state
  4:4         nominal          4:10        NO CONNECT
  4:5         nominal          4:11        nominal
  4:6         nominal          4:12        NO CONNECT
  4:7         nominal          4:13        nominal
  4:8         NO CONNECT       4:14        NO CONNECT
  4:9         nominal          4:15        nominal

Targets 4 through 7 are entirely unaffected. From target 8 upward the failures alternate without exception. Four devices failing independently in the same second is already implausible; four devices failing in an arithmetic sequence of bus addresses is not a failure mode that independent devices exhibit. The distribution points at a shared stage, and specifically at a subset of lanes within it.

Error class

Every logged error carried hostbyte=DID_NO_CONNECT with driverbyte=DRIVER_OK. That combination states that the host adapter was unable to reach the target at all. It is categorically different from a device returning a medium error, an unrecovered read error, or a timeout, which are what a failing device produces. The driver layer reported itself healthy while the transport reported no path.

Simultaneity, established from array metadata

All eight surviving members carried identical event counters and update timestamps, with a consistent array state bitmap across every member:

Identical event counters across every survivor establish that the array stopped at a single instant with no split-brain condition and no torn writes. Progressive device degradation produces divergent event counters as members drop at different times. This state is the strongest single evidence of a bus event, and it is what made recovery low risk: the surviving stripe set was internally coherent.

Array State Forensics

Physical presence maintained

All sixteen block devices remained enumerated under /dev for the duration; nothing disappeared from the bus. The devices were addressable as objects while unreachable for I/O, consistent with a fan-out stage that has stopped forwarding transactions rather than one that has dropped its targets.

Recovery behavior

A cold power cycle restored all four devices to full function; a genuinely failed storage device rarely does, while a hung controller routinely does. With the address pattern and the coherent surviving metadata, the evidence is substantially more consistent with a hung shared transport stage than with four independent device failures. Confirmatory rather than diagnostic, but it weighs heavily against device failure.

The array reassembled with ten of twelve members. Understanding why two rejoined and two did not is central to the remediation plan, because the two categories require different handling.

Consequence for redundancy

A twelve-device RAID 6 sustains two absent members. The array currently has exactly two absent. It is therefore operating correctly but with zero remaining tolerance: a single further device event during the present window would take the volume. This is the dominant residual risk and it governs the sequencing in Section 11.

Array State Forensics · Trigger Analysis

Four bay indicators continued to show error state after recovery, including two bays whose devices were at that moment active, in-sync array members. Inspection of the platform configuration store found exactly four persisted per-device error keys, one per illuminated bay.

The mechanism is established. The trigger is not. The unit had been running thirty-two days without incident and had not been physically disturbed, opened, or moved for several years prior. The following remain open and are ordered by assessed likelihood:

Hypotheses Tested

Five hypotheses were formed during the response and none survived testing. The verdict on each is worded to match the strength of the test behind it. They are recorded in full because the eliminations are part of the finding, and because two of them were held with high confidence before being tested.

H1 · Four devices failed independently

Rejected on the evidence. All four returned to full function after power cycling and subsequently tested indistinguishably from their peers on sequential read and seek latency. Four independent failures do not distribute across an arithmetic sequence of bus addresses, and the simultaneous return of all four on a single power cycle is not a behavior independent failures produce.

H2 · Backplane or cabling fault

Substantially less likely. This was the leading hypothesis through the first phase of the response and it did not survive the recovery. The alternating-lane distribution fit a physical connector or lane fault well, and hardware replacement was being scoped on that basis before the cold cycle. A fixed connection fault does not repair itself when power is removed and restored; an intermittent contact fault could, so this one is downgraded rather than closed, and Section 8 retains backplane degradation as an open, lower-ranked candidate.

H3 · Array metadata corruption

Disproved by direct observation. Metadata reported absent on four devices during the fault proved intact on two of them once the bus recovered. Superblocks in this format reside at the end of the member partition; a read of that region is still a read, and an unreachable device returns the same result as an unwritten one. The platform tool did not distinguish the two cases.

Hypotheses Tested, continued

H4 · Array write performance limited by stripe cache depth

Disproved by measurement. During post-incident performance analysis of NAS-03, the RAID stripe cache was found configured at half the kernel default. On a parity array a starved stripe cache prevents write coalescing into full stripes and forces read-modify-write cycles, which fit the observed write throughput closely. The parameter was raised through four values spanning a 64-fold increase and measured at each step.

Flat within measurement noise. The hypothesis was rejected and the parameter returned to its original value rather than left changed on the strength of a theory.

H5 · Array write performance limited by filesystem stripe misalignment

Disproved by measurement, but a real defect was found and corrected. The ext4 stripe width recorded on the affected volume corresponded to a smaller member count than the array actually has, consistent with the array having been expanded at some point without the filesystem geometry hint being updated. Every write batch was therefore being aligned to a boundary that did not match the true stripe, which forces read-modify-write on a parity array and fit the symptom precisely.

The geometry was corrected to match the array. Throughput before and after was measured across three runs and did not move. The misconfiguration was genuine and worth fixing on its own merits, but it was not the constraint. Both H4 and H5 were plausible, both were measured rather than assumed, and both were discarded when the data did not support them.

Response Methodology

The array was recoverable or it was not, and that could not be known until it was characterized. Sequencing every step to preserve the means of recovery, before attempting any recovery, followed from that.

Read-only until characterized

No corrective action was taken until the failure mode was understood. The diagnostic set comprised device enumeration, bus topology mapping, kernel ring buffer analysis, array superblock inspection across all members, raw device reads, and platform storage state queries. None of these writes to the array. A rebuild initiated at this stage against the device-failure hypothesis, which was the initial reading, would have written parity across a coherent stripe set on the basis of a misdiagnosis.

Metadata capture preceding any state change

Complete superblock output for every surviving member, plus partition geometry, bus topology and platform storage state, was written off the device and committed to two independent repositories before the shutdown command was issued. That capture is what would have permitted a manual array reconstruction had assembly failed on restart, since it preserves device role ordering, chunk size, metadata format and array identity.

The capture also caught an error. A geometry parameter recorded in prior working notes did not match the value in the captured superblocks. Reconstruction using the incorrect value would have written a stripe layout inconsistent with the on-disk data and destroyed the volume irrecoverably. The corrected value was propagated to the documentation before any recovery path was contemplated.

Cold cycle selected deliberately over warm restart

A warm restart does not remove power from the storage backplane or the fan-out stage, so against a transport-level hang it changes nothing while consuming the one clean shutdown state available. Full power removal was selected specifically because the diagnosis placed the fault below the device layer.

When the shutdown stalled past ten minutes, forced power removal was authorized only after establishing three conditions: the bulk volume was already unmounted so no writes were in flight, the surviving members held coherent metadata so no in-progress stripe update could be truncated, and the only mounted filesystem was the near-idle journaled flash pool.

Inventory preceding protection

A full recursive listing of the volume was captured while mounted and before any copy operation began: 45,052 file entries with sizes and timestamps, and 13,622 directories. Had the array been lost during the subsequent multi-hour copy, that listing would have been the authoritative record of contents.

A first attempt at the inventory silently produced directories but no file entries, since the platform's embedded userland lacks the printf extension used. The failure was caught by validating the output row count against expected magnitude rather than by assuming success, and the method was changed to a recursive long listing.

Data Protection Engineering

With the array carrying no remaining tolerance and the trigger not yet established, protective copies were started ahead of the rebuild. Path and protocol selection were driven by measurement on the real path rather than by specification, and several intuitive assumptions did not survive testing.

Benchmark validity

Three measurement traps were identified and controlled for. Each would have produced a materially wrong plan.

Protocol selection

Two transports were measured on identical hardware and path. The selected protocol delivered 2.2 times the throughput of the alternative, with the difference attributable to per-byte processor cost on the ARM SoC rather than to the network. With four transfer legs running concurrently against the same processor, the lower-overhead protocol was selected for all legs rather than mixed, so that no leg competes for cycles with the others.

Multipath Engineering

NAS-02 carries the largest transfer leg and sits behind a 1 Gb/s link, making it the critical path for total elapsed time. Two options were evaluated to raise its throughput.

Link aggregation, evaluated and rejected

The core switch supports 802.3ad and exposes link aggregation group configuration, membership, and per-port protocol parameters. It exposes no load-balancing or hash policy setting at any level of its configuration interface. With a fixed firmware hash, every frame between one source address and one destination address resolves to the same physical member, so a single transfer between two hosts would occupy one link and gain nothing. Aggregation was rejected on that basis before any cabling was changed.

Discrete-address multipath, implemented

The second network interface on NAS-02 was assigned its own address in the same subnet, and the transfer was split into two independent streams targeting the two addresses. Distinct destination addresses on distinct interfaces produce genuinely distinct paths with no dependence on switch hash behavior.

The known failure mode for same-subnet multi-homing is the host answering address resolution for either address on either interface, which collapses both streams onto one link. Address resolution behavior was constrained so that each interface responds only for its own address, and the result was verified from the source host rather than assumed: the two destination addresses resolve to two different hardware addresses, matching the two physical interfaces.

Execution and Current State

Transfers use rsync over NFS, driven from the shell on NAS-01 itself rather than through the appliance's own replication interface. The source host holds local read access to every dataset, whereas an intermediary would require additional exports for two of the four datasets and would introduce a second network traversal. The source host is also the only component whose failure would halt the work regardless of where it was orchestrated, so placing the orchestrator there adds no failure mode.

Tooling Selection

Why the command line rather than the appliance's own sync tooling

The appliance ships backup and replication tooling that would have moved this data. It was not used, because this work had four requirements that are more readily met by driving rsync directly.

The trade is that the work is scripted rather than configured, which is the correct trade for a sequence that runs unattended against an array with no redundancy. This is a selection on fit, not a judgment on the vendor tooling, which is appropriate for routine scheduled replication.

Secondary Findings

Identified while evaluating NAS-03 as a copy destination. One device in its five-member parity array carries a predictive health warning from platform monitoring. The array is at full redundancy and the device performs normally under test.

Backup Coverage Audit

The backup platform was examined during the response to establish whether the incident placed guest backups at risk. It did not. The datastore resides on NAS-03's flash tier, a different array on different hardware from the affected pool, and was available and intact throughout.

Neither standard device health tooling nor the platform's own health utility returns usable output on the NAS-01 hardware. Standard tooling is absent from the firmware image and the vendor utility returns empty results for every queried slot, including slots whose devices are demonstrably healthy. Device condition on this array is therefore currently inferred from throughput and latency testing rather than read from the devices.

This is a material limitation. A declaration that the array is healthy cannot presently be made on evidence, only on the absence of contrary indication. Resolving it is a prerequisite for closing the incident.

Actions and Decisions

• Critical Restore array redundancy. Add the two non-member devices and rebuild to full membership. They hold no array metadata and therefore join as new members rather than rejoining, requiring a full parity reconstruction. Window required. Several hours at sustained load, and it is the heaviest I/O the array will see. Must not start until protective copies complete.
• High Instrument for recurrence. Monitoring for the specific transport error class has been specified and should be wired to the existing alert path. A single recurrence following a clean rebuild changes the platform decision from repair to replace.
• High Resolve device health telemetry. Prerequisite for declaring the array healthy on evidence rather than on absence of symptoms. See Section 18.
• High Review controller firmware. Establish whether a revision exists addressing fan-out or transport behavior. Simplest available mitigation if one exists. Complete before the rebuild window.
• High Clear stale bay indicators. Requires a restart. Bundle into the rebuild window rather than consuming a separate outage.
• Medium Replace the flagged NAS-03 device. Procure, then hot swap while redundancy is intact. Window required for the rebuild that follows.
• Medium Extend the protection tier for the two highest-value datasets. Both are held on dual-parity storage and were never at risk during this event. Adding a second independent tier is scheduled work, brought forward as a direct output of this incident.
• Medium Confirm first scheduled backup of the two newly provisioned guests and complete the retention policy review. See Section 17.
• Low Persist the multipath interface configuration. Currently applied at runtime and will not survive a restart of that host, which would stall one copy leg into indefinite retry.

Open Questions and Assessment

Contained with no evidence of data loss, restored in thirty-two minutes, the failure mechanism identified to the subsystem. The affected hardware is operating within normal parameters and has logged no errors since recovery.

The platform is not yet proven and is being treated accordingly. Redundancy restoration is scheduled rather than assumed, recurrence instrumentation is specified, health telemetry is flagged as a gap rather than glossed, and the highest-value datasets are being copied off the array while the trigger question stays open. Two findings unrelated to the incident were surfaced by the response and are carried above with priorities.