Incident Report · Follow-Up

Storage Array Multi-Device Drop:
Recovery Follow-Up

Incident WI-2026-0731-01 · Follow-up report · 7 August 2026

The storage array that lost four of its twelve drives on 31 July is now back to full redundancy. It has all twelve drives, tolerates two failures again, and the array itself reports no errors.

Getting there took six days, and that was deliberate. The array came back on 31 July running on ten drives with no capacity to absorb another failure. Repairing it means several hours of the heaviest work the machine ever does, on a machine whose fault was not yet understood. Walter Infrastructure (WI) copied the data off first and repaired second, so that a failure during the repair would be recoverable. The copies finished on 3 August and were verified. The repair began the same morning.

One drive failed during that repair, on 3 August. It was not a surprise in kind, since the array was operating without redundancy, and the copies were already complete. That drive was found to have physically damaged areas on its surface. It was replaced on 6 August and the array rebuilt overnight to full redundancy.

Incident Report · Follow-Up

Two findings from this period change the conclusions of the first report.

The first report identified the failure mechanism but left the trigger unexplained. The trigger is now identified. The appliance runs a scheduled routine every morning that reads health data from every drive at once, and the controller fault reproduces immediately behind it, on schedule. The fault has been observed on five separate mornings. On none of those occasions did it remove a drive from the array or produce a filesystem error.

The second finding concerns what is at fault. When the failed drive was removed, the daily errors stopped for thirteen hours, which suggested that one bad drive had been provoking the shared controller. That reading was tested and did not survive: the errors resumed on schedule with the drive physically out of the machine, affecting all remaining drives. The shared controller stage is therefore weak independently of any one drive. This is the finding that determines the replacement hardware requirement.

Incident Report · Follow-Up

22 TB was copied to two separate appliances and verified before any repair action was taken. The sequence was ordered by reacquirability, highest value first, so that the least replaceable data was protected earliest.

The run took 54 hours and recorded no transfer errors, no retries and no stalled mounts.

Datasets A and B were treated as the protected set and completed first, on the reasoning that they have no reacquisition path. Datasets C and D can be reconstructed from external sources and were copied second.

Incident Report · Follow-Up

Copy completion was verified by comparing object inventories between source and destination. Size totals were not used, and that mattered here.

For Dataset D the totals were misleading in both directions at once. The destination held more objects than the source, because the copy ran without a delete pass and retained superseded versions of objects that had been replaced during the run. The destination also held fewer bytes than the source, because objects added late in the run had not been captured. The two effects partly canceled, producing a total that looked approximately correct while thirteen objects were in fact missing.

The inventory comparison found them. They were copied in a follow-up pass and the comparison re-run, returning zero missing of 15,776.

Incident Report · Follow-Up

Before the repair, the volume was audited against the copy set to confirm nothing had been missed. Three datasets were found that had never been in scope, totaling 8.5 GB.

The rescue plan had been scoped around a stated understanding of what could not be replaced. That understanding was incomplete. Dataset E in particular is authored work with no external source, and it had been on the array long enough that its presence was not recalled.

All three were copied and verified before the repair proceeded. The copies were correct against the plan. The plan was wrong. What found the gap was auditing the volume itself, and that ordering is now a standing step.

Incident Report · Follow-Up

The appliance management interface could not perform the repair. Its rebuild dialog reported zero devices available out of sixteen and would not proceed.

The cause was the stale device error flags described in Section 7 of the previous report. Those flags mark a device unhealthy in the appliance database, and the management interface will not offer a device it believes is unhealthy as a rebuild target. Both candidate devices carried flags set during the original fault. The array layer showed both as available, correctly sized and error-free, and the array itself as clean and degraded with zero failed devices.

The repair was performed at the array layer instead, which added both devices online without stopping the array or interrupting service.

Both devices were added within seconds of each other, deliberately. A dual-parity array reconstructs two members in a single pass if both are present before reconstruction begins. Adding the second afterwards queues an independent second pass, roughly doubling both elapsed time and the period of sustained load. Measured throughput was 72 MB/s reconstructing one member against 49 MB/s reconstructing two, so a single pass costs materially less time and sustained load than two sequential passes.

Incident Report · Follow-Up

Approximately forty minutes into reconstruction, one of the two devices being rebuilt failed and was removed from the array by the kernel.

The failure signature was distinct from the original event. The device accepted 299 queued commands and completed none of them. The controller aborted the queue, issued a hard reset, and the device did not return. The kernel marked it offline after seventeen minutes of recovery attempts.

Scope was contained. Each of the following was measured:

• Zero filesystem errors. No read-only remount, no journal abort.
• All 27 I/O error entries named the failed device and no other.
• The volume remained mounted read-write with all data present and readable.
• The remaining eleven devices passed direct read tests.
• Reconstruction of the second device continued without interruption and completed successfully.

The array reached eleven of twelve members and single-device fault tolerance was restored.

Incident Report · Follow-Up

The failure was caused by damaged areas on the drive surface. The evidence points to the device itself and away from its connection.

The I/O errors cluster on specific physical regions. Twenty-six of twenty-seven fall within a 3.4 MB band approximately 125 GB into the device. The twenty-seventh falls within 36 MB of an unrecoverable read error the same device had logged two days earlier, at a different location. That indicates at least two separate damaged regions.

Interface error counters on the device read zero, identical to a healthy peer. Signal integrity faults in cabling, connectors or backplane increment those counters. They did not move, which points away from the connection and to the device.

The behavior is consistent with a drive entering internal recovery on marginal media, ceasing to service its command queue while it retries, and exceeding the controller timeout. A drive that returns an error is behaving correctly. This device accepted hundreds of commands and then stopped answering.

Incident Report · Follow-Up

The failed device was replaced on 6 August. The old device was removed from the array cleanly before physical removal so that it left no stale entry.

The appliance handled the replacement device without intervention: it detected it, wrote the full partition layout, and added it to the array automatically. A device carrying no error flag proceeds through the normal automated path, which confirms the diagnosis in Section 4. The flags were what blocked the repair.

Reconstruction ran overnight and completed. The array reports twelve of twelve members, state clean, zero failed devices.

Incident Report · Follow-Up

The previous report recorded the trigger as unexplained. It is now identified.

The appliance runs a scheduled routine each morning that collects health telemetry from every device. The controller fault reproduces within minutes of that routine, on schedule, and has been observed doing so on five separate mornings.

This is consistent with the failure mechanism described previously. The fault appears under concurrent access to all devices behind the shared controller stage, and the telemetry sweep is the only routine workload that touches every device at once.

Observed consequences across all occurrences: no device removed from the array, no filesystem error, no data affected. The signature is treated as known and non-destructive, and is monitored without alarming.

Incident Report · Follow-Up

When the failed device was removed on 3 August, the controller errors stopped completely for thirteen hours, including the eight and a half hours of parity reconstruction that followed. Reconstruction is the heaviest concurrent load the controller sees. Passing it in silence was strong apparent evidence that the failing device had been the source of the fault all along.

That reading was recorded as provisional and identified the daily telemetry sweep as the test that would settle it.

It did not survive. The errors resumed on the next scheduled sweep, with the device physically removed from the machine, and affected all eleven remaining devices. The thirteen hours of quiet were simply the interval between one daily sweep and the next.

The rejected reading was believed and acted upon for thirteen hours, and the conclusion it would have overturned governs replacement hardware selection. A single quiet interval is not a controlled test when the fault has a daily period.

Incident Report · Follow-Up

The previous report described four stale device error flags left by the original fault, marking devices as failed that were demonstrably healthy, and recorded that clearing them was expected to require a service interruption.

The flags matter more than their cosmetic effect suggested. A single stored flag drives the panel indicator, the health field in the management API, the device status in the array member list, and the exclusion of that device from the rebuild dialog. The inability to repair through the management interface, described in Section 4, was caused by these flags.

They do not clear themselves. A flag is written when an error occurs and is removed only by an explicit clear. A device that returns to normal operation never generates one, so a flag set during an incident persists indefinitely.

One flag remains and is retained deliberately. It is keyed to the identifier of the failed device, which is no longer in the machine, so nothing present matches it. If that device were reinstalled it would be flagged immediately, which is correct.

Incident Report · Follow-Up

The previous report recorded that per-device health data could not be obtained on this platform, and that device health was therefore inferred from performance.

One caution applies to the API route. Its health field reports stored fault state, not current condition: it reported four devices abnormal while two of those four were healthy, in-sync array members with all attribute counters at zero. The attribute tables are authoritative. The health field is not.

A second caution applies to comparison across vendors. The replacement device is from a different manufacturer than the eleven originals and encodes several attributes as packed rate values rather than counts. Read literally against the other devices' zeros, three of its attributes appear to show large error counts and do not. Device health on a mixed-vendor array must be assessed per vendor.

Health baselines were captured for all twelve devices and are now the reference for change detection.

Incident Report · Follow-Up

Second copies of the protected datasets are retained. They are not a permanent storage tier and their disposition is already scheduled.

Incident Report · Follow-Up

The observation period's pass criterion is device loss from the array. Controller log activity alone does not fail it. The daily controller fault is expected and has been shown across five occurrences not to affect array membership or data. A device leaving the array would be a different finding.

Incident Report · Follow-Up

The finding in Section 9 sets the requirement directly.

The fault is in a stage shared by all twelve devices. It is not attributable to any individual device, and it reproduces on demand under concurrent all-device access. Replacement hardware must therefore give each device an independent path to the host, rather than multiplexing devices behind a shared stage.

This is a specific, testable requirement. It can be verified against a candidate before purchase by reproducing the all-device access pattern.

Most appliance vendors do not publish how their bays are wired. For a twelve-bay platform it is a question that always has an answer, because no mainstream processor in that class provides twelve native ports. Something is in the path. Where topology could not be established from vendor documentation, the candidate was not considered.

Incident Report · Follow-Up

The array is fully redundant, clean, and has lost no data. The failure mechanism was identified in the first report and the trigger is identified in this one. The fault is reproducible, which makes the replacement decision testable.

The six days at reduced redundancy came from a deliberate ordering. Copying 22 TB and verifying it took 54 hours, and repairing before that completed would have meant performing the heaviest workload the machine does, on an array with no fault tolerance, without a second copy of the data. One device did fail during the repair. Had that repair run first, the same failure would have occurred against unprotected data.

Two corrections to the previous report are carried here. The stale device flags did not require a service interruption to clear, and per-device health telemetry was available throughout. Both were stated at the width of what had been checked at the time, and both checks were narrower than the conclusions drawn from them.

The appliance is treated as serviceable but not proven. It carries a known, reproducible fault in a shared component that has not caused data loss across five observed occurrences and one full reconstruction. Replacement is specified and in progress. The protected datasets hold verified second copies in the meantime.