장애 내성 설계: AWS에서 복원력 있는 시스템 구축하기

In cloud computing, failure in distributed systems isn’t a matter of if, but when. Modern applications span servers, Availability Zones, and Regions. Each component represents a potential point of failure. Resilient applications engineer fault tolerance into their architecture, building systems that self-recover and maintain availability. This post is written for engineers and architects who run distributed data systems such as Apache Cassandra, Apache Kafka, or HDFS on Amazon Elastic Compute Cloud (Amazon EC2) and want to build resilience against hardware failure.

We were working with a customer during one such incident and wanted to share the example. The customer runs a web application that uses Cassandra as its data store, handling both read-heavy and write-heavy workloads at a scale of millions of queries per day. Consider a platform that monitors millions of enterprise network devices across hospitals, universities, and airports worldwide. It detects problems before IT teams even notice them. For that platform, a 2 AM page is more than inconvenient.

When your value proposition is catching failures before anyone else does, being caught off-guard by your own infrastructure failure is existential. The engineering team was deep in quarterly planning when their monitoring dashboard lit up. Three Cassandra nodes had gone dark simultaneously. This was not a graceful shutdown or a rolling restart. It was a hard failure with no warning. Their architecture is typical of high-scale telemetry platforms.

Kafka-powered microservices ingest device telemetry, Apache Flink handles real-time anomaly detection, and Apache Airflow orchestrates batch analytics and firmware updates. All of these rely on Apache Cassandra as the distributed database backbone. The database stores billions of daily writes and handles millions of queries per day. 4xlarge instances running Cassandra nodes failed simultaneously in the SFO region. Investigation revealed that all three instances were colocated on the same physical host. That host suffered a hardware failure, taking all three instances offline at once.