AmazonS3
tcp/443
CloudFront
tcp/80
.DS_Store” is an abbreviation for “Desktop Services Store”. These files are created automatically by Apples “Finder” software (which is part of their OS).
They store information about the files within a folder, including display options of folders, such as icon positions and view settings.
It may happen that .DS_Store files inadvertently leak filenames such as database backups or private administration panels.
Severity: low
Fingerprint: 5f32cf5d6962f09c1a5d9b0f1a5d9b0fb246cfc7db4a95d2a31935b7a31935b7
Found 3 files trough .DS_Store spidering: /img /img/authors /img/diagrams
.DS_Store” is an abbreviation for “Desktop Services Store”. These files are created automatically by Apples “Finder” software (which is part of their OS).
They store information about the files within a folder, including display options of folders, such as icon positions and view settings.
It may happen that .DS_Store files inadvertently leak filenames such as database backups or private administration panels.
Severity: low
Fingerprint: 5f32cf5d6962f09c1a5d9b0f1a5d9b0fb246cfc7db4a95d2a31935b7a31935b7
Found 3 files trough .DS_Store spidering: /img /img/authors /img/diagrams
Open service 108.138.7.97:443 · www.firstprincipleresilience.com
2026-01-09 12:18
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
Date: Thu, 08 Jan 2026 23:29:29 GMT
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
ETag: "267e81a504df9f0012199f4694a50763"
Server: AmazonS3
X-Cache: Hit from cloudfront
Via: 1.1 e1f996a9009532eeea33edfd32ef3240.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: F-1gAyrEH4u3x6MZV6ZoUuqMnZke-Sqe15hhH7he1tQIyv_-hTfCAQ==
Age: 46123
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.9:443 · firstprincipleresilience.com
2026-01-09 12:18
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
Date: Thu, 08 Jan 2026 23:29:29 GMT
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
ETag: "267e81a504df9f0012199f4694a50763"
Server: AmazonS3
X-Cache: Hit from cloudfront
Via: 1.1 c0c6d7afa25d841027d75444425d2010.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: zbnPE3Px8_Wgx6IjiKVC4VrDRUdgjHFLLbgMSrytsBjERCNbnDWTHg==
Age: 46122
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.9:443 · firstprincipleresilience.com
2026-01-02 15:37
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
Server: AmazonS3
Date: Thu, 01 Jan 2026 21:20:21 GMT
ETag: "267e81a504df9f0012199f4694a50763"
X-Cache: Hit from cloudfront
Via: 1.1 f61953901038b0c4b4c82c311140f1b8.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: CjBCfoFLYUAjvnkMh55Q0dyRdaO7OvHkEMALm33iGfXR_w4MoHYAuQ==
Age: 65850
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.97:443 · www.firstprincipleresilience.com
2026-01-02 15:37
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
Server: AmazonS3
Date: Thu, 01 Jan 2026 21:20:21 GMT
ETag: "267e81a504df9f0012199f4694a50763"
X-Cache: Hit from cloudfront
Via: 1.1 6b85d8725dd6471c3db1f65d4096afc4.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: LrEFsgwEY65anIfC2i_dlw6MOjnbzXMuiJST_ko5gSR1j_iBv0cwNQ==
Age: 65850
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.9:443 · firstprincipleresilience.com
2025-12-23 01:12
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
Server: AmazonS3
Date: Mon, 22 Dec 2025 06:29:31 GMT
ETag: "267e81a504df9f0012199f4694a50763"
X-Cache: Hit from cloudfront
Via: 1.1 61c90c70feca5f532bf48bc0dc85d516.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: oRLmzveRYdL7X4EsOrI6HDMLBQOO8wPS_2_ovQN6JdbVQNUPTL0P7w==
Age: 67388
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.97:443 · www.firstprincipleresilience.com
2025-12-23 01:12
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
Server: AmazonS3
Date: Mon, 22 Dec 2025 06:29:31 GMT
ETag: "267e81a504df9f0012199f4694a50763"
X-Cache: Hit from cloudfront
Via: 1.1 6b85d8725dd6471c3db1f65d4096afc4.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: meVtIIWIv2a30yJbrt0og1c6qUxS_jVOaK4n-y_T6WEsOQP9-0_3eA==
Age: 67387
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.9:443 · firstprincipleresilience.com
2025-12-21 09:16
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
Server: AmazonS3
Date: Sun, 21 Dec 2025 05:11:46 GMT
ETag: "267e81a504df9f0012199f4694a50763"
X-Cache: Hit from cloudfront
Via: 1.1 002af2e4f72157b8b4bd2de012b5b57c.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: gfc6mBL2onhLCrSRG5yFyZnTeqe9dgFI8Hn_vaQ_D9yE67ykfMfjlw==
Age: 14656
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.97:443 · www.firstprincipleresilience.com
2025-12-21 09:16
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
Server: AmazonS3
Date: Sun, 21 Dec 2025 05:11:46 GMT
ETag: "267e81a504df9f0012199f4694a50763"
X-Cache: Hit from cloudfront
Via: 1.1 f13110b40e6214ad566c753a838f49f4.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: VqVdUdmee43obhbTyldc6M_r5_eE24S7zUeG6PUplg1EnpcyINKgDg==
Age: 14655
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.9:443 · firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
Date: Thu, 18 Dec 2025 03:21:23 GMT
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
ETag: "267e81a504df9f0012199f4694a50763"
Server: AmazonS3
X-Cache: Hit from cloudfront
Via: 1.1 bfb5bffe90e3b0e760933a7a07d850ba.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: U2eLRVU_rB_wykPUCUgdp2gwHHVnIqahn8O7rpfopNy1T6CVYsgNAg==
Age: 82591
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.97:443 · firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
Date: Thu, 18 Dec 2025 03:21:23 GMT
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
ETag: "267e81a504df9f0012199f4694a50763"
Server: AmazonS3
X-Cache: Hit from cloudfront
Via: 1.1 6278ee254a7d35c23aae5e936b5a56ee.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: TnLVsEfmLAPoaUNm7UjzgV93fZs2dpKWhxeOrFL-Kj3Y9GiHWCxrnA==
Age: 82591
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.97:80 · firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 301 Moved Permanently Server: CloudFront Date: Fri, 19 Dec 2025 02:17:53 GMT Content-Type: text/html Content-Length: 167 Connection: close Location: https://firstprincipleresilience.com/ X-Cache: Redirect from cloudfront Via: 1.1 dd09b3b5f5b8dc626e1ba6804a73af40.cloudfront.net (CloudFront) X-Amz-Cf-Pop: FRA56-P6 X-Amz-Cf-Id: gc8psSiBUwh9e9tsBmGQlR6mOd2BQ2vwaiOz3MhXl0VUsehZehtVDA== Page title: 301 Moved Permanently <html> <head><title>301 Moved Permanently</title></head> <body> <center><h1>301 Moved Permanently</h1></center> <hr><center>CloudFront</center> </body> </html>
Open service 108.138.7.81:443 · firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
Date: Thu, 18 Dec 2025 03:21:23 GMT
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
ETag: "267e81a504df9f0012199f4694a50763"
Server: AmazonS3
X-Cache: Hit from cloudfront
Via: 1.1 b26b931354407da013ac53d2c1c55034.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: BE69OEZAVwtzeB98wZL0dtVlPXihHDvi2N6idb7sdRS4aqk5WaCVYw==
Age: 82591
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.97:80 · www.firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 301 Moved Permanently Server: CloudFront Date: Fri, 19 Dec 2025 02:17:53 GMT Content-Type: text/html Content-Length: 167 Connection: close Location: https://www.firstprincipleresilience.com/ X-Cache: Redirect from cloudfront Via: 1.1 88cabd6b8652306789c6bc8090fbcb1a.cloudfront.net (CloudFront) X-Amz-Cf-Pop: FRA56-P6 X-Amz-Cf-Id: WLhGnBSKVoKcgBQXWEncCbHLbVdVXjc8uD3BhMYEnI1mwPX3_PZ9zQ== Page title: 301 Moved Permanently <html> <head><title>301 Moved Permanently</title></head> <body> <center><h1>301 Moved Permanently</h1></center> <hr><center>CloudFront</center> </body> </html>
Open service 108.138.7.82:443 · www.firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
Date: Thu, 18 Dec 2025 03:21:23 GMT
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
ETag: "267e81a504df9f0012199f4694a50763"
Server: AmazonS3
X-Cache: Hit from cloudfront
Via: 1.1 e96aebc8d7c9ec82b88c3160a18fed96.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: VbA4KmB5L00CLpcTbTrLB_imn10p7CMHz4r367fZAdxXf4iM2aNMdw==
Age: 82591
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.82:80 · firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 301 Moved Permanently Server: CloudFront Date: Fri, 19 Dec 2025 02:17:53 GMT Content-Type: text/html Content-Length: 167 Connection: close Location: https://firstprincipleresilience.com/ X-Cache: Redirect from cloudfront Via: 1.1 002af2e4f72157b8b4bd2de012b5b57c.cloudfront.net (CloudFront) X-Amz-Cf-Pop: FRA56-P6 X-Amz-Cf-Id: Qfqw68ztfXJ93wTIvLt2li5YAqo_0rLUxVjnHK27X1dX36jVprob7A== Page title: 301 Moved Permanently <html> <head><title>301 Moved Permanently</title></head> <body> <center><h1>301 Moved Permanently</h1></center> <hr><center>CloudFront</center> </body> </html>
Open service 108.138.7.81:80 · firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 301 Moved Permanently Server: CloudFront Date: Fri, 19 Dec 2025 02:17:53 GMT Content-Type: text/html Content-Length: 167 Connection: close Location: https://firstprincipleresilience.com/ X-Cache: Redirect from cloudfront Via: 1.1 c3fc8d1fb362a6655af993732c376dc4.cloudfront.net (CloudFront) X-Amz-Cf-Pop: FRA56-P6 X-Amz-Cf-Id: yY6bQT0HC3Vv65cBompgKJJa7p_Fs523ySNXZaAA71HKJiPQISmW-w== Page title: 301 Moved Permanently <html> <head><title>301 Moved Permanently</title></head> <body> <center><h1>301 Moved Permanently</h1></center> <hr><center>CloudFront</center> </body> </html>
Open service 108.138.7.81:443 · www.firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
Date: Thu, 18 Dec 2025 03:21:23 GMT
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
ETag: "267e81a504df9f0012199f4694a50763"
Server: AmazonS3
X-Cache: Hit from cloudfront
Via: 1.1 dd09b3b5f5b8dc626e1ba6804a73af40.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: EvfbCAyqdVu3fHZ5iqekmK08x9V2wcXDEjPUqTbzFsam1bPeqWdlew==
Age: 82591
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.81:80 · www.firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 301 Moved Permanently Server: CloudFront Date: Fri, 19 Dec 2025 02:17:52 GMT Content-Type: text/html Content-Length: 167 Connection: close Location: https://www.firstprincipleresilience.com/ X-Cache: Redirect from cloudfront Via: 1.1 149b1af6ad8d2c0fedea82bfb1c29c66.cloudfront.net (CloudFront) X-Amz-Cf-Pop: FRA56-P6 X-Amz-Cf-Id: _AX17mjls2YPFTWK9Z1er3yebUYFaF_duNZwQUwnR8Mrtnwl3_2wqQ== Page title: 301 Moved Permanently <html> <head><title>301 Moved Permanently</title></head> <body> <center><h1>301 Moved Permanently</h1></center> <hr><center>CloudFront</center> </body> </html>
Open service 108.138.7.82:80 · www.firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 301 Moved Permanently Server: CloudFront Date: Fri, 19 Dec 2025 02:17:52 GMT Content-Type: text/html Content-Length: 167 Connection: close Location: https://www.firstprincipleresilience.com/ X-Cache: Redirect from cloudfront Via: 1.1 ab68583a58d574d6a9e5fca1fb1e6316.cloudfront.net (CloudFront) X-Amz-Cf-Pop: FRA56-P6 X-Amz-Cf-Id: Kx8ceq2VxGLluaDuW5s2vypwUf1_A1ZFC4Vmj7-4aGGrDEryZrOUKA== Page title: 301 Moved Permanently <html> <head><title>301 Moved Permanently</title></head> <body> <center><h1>301 Moved Permanently</h1></center> <hr><center>CloudFront</center> </body> </html>
Open service 108.138.7.9:80 · firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 301 Moved Permanently Server: CloudFront Date: Fri, 19 Dec 2025 02:17:52 GMT Content-Type: text/html Content-Length: 167 Connection: close Location: https://firstprincipleresilience.com/ X-Cache: Redirect from cloudfront Via: 1.1 149b1af6ad8d2c0fedea82bfb1c29c66.cloudfront.net (CloudFront) X-Amz-Cf-Pop: FRA56-P6 X-Amz-Cf-Id: S6hMG406uRHP428Nl9EAKY1IG23FA5nW8UXZHEb-I-6GSRYDNaz91w== Page title: 301 Moved Permanently <html> <head><title>301 Moved Permanently</title></head> <body> <center><h1>301 Moved Permanently</h1></center> <hr><center>CloudFront</center> </body> </html>
Open service 108.138.7.82:443 · firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
Date: Thu, 18 Dec 2025 03:21:23 GMT
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
ETag: "267e81a504df9f0012199f4694a50763"
Server: AmazonS3
X-Cache: Hit from cloudfront
Via: 1.1 cfbb6e543d97587a32117dbabb25fc86.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: md6fvtiIFedfUg-6PTgeLtS6e2c4H6DLjx_gRSgpEd6vNa_9pjduhw==
Age: 82590
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.97:443 · www.firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
Date: Thu, 18 Dec 2025 03:21:23 GMT
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
ETag: "267e81a504df9f0012199f4694a50763"
Server: AmazonS3
X-Cache: Hit from cloudfront
Via: 1.1 0ece2d48b2ca1badca11fa675b7785ea.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: ILHZd_Yl6r6DjOhQxBdq7fC2SVvjDwyZXvwAbn1CKz3R0iqtgeAPMw==
Age: 82590
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.9:443 · www.firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12197
Connection: close
Date: Thu, 18 Dec 2025 03:21:23 GMT
x-amz-meta-codebuild-content-sha256: 7073379b5fc887409198a749351eab536de4a6665878d02e82d5c70448053d37
x-amz-meta-codebuild-buildarn: arn:aws:codebuild:us-east-2:545212347593:build/CodeBuildProject-QWRUZY9RO6D4:0d01349c-f487-4de2-a36a-a12ddf643199
x-amz-meta-codebuild-content-md5: cb654ab6a7ce45d30d63d4524930bcf0
Last-Modified: Fri, 26 May 2023 23:34:56 GMT
ETag: "267e81a504df9f0012199f4694a50763"
Server: AmazonS3
X-Cache: Hit from cloudfront
Via: 1.1 fb5610ec56d3f427bcbcfdd851770614.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA56-P6
X-Amz-Cf-Id: YiqPZnQd2Z9_ReJcxVi3gewTw9i5V7mLuha07hbey2GJ5Zj3XG5LvA==
Age: 82590
Page title: Principle Resilience
<!DOCTYPE html>
<html class="no-js" lang="en">
<head>
<meta name="generator" content="Hugo 0.70.0" />
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Principle Resilience</title>
<script>(function(d,e){d[e]=d[e].replace("no-js","js");})(document.documentElement,"className");</script>
<meta name="description" content="">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="dns-prefetch" href="//fonts.googleapis.com">
<link rel="dns-prefetch" href="//fonts.gstatic.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:400,400i,700">
<link rel="stylesheet" href="/css/style.css">
<link rel="alternate" type="application/rss+xml" href="/index.xml" title="Principle Resilience">
<link rel="shortcut icon" href="/favicon.ico">
</head>
<body class="body">
<div class="container container--outer">
<header class="header">
<div class="container header__container">
<div class="logo logo--mixed">
<a class="logo__link" href="/" title="Principle Resilience" rel="home">
<div class="logo__item logo__imagebox">
<img class="logo__img" src="/img/IMG_2471.png">
</div><div class="logo__item logo__text">
<div class="logo__title">Principle Resilience</div>
<div class="logo__tagline">A place for learning distributed systems resilience</div>
</div>
</a>
</div>
<nav class="menu">
<button class="menu__btn" aria-haspopup="true" aria-expanded="false" tabindex="0">
<span class="menu__btn-title" tabindex="-1">Menu</span>
</button>
<ul class="menu__list">
<li class="menu__item">
<a class="menu__link" href="/about/">
<span class="menu__text">About</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/articles/">
<span class="menu__text">Articles</span>
</a>
</li>
<li class="menu__item">
<a class="menu__link" href="/products/">
<span class="menu__text">Products</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="wrapper flex">
<div class="primary">
<main class="main list" role="main">
<article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howtosimulatehowcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/SimulateHowCloudNetworksFail.png" alt="How to Simulate How Cloud Networks Fail">
</a>
</figure>
<header class="list__header">
<h2 class="list__title post__title">
<a href="/articles/howtosimulatehowcloudnetworksfail/" rel="bookmark">
How to Simulate How Cloud Networks Fail
</a>
</h2>
<p class="list__lead post__lead">Cloud networks, and most massively scaled networks, are subject to partial failures that will impact some of your application connections but not others. Congestion will lead to intermittent packet loss and is another flavor of partial impact that occurs in multi-tenant environments. How do you know if your monitoring, health checks and failure mitigations, like host removal and retry, will mitigate these kinds of failures? You test! This article will show you how you can use Linux IPTables, routing policies and network control to simulate cloud network failures.</p>
</header>
<div class="content list__excerpt post__content clearfix">
Linux has come a long way with tooling for simulating network failures. The traffic control (tc) tool has a wonderful bag of tricks for simulating total packet loss, intermittent packet loss and delays. Tc even provides random distributions of packet loss for additional realism in testing. However, tc doesn’t support random % of network flow impacts. As a reminder, network flows are defined as a 5-tuple of ip address, ports and protocol.
</div>
</article><article class="list__item post">
<figure class="list__thumbnail thumbnail">
<a class="thumbnail__link" href="/articles/howcloudnetworksfail/">
<img class="thumbnail__image" src="/img/diagrams/CloudNetwor
Open service 108.138.7.9:80 · www.firstprincipleresilience.com
2025-12-19 02:17
HTTP/1.1 301 Moved Permanently Server: CloudFront Date: Fri, 19 Dec 2025 02:17:52 GMT Content-Type: text/html Content-Length: 167 Connection: close Location: https://www.firstprincipleresilience.com/ X-Cache: Redirect from cloudfront Via: 1.1 dd09b3b5f5b8dc626e1ba6804a73af40.cloudfront.net (CloudFront) X-Amz-Cf-Pop: FRA56-P6 X-Amz-Cf-Id: 7E18QDLguaozqmrwJ5i7wOtL2qDekcp8voLoYUQHwEdr-2cw3opRWQ== Page title: 301 Moved Permanently <html> <head><title>301 Moved Permanently</title></head> <body> <center><h1>301 Moved Permanently</h1></center> <hr><center>CloudFront</center> </body> </html>