Designing for Graceful Failure by Separating Failure Domains

Mon, 06 Mar 2023 00:00:00 +0000

Context

Some time ago my team in a company that I work on had really serious issue. I fixed one feature that started working as expected, but it also started sending a lot more notification emails to our clients - nearly 150k / a day.

As you probably guess we ran out of quota. What’s worse it happened for all applications we develop.

How could we prevent it?

My advice probably is not ideal and won’t save you in similar case, but it will:

help you survive longer
give more time to prepare a fix
won’t crash all applications or modules that are sending emails, fail gradually in other words

Solution

Solution is very simple. When developing such thing as sending emails that are very important from business perspective. You should divide quota per every application or module that would be using it.

Thanks to that:

only one app or module would be affected or ran out of quota (the rest would be working as intended)
it will give you a space to shift quota a little if needed
it will give you more time to prepare a fix, because not all parts of your app would be affected

Additional steps

Of course this is not only thing you could do. You could:

implement sending emails or notifications in bulk as my very smart colleague did :)
implement metrics and alerting in order to spot your problem before your app would be crashed

Summary