15. Failure, retries and timeouts

Sending a text message means calling somebody else’s service over the network. That service can be down for a few seconds, it can be slow, and it can refuse a particular number. The confirmation sender from chapter 14 only printed its messages. We’ll rebuild it around a stand-in gateway that fails for a while or never answers.

15.1. A gateway that misbehaves

Delete the Confirmations object from concurrency.scala. Nothing in that file uses Queue any more, so change its fourth line to import cats.effect.std.Semaphore. A better Confirmations goes in a new file, confirmations.scala, which starts like this:

package fullybooked

import cats.effect.{IO, IOApp, Ref}
import cats.effect.std.Queue
import cats.syntax.all.*
import java.io.IOException
import java.util.concurrent.TimeoutException
import org.typelevel.log4cats.Logger
import org.typelevel.log4cats.slf4j.Slf4jLogger
import scala.concurrent.duration.*

trait Messenger:
  def send(phone: Phone, text: String): IO[Unit]

class FlakyGateway(calls: Ref[IO, Map[Phone, Int]], failsTwice: Set[Phone], hangs: Set[Phone])
    extends Messenger:
  def send(phone: Phone, text: String): IO[Unit] =
    calls.updateAndGet(seen => seen.updated(phone, seen.getOrElse(phone, 0) + 1)).flatMap: seen =>
      if hangs.contains(phone) then IO.never
      else if failsTwice.contains(phone) && seen(phone) <= 2 then
        IO.raiseError(IOException("gateway unavailable"))
      else IO.unit

Some of those imports aren’t used until the file is finished. Messenger is the interface to whoever sends our texts. A real implementation would make an HTTP call to a provider. Ours is FlakyGateway, a stand-in that fails on demand so that we can see each kind of trouble without waiting for it to happen.

The gateway counts the calls made for each number in a Ref that holds a Map. On each call it uses updated to make a copy of the map with that number’s count raised by one. It stores the copy with updateAndGet, which is update that also hands back the new value. Numbers in failsTwice get an IOException on their first two calls and succeed on the third. Numbers in hangs get IO.never, a program that never finishes, which is what a call to a dead server looks like from our side.

15.2. Trying again

A gateway that’s unavailable now may well be back in a moment, so the first defence is to try again. Add this to the end of confirmations.scala:

def retrying[A](program: IO[A], attempts: Int, pause: FiniteDuration): IO[A] =
  program.handleErrorWith: error =>
    if attempts > 1 then IO.sleep(pause) >> retrying(program, attempts - 1, pause * 2)
    else IO.raiseError(error)

If program fails and there are attempts left, retrying waits and then calls itself to run the same program again, with one attempt fewer. With no attempts left, retrying fails with the last error. The pause is doubled each time. A FiniteDuration can be multiplied, so that’s pause * 2. The doubling is called backoff: a service that’s struggling isn’t helped by every client asking again at once, and a growing pause gives it a chance to recover.

Create confirmations.test.scala:

package fullybooked

import cats.effect.IO
import cats.effect.unsafe.implicits.global
import java.io.IOException
import java.util.concurrent.TimeoutException
import scala.concurrent.duration.*

class ConfirmationTests extends munit.FunSuite:
  def failing(times: Int): IO[IO[Int]] =
    IO.ref(0).map: calls =>
      calls.updateAndGet(_ + 1).flatMap: n =>
        if n <= times then IO.raiseError(IOException(s"attempt $n failed")) else IO.pure(n)

  test("retrying carries on until the program succeeds"):
    val result = failing(2).flatMap(program => retrying(program, attempts = 3, pause = 1.milli))
    assertEquals(result.unsafeRunSync(), 3)

  test("retrying gives up with the last error"):
    val result = failing(5).flatMap(program => retrying(program, attempts = 3, pause = 1.milli))
    assertEquals(result.attempt.unsafeRunSync().left.map(_.getMessage), Left("attempt 3 failed"))

The helper failing(2) makes a program that fails on its first two runs and then gives the number of the run. The return type of failing is an IO of an IO. Running the outer one creates the counter and gives back the inner one, the program that uses it. Run the tests.

Retrying only helps with failures that might go away. A gateway that’s down may come back, but a number it has refused will be refused again. A retry can also do the job twice: if the gateway sent the text and its reply was lost on the way back, our second attempt sends another. A second confirmation text does no harm, but where a repeat would matter the service being called has to be able to recognise a duplicate. That usually means we send the same identifier with every attempt.

15.3. Giving up

A server that never answers is worse than one that says no, because the program waiting on it is never told anything. Add three tests to the end of ConfirmationTests and run the tests:

  test("a program that takes too long is cancelled"):
    var cancelled = false
    val slow = IO.sleep(1.hour).onCancel(IO.delay { cancelled = true })
    val result = slow.timeout(50.millis).attempt.unsafeRunSync()
    assert(result.left.exists(error => error.isInstanceOf[TimeoutException]))
    assert(cancelled)

  test("timeoutTo answers with something else"):
    val slow = IO.sleep(1.hour).map(_ => "answer")
    assertEquals(slow.timeoutTo(50.millis, IO.pure("gave up")).unsafeRunSync(), "gave up")

  test("a timeout can't cut IO.blocking short, but it can IO.interruptible"):
    val stuck = IO.blocking { Thread.sleep(300); "late" }
    assertEquals(stuck.timeoutTo(50.millis, IO.pure("gave up")).unsafeRunSync(), "late")
    val interrupted = IO.interruptible { Thread.sleep(300); "late" }
    assertEquals(interrupted.timeoutTo(50.millis, IO.pure("gave up")).unsafeRunSync(), "gave up")

Calling timeout on a program puts a limit on it. When the time runs out the slow program is cancelled and the result is a failed IO holding a TimeoutException. The first of these tests checks both, with an onCancel to see the cancellation. Its assertion on the result uses isInstanceOf, Scala’s instanceof, inside left.exists, which asks whether there’s a Left whose value satisfies a condition. In the second test, timeoutTo runs another program where timeout would fail.

I put a timeout on every call that leaves the process. Client libraries often have a default of several minutes or none at all. With no limit, a service can end up with every request waiting on a server that’s never going to answer.

A timeout works by cancelling the program it’s limiting, which chapter 14 said can happen between steps or while a fiber waits in something like IO.sleep. A call inside IO.blocking is a single step with a real thread stuck in it, so there’s no point at which to cancel. In the third test timeoutTo has to wait for the sleep to finish and gives the late answer. IO.interruptible is the version of IO.blocking that interrupts the thread when the fiber is cancelled. That works for code which responds to being interrupted, as Thread.sleep does. Some blocking I/O doesn’t, so with a blocking Java client I set the client’s own timeouts as well.

15.4. Cleaning up whatever happens

A program can finish, fail or be cancelled. Code that must run in all three cases goes in guarantee. Add one more test to ConfirmationTests and run the tests:

  test("guarantee runs after success, failure and cancellation"):
    val program =
      for
        ran <- IO.ref(0)
        _ <- IO.unit.guarantee(ran.update(_ + 1))
        _ <- IO.raiseError(IOException("no")).guarantee(ran.update(_ + 1)).attempt
        _ <- IO.never.guarantee(ran.update(_ + 1)).timeoutTo(50.millis, IO.unit)
        count <- ran.get
      yield count
    assertEquals(program.unsafeRunSync(), 3)

The counter reaches three: once for IO.unit, once for the IO.raiseError and once for the IO.never that the timeout cancels. guarantee does the job of Java’s finally, and for code that should run in only one of the cases there are onError and the onCancel we’ve used already.

15.5. Sending the confirmations

Add the rest of confirmations.scala, the sender itself:

def confirm(messenger: Messenger, logger: Logger[IO], booking: Booking): IO[Unit] =
  booking.party.phone match
    case None =>
      logger.info(s"${booking.party.name}: no number, nothing sent")
    case Some(phone) =>
      val text = s"${booking.party.name}, table ${booking.table.number} at ${booking.time}"
      val sending = messenger.send(phone, text).timeout(200.millis)
      retrying(sending, attempts = 3, pause = 20.millis)
        .flatMap(_ => logger.info(s"${booking.party.name}: sent"))
        .recoverWith:
          case _: TimeoutException => logger.warn(s"${booking.party.name}: gateway didn't answer")
          case error: IOException  => logger.warn(s"${booking.party.name}: ${error.getMessage}")

def worker(
    queue: Queue[IO, Option[Booking]],
    messenger: Messenger,
    logger: Logger[IO],
    handled: Ref[IO, Int]
): IO[Unit] =
  queue.take.flatMap:
    case None => IO.unit
    case Some(booking) =>
      confirm(messenger, logger, booking).guarantee(handled.update(_ + 1)) >>
        worker(queue, messenger, logger, handled)

object Confirmations extends IOApp.Simple:
  val logger: Logger[IO] = Slf4jLogger.getLogger[IO]

  def numbersOf(name: String): Set[Phone] =
    tonight.filter(_.party.name == name).flatMap(_.party.phone).toSet

  def run: IO[Unit] =
    val confirmed = tonight.filter(_.status == Status.Confirmed)
    for
      calls <- IO.ref(Map[Phone, Int]())
      handled <- IO.ref(0)
      gateway = FlakyGateway(calls, numbersOf("Lindqvist"), numbersOf("Moreau"))
      queue <- Queue.bounded[IO, Option[Booking]](10)
      sender <- worker(queue, gateway, logger, handled).start
      _ <- confirmed.traverse_(booking => queue.offer(Some(booking)))
      _ <- queue.offer(None)
      _ <- sender.joinWithNever
      count <- handled.get
      attempts <- calls.get
      _ <- logger.info(s"Handled $count bookings with ${attempts.values.sum} calls to the gateway")
    yield ()

In confirm, the timeout is inside the retry, so each of up to three attempts gets a fifth of a second. Putting the timeout outside would give one limit for the whole job, which is exercise 4. The recoverWith at the end takes a case lambda and deals only with the failures that match one of its cases. Its first pattern, case _: TimeoutException, matches by type without giving the exception a name. When every attempt has timed out or met an IOException, confirm logs a warning and finishes normally, because one undeliverable message mustn’t stop the rest. An exception of any other type is still a failure.

The queue carries Option[Booking] so that run can tell the worker when to stop. A Some holds a booking to confirm, and on None the worker stops calling itself and ends. Each confirmation goes through guarantee, so the handled counter goes up even for a booking whose confirm fails. The run method calls start to put the worker on a fiber of its own, then puts tonight’s confirmed bookings and the final None on the queue. It waits in joinWithNever for the worker to finish. Had the worker failed with an exception we didn’t plan for, joinWithNever would fail the whole program with it. Run Confirmations:

$ scala run . --main-class fullybooked.Confirmations
INFO  Confirmations: Okonkwo: sent
INFO  Confirmations: Lindqvist: sent
WARN  Confirmations: Moreau: gateway didn't answer
INFO  Confirmations: Fitzgerald: no number, nothing sent
INFO  Confirmations: Handled 4 bookings with 7 calls to the gateway

The Okonkwo message went first time. The Lindqvists’ message failed twice and went on the third call, which shows only in the total. The Moreaus’ number hangs, so all three attempts timed out and the log has a warning. With no number for the Fitzgeralds, that’s one call, three and three: the seven in the last line.

15.6. Two kinds of failure

Every failure in this chapter has been an exception inside a failed IO, where chapter 9 argued for errors as values. A Left is still for what chapter 9 called failures that are part of the job, such as a full restaurant. The Left is in the return type, so the compiler makes the caller look at it. There’d be no point retrying a call that returned one. A failed IO is for trouble with the machinery, such as a network that’s down. It travels up through every flatMap without being mentioned, until code like confirm retries it or logs it.

With attempt, a failed IO becomes a successful one holding a Left, and IO.fromEither goes the other way when the Left holds an exception. Where several steps each return an IO of an Either, chaining them gets long-winded. Many codebases use a wrapper from Cats called EitherT for that. I match on the Either by hand, as worker matches on its Option, until a method has enough steps to make it worth the extra type.

15.7. Exercises

Solutions are in the book’s downloadable code, under ch15/exercises.

  1. Define case class Refused(reason: String) extends Exception(reason) and write retryingTransient, which behaves like retrying except that a Refused is never retried. The (reason) after Exception passes the reason to its constructor. handleErrorWith accepts a case lambda too, but its cases have to cover every error.

  2. Write retryingCapped[A](program: IO[A], attempts: Int, pause: FiniteDuration, most: FiniteDuration): IO[A], in which the pause doubles as before but never grows beyond most. A FiniteDuration has a min method.

  3. Write sendWithFallback(first: Messenger, second: Messenger, phone: Phone, text: String): IO[String], which tries a second messenger when the first fails and returns "first" or "second" to say which one worked. In a test, new Messenger: followed by an indented send makes a messenger on the spot.

  4. Write withinLimit[A](program: IO[A], limit: FiniteDuration): IO[A], which retries up to a hundred times, starting with a pause of 5 milliseconds, but gives the whole job a single time limit.