17. Direct style
Chapter 13 gave a rule that anything which waits goes in IO.blocking. The rule exists because a waiting thread can’t be used for anything else, and real threads are heavy: each has a stack of its own and takes time to start. Fibers and IO.sleep are Cats Effect’s way round that.
Java 21 added virtual threads, which a program can have by the hundred thousand. We’ll write the last-table race and the confirmation sender again on virtual threads, without IO, and then compare the two ways of working.
17.1. A hundred thousand threads
The code in this chapter uses Ox, a library from a company called SoftwareMill. Add it to project.scala, under the line for fs2:
//> using dep com.softwaremill.ox::core:1.1.0
Then create direct.scala, with imports for the whole chapter and a first program that only uses Java’s own API:
package fullybooked
import java.io.IOException
import java.time.LocalTime
import java.util.concurrent.TimeoutException
import java.util.concurrent.atomic.AtomicInteger
import ox.{either, fork, mapPar, par, sleep, timeout, Fork, Ox}
import ox.either.ok
import ox.resilience.retry
import ox.scheduling.Schedule
import scala.concurrent.duration.*
@main def manyThreads(): Unit =
val seated = AtomicInteger(0)
val threads = List.fill(100000):
Thread.ofVirtual().start: () =>
Thread.sleep(1000)
seated.incrementAndGet()
threads.foreach(_.join())
println(s"${seated.get} parties seated")
$ scala run . --main-class fullybooked.manyThreads
100000 parties seated
The program starts its threads through Java’s Thread.ofVirtual(). A Scala lambda stands in for the Runnable. Every one of the hundred thousand threads slept for a full second in plain Thread.sleep, yet the whole run takes about two seconds on my machine.
A virtual thread is an object on the heap. It runs on one of a small pool of real threads until it blocks, in a sleep, a lock or a read from a socket. At that point the JVM takes it off the real thread and puts another virtual thread on. That’s the arrangement Cats Effect has for fibers, done by the JVM for ordinary blocking code.
So a method can go back to blocking while it waits and returning its answer when it has one. Code written that way, with no wrapper type around the result, is called direct style.
17.2. The last table, a third time
The reserve in availability.scala is already in direct style: it blocks while the slow store sleeps and then returns a plain Either. Add these to the end of direct.scala:
def lastTableDirect(store: BookingStore): (String, String) =
def first = reply(ibrahim.name, reserve(store, ibrahim, eight))
def second =
sleep(100.millis)
reply(castellanos.name, reserve(store, castellanos, eight))
par(first, second)
class DirectBookings(private var bookings: List[Booking]):
def all: List[Booking] = synchronized(bookings)
def reserve(party: Party, time: LocalTime): Either[BookingError, Booking] =
synchronized:
decide(bookings, party, time).map: booking =>
bookings = bookings :+ booking
booking
def lastTableDirectFixed(bookings: DirectBookings): (String, String) =
def first = reply(ibrahim.name, bookings.reserve(ibrahim, eight))
def second =
sleep(100.millis)
reply(castellanos.name, bookings.reserve(castellanos, eight))
par(first, second)
@main def directLastTable(): Unit =
def show(title: String, answers: (String, String)): Unit =
println(s"$title\n ${answers(0)}\n ${answers(1)}")
val store = SlowStore(300)
regulars.foreach(store.add)
show("Two requests at once:", lastTableDirect(store))
show("With a synchronized class:", lastTableDirectFixed(DirectBookings(regulars)))
$ scala run . --main-class fullybooked.directLastTable
Two requests at once:
Ibrahim: table 6 at 20:00
Castellanos: table 6 at 20:00
With a synchronized class:
Ibrahim: table 6 at 20:00
Castellanos: sorry, no table for 8 is free at 20:00
Compare lastTableDirect with chapter 14’s lastTableRace. Both have a first, a second held back by a sleep, and a last line that runs the two together. The map and >> have gone, because there’s nothing to chain: each reservation is a method call that returns when it’s finished. Ox’s par runs its two arguments on a virtual thread each and gives back both results as a tuple.
In chapter 14 first and second were vals, each holding an IO that hadn’t run. Here they have to be defs, which is easy to miss. A val would make the reservation there and then, on the current thread, before par was ever called. par takes its arguments by name, like whenOpen in chapter 7, so a def runs when par uses it.
The first pair of answers in the output is the double booking from chapters 4 and 14 again, this time on blocking threads. DirectBookings fixes it as a Java class would, with a var and synchronized around the check and the save. Putting private var in the parameter list makes bookings a field that only the class can touch. In Scala synchronized is a method that takes the block to lock around. Inside the lock, decide does the checking.
Until Java 24 a virtual thread that blocked inside synchronized kept hold of its real thread. If you’ve read advice against mixing the two, that’s what it was about.
Create direct.test.scala, with imports for the tests still to come, and run the tests:
package fullybooked
import java.io.IOException
import java.util.concurrent.atomic.AtomicInteger
import ox.{fork, mapPar, sleep, supervised}
import scala.concurrent.duration.*
import scala.util.Try
class DirectTests extends munit.FunSuite:
test("a synchronized class never gives the last table to two parties"):
val bookings = DirectBookings(regulars)
val parties = List.tabulate(20)(n => Party(s"Party $n", 7))
val results = parties.mapPar(20)(party => bookings.reserve(party, eight))
assertEquals(results.count(_.isRight), 1)
assertEquals(bookings.all.count(_.table.number == 6), 1)
The test sends twenty parties after table 6 with mapPar(20). That’s map with up to twenty elements in progress at once, each on a virtual thread of its own.
17.3. Scopes
Chapter 14 said that a fiber started by hand and forgotten is an easy leak to write. Ox’s fork can only start a thread inside a scope, which says where the thread’s life ends. Add a test to the end of DirectTests in which one of two threads fails:
test("a scope ends all of its forks when one of them fails"):
val finished = AtomicInteger(0)
val result = Try:
supervised:
fork:
sleep(1.hour)
finished.incrementAndGet()
fork:
sleep(50.millis)
throw IOException("fire alarm")
sleep(1.hour)
assertEquals(result.failed.map(_.getMessage), Try("fire alarm"))
assertEquals(finished.get, 0)
In the test, supervised opens a scope and each fork starts a virtual thread inside it. A scope ends when its block ends: any fork still running is interrupted and the scope waits for it to stop. The sleep(1.hour) on the block’s last line is only there to keep this scope open. Like the one in lastTableDirect, it’s Ox’s own sleep, which takes a duration.
After a twentieth of a second the fork with the short sleep throws. The scope interrupts everything else in it, waits, and throws the same exception itself. The first fork never gets to its last line, which is why finished stays at 0. The Try around the scope catches the exception. Calling failed on a failed Try gives a successful one holding the exception, so that the test can compare the message. par and mapPar open scopes of their own. Tying a thread’s life to a block of code like this is called structured concurrency.
Ox interrupts a fork with Java’s Thread.interrupt. A thread that’s blocked in a sleep or a wait gets an InterruptedException and unwinds.
A method that forks without opening a scope of its own has to be called from inside one. If you added this to direct.scala, it wouldn’t compile:
def reserveLater(
bookings: DirectBookings,
party: Party,
after: FiniteDuration
): Fork[Either[BookingError, Booking]] =
fork:
sleep(after)
bookings.reserve(party, eight)
$ scala compile .
[error] ./direct.scala:61:35
[error] This operation must be run within a `supervised` or `supervisedError` block. Alternatively, you must require that the enclosing method is run within a scope, by adding a `using Ox` parameter list.
[error] bookings.reserve(party, eight)
[error] ^
Compilation failed
The working version takes the message’s second suggestion, a using Ox parameter list. It goes after directLastTable, in place of the broken one if you tried it:
def reserveLater(bookings: DirectBookings, party: Party, after: FiniteDuration)(using
Ox
): Fork[Either[BookingError, Booking]] =
fork:
sleep(after)
bookings.reserve(party, eight)
The using Ox is a using parameter of the kind chapter 11 introduced, without a name this time because nothing refers to it. supervised supplies a given Ox inside its block and fork asks for one, so the compiler can tell whether a fork has a scope to belong to. A value that stands for permission to do something is called a capability. reserveLater returns a Fork, a handle to the running thread. Calling join on the handle waits for the thread’s result. Add a test to the end of DirectTests that calls reserveLater from a scope and joins both forks:
test("a method that forks is called inside a scope"):
val bookings = DirectBookings(regulars)
val answers = supervised:
val late = reserveLater(bookings, castellanos, 50.millis)
val early = reserveLater(bookings, ibrahim, 0.millis)
(early.join(), late.join())
assert(answers(0).isRight)
assert(answers(1).isLeft)
17.4. Timeouts and retries
Add a direct-style version of chapter 15’s confirmation sender to the end of direct.scala:
trait DirectMessenger:
def send(phone: Phone, text: String): Unit
class FlakyDirectGateway(failsTwice: Set[Phone], hangs: Set[Phone]) extends DirectMessenger:
private var calls = Map[Phone, Int]()
def totalCalls: Int = synchronized(calls.values.sum)
def send(phone: Phone, text: String): Unit =
val seen = synchronized:
calls = calls.updated(phone, calls.getOrElse(phone, 0) + 1)
calls(phone)
if hangs.contains(phone) then sleep(1.hour)
else if failsTwice.contains(phone) && seen <= 2 then throw IOException("gateway unavailable")
def confirmDirect(messenger: DirectMessenger, booking: Booking): String =
val name = booking.party.name
booking.party.phone match
case None => s"$name: no number, nothing sent"
case Some(phone) =>
val text = s"$name, table ${booking.table.number} at ${booking.time}"
try
retry(Schedule.exponentialBackoff(20.millis).maxRetries(2)):
timeout(200.millis)(messenger.send(phone, text))
s"$name: sent"
catch
case _: TimeoutException => s"$name: gateway didn't answer"
case error: IOException => s"$name: ${error.getMessage}"
@main def directConfirmations(): Unit =
def numbersOf(name: String): Set[Phone] =
tonight.filter(_.party.name == name).flatMap(_.party.phone).toSet
val gateway = FlakyDirectGateway(numbersOf("Lindqvist"), numbersOf("Moreau"))
val confirmed = tonight.filter(_.status == Status.Confirmed)
val results = confirmed.mapPar(4)(booking => confirmDirect(gateway, booking))
results.foreach(println)
println(s"Handled ${results.size} bookings with ${gateway.totalCalls} calls to the gateway")
$ scala run . --main-class fullybooked.directConfirmations
Okonkwo: sent
Lindqvist: sent
Moreau: gateway didn't answer
Fitzgerald: no number, nothing sent
Handled 4 bookings with 7 calls to the gateway
The results are chapter 15’s, down to the seven calls. The gateway fails twice for the Lindqvists and hangs for the Moreaus, as before. confirmDirect again makes up to three attempts of a fifth of a second each, with a pause that starts at 20 milliseconds and doubles. This time retry and its schedule come from Ox, where chapter 15 had the retrying we wrote. Ox’s timeout works by interrupting the thread. Failures arrive as exceptions, so recoverWith has become try and catch, with finally available where chapter 15 used guarantee.
In directConfirmations, mapPar(4) sends up to four messages at a time and still returns the results in the order of the list. Two more tests for the end of DirectTests check the retry and the timeout. Each makes a set of one phone number by calling toSet on the Option that Phone.parse returns:
test("a message that fails twice goes on the third call"):
val number = Phone.parse("07700 900141").toSet
val gateway = FlakyDirectGateway(failsTwice = number, hangs = Set())
val booking = Booking(Party("Lindqvist", 2, number.headOption), Table(1, 2), eight, Status.Confirmed)
assertEquals(confirmDirect(gateway, booking), "Lindqvist: sent")
assertEquals(gateway.totalCalls, 3)
test("a gateway that hangs is given up on"):
val number = Phone.parse("07700 900141").toSet
val gateway = FlakyDirectGateway(failsTwice = Set(), hangs = number)
val booking = Booking(Party("Moreau", 2, number.headOption), Table(1, 2), eight, Status.Confirmed)
assertEquals(confirmDirect(gateway, booking), "Moreau: gateway didn't answer")
assertEquals(gateway.totalCalls, 3)
Interruption has the limit chapter 15 found with IO.blocking: a timeout can only stop code that responds to being interrupted. confirmDirect can be interrupted from outside as well, when the scope that mapPar opened is shutting down. Because InterruptedException is an Exception, a handler written as case error: Exception catches it and leaves a thread that has been told to stop still running. That’s why confirmDirect names the two types it means to handle.
17.5. Errors as values, without the for
A refused booking is still a Left in direct style. For chaining methods that return an Either, Ox has something shorter than a for. Add this to the end of direct.scala:
def bookDirect(bookings: DirectBookings, request: BookingRequest): Either[List[BookingError], Booking] =
either:
val (party, time) = validate(request).left.map(_.toList).ok()
bookings.reserve(party, time).left.map(error => List(error)).ok()
Inside an either block, .ok() on a Right gives the value in it. On a Left it ends the whole block with that Left as the result. It reads like the for in chapter 9’s validateFast with the arrows taken out. Underneath, either uses boundary and break, which have been in Scala’s standard library since 3.3. They’re how direct-style code leaves a block early. A last test for DirectTests:
test("an either block stops at the first Left"):
val bookings = DirectBookings(List())
val good = bookDirect(bookings, BookingRequest("Hughes", "4", "18:30", ""))
assertEquals(good.map(_.table), Right(Table(3, 4)))
val bad = bookDirect(bookings, BookingRequest("Olawale", "six", "19:15", ""))
assertEquals(bad, Left(List(BookingError.BadPartySize("six"))))
17.6. What a signature no longer says
On chapter 14’s Bookings, reserve returns an IO. The signature says it does something to the outside world, and the compiler won’t let the result be used as if it were already a booking. Both the reserve on DirectBookings and decide return an Either. Nothing in the two types shows that one of them changes the list of bookings and the other has no effects.
The Scala team is working on ways to put that information back without the wrapper, through capabilities. The using Ox parameter is one already: a value passed as an ordinary parameter that the compiler keeps track of. Capture checking, an extension to the type system, goes further by following which capabilities a value holds on to. In Scala 3.9 it’s experimental and has to be switched on with a language import, so it isn’t something to build a service on yet.
17.7. Choosing
With Cats Effect a program is a value, and direct style has no counterpart to that. An IO can be handed to retrying, given a timeout or run beside another by code that knows nothing else about it. Its type tells every caller that it has effects. Cancelling works on an IO without the code inside having to cooperate, apart from cases like the IO.blocking one in chapter 15. Cats Effect also has the largest set of service libraries in Scala, including the ones Part IV’s service uses.
There’s very little to learn in direct style. The code reads like the Java it replaces and can call any blocking Java library as it stands. A stack trace shows the path the code took and a debugger steps through it line by line. Against that, the types say less, a val in the wrong place runs too early, and a thread only stops if the code running on it notices the interrupt.
I wouldn’t move a team that already runs Cats Effect services. A team arriving from Java with a new service to write will probably be productive sooner in direct style. Tapir, the library chapter 19 uses for HTTP, is SoftwareMill’s too and has a direct-style server that uses Ox. I’ve kept the rest of this book on Cats Effect because http4s and doobie, the libraries behind Part IV’s service, are built on it, and because the Scala services you might be asked to work on were probably written before virtual threads existed.
17.8. Exercises
Solutions are in the book’s downloadable code, under ch17/exercises.
-
Write
lookUpAllDirect(names: List[String]): List[String], in which each lookup sleeps for 200 milliseconds and all of them run at once. Test that three take well under 600 milliseconds.System.nanoTime()will do for the clock. -
Ox’s
raceSuccess(a, b)runs two computations and returns the result of whichever succeeds first, interrupting the other. Use it to writefirstToAnswerDirect(a: => String, b: => String): String. -
timeoutOptionistimeoutthat returns anOptionwheretimeoutwould throw. Writewithin[A](limit: FiniteDuration)(work: => A): Option[A]. -
Write
seatAllDirect(bookings: List[Booking], at: LocalTime): Either[BookingError, List[Booking]], theseatAllof chapter 9’s exercises, using aneitherblock..ok()works inside the lambda passed tomap.