16. Streams with fs2
An agency takes some of the restaurant’s bookings over the phone. Each night it sends them across as a file with one booking to a line: name, party size, time and phone number. Create nightly.csv in the project directory, beside the Scala files:
Hughes,4,18:30,07700 900201
Banerjee,2,19:00,
Olawale,six,19:15,07700 900203
Kowalczyk,3,19:30,07700 900204
,2,19:45,07700 900205
Mbeki,8,20:00,07700 900206
Tanaka,8,20:00,07700 900207
Dubois,2,7pm,call the hotel
Three of the eight lines have something wrong with them. Lines 6 and 7 are two parties who both want a table for eight at eight o’clock. The import has to book what it can and say what was wrong with the rest. It also has to cope with a file of any length, so reading every line into a List before starting is out.
fs2 is the streaming library that goes with Cats Effect. Two of Part IV’s libraries are built on fs2: http4s carries the body of a request or a response as an fs2 stream, and doobie can return the rows of a query as one.
16.1. Reading the file
Add the dependency to project.scala, under the one for Cats Effect:
//> using dep co.fs2::fs2-io:3.14.0
Then create importer.scala. This first version only prints the file:
package fullybooked
import cats.effect.{ExitCode, IO, IOApp}
import fs2.Stream
import fs2.io.file.{Files, Path}
import org.typelevel.log4cats.Logger
import org.typelevel.log4cats.slf4j.Slf4jLogger
object ImportBookings extends IOApp.Simple:
def run: IO[Unit] =
Files[IO]
.readUtf8Lines(Path("nightly.csv"))
.zipWithIndex
.filter((line, _) => line.trim.nonEmpty)
.evalMap((line, index) => IO.println(s"${index + 1}: $line"))
.compile
.drain
$ scala run . --main-class fullybooked.ImportBookings
1: Hughes,4,18:30,07700 900201
2: Banerjee,2,19:00,
3: Olawale,six,19:15,07700 900203
4: Kowalczyk,3,19:30,07700 900204
5: ,2,19:45,07700 900205
6: Mbeki,8,20:00,07700 900206
7: Tanaka,8,20:00,07700 900207
8: Dubois,2,7pm,call the hotel
The imports are for the finished file. Read the chain in run from the top, as you would a chain of calls on a List: read the lines, number them with zipWithIndex, drop the blank ones, print what’s left. The numbers are Longs counting from 0. I filter after numbering so that the numbers still match the file when a blank line turns up in the middle. There’s usually one blank at the end as well. If the file finishes with a line break, as most editors leave it, readUtf8Lines reports the empty text after it as a last line.
The type that readUtf8Lines returns is Stream[IO, String], a stream of strings that needs IO to produce them. Where map takes an ordinary function, evalMap takes one that returns an IO and runs that IO for each element in turn. At the end, compile.drain turns the stream into an IO[Unit]. Running that IO pulls every element through the stream, so that the effects happen, and throws the elements away.
Nothing in the file is read until IOApp runs that IO. Even then the file is never in memory as a whole. fs2 reads a piece of it, splits the piece into lines, passes them down to the println and lets them go before it reads the next piece.
16.2. A stream is a description
Create streams.test.scala, with imports for all the tests in this chapter, and run the tests:
package fullybooked
import cats.effect.IO
import cats.effect.unsafe.implicits.global
import fs2.Stream
import fs2.io.file.{Files, Path}
import scala.concurrent.duration.*
class StreamTests extends munit.FunSuite:
test("a stream without effects can be endless"):
val sevens = Stream.iterate(1)(_ + 1).filter(_ % 7 == 0).take(3)
assertEquals(sevens.toList, List(7, 14, 21))
test("a stream of effects does nothing until it's compiled and run"):
val program =
for
counter <- IO.ref(0)
ticks = Stream.eval(counter.update(_ + 1)).repeat.take(3)
before <- counter.get
_ <- ticks.compile.drain
after <- counter.get
yield (before, after)
assertEquals(program.unsafeRunSync(), (0, 3))
The first test takes three multiples of seven from a stream of every whole number from 1 upwards. Stream.iterate(1)(_ + 1) has no end. That’s allowed because a stream produces an element only when something downstream asks for one, and take(3) stops asking after the third. A stream with no effects in it can be turned straight into a list with toList.
In the second test ticks is another endless stream, one that adds 1 to a counter for every element, cut off after three. Stream.eval makes a one-element stream out of an IO and repeat goes round it again and again. The counter still read 0 after ticks had been built, because a stream of effects is a description, as an IO is. It reached 3 only when the IO from compile.drain was run.
A stream of effects goes through compile before it can be run. The method called on compile gives an IO and decides what that IO returns: drain returns nothing, toList returns every element, count returns how many there were, last returns the final one in an Option, and fold combines them as foldLeft does.
16.3. The import
Add these to importer.scala, between the imports and ImportBookings:
def requestFrom(line: String): BookingRequest =
val fields = line.split(",", -1).toList.map(_.trim).padTo(4, "")
BookingRequest(fields(0), fields(1), fields(2), fields(3))
case class Rejected(lineNumber: Long, errors: List[BookingError])
def importLine(bookings: Bookings, line: String, lineNumber: Long): IO[Either[Rejected, Booking]] =
validate(requestFrom(line)) match
case Left(errors) =>
IO.pure(Left(Rejected(lineNumber, errors.toList)))
case Right((party, time)) =>
bookings.reserve(party, time).map(_.left.map(error => Rejected(lineNumber, List(error))))
def importFile(bookings: Bookings, path: Path): Stream[IO, Either[Rejected, Booking]] =
Files[IO]
.readUtf8Lines(path)
.zipWithIndex
.filter((line, _) => line.trim.nonEmpty)
.evalMap((line, index) => importLine(bookings, line, index + 1))
The first method, requestFrom, turns a line into the four strings of chapter 9’s BookingRequest. Java’s split drops empty fields from the end of a line unless it’s given -1. padTo(4, "") fills a short line out with empty strings, so fields(3) is always there. A real CSV file can have commas inside quoted fields. For those I’d use a CSV library such as fs2-data-csv.
In importLine the work is done by code from earlier chapters. Chapter 9’s validate collects every problem with the four strings. A request that passes goes to chapter 14’s Bookings, where the reservation is made in one atomic step. A refusal from either of them ends up in a Rejected with the number of its line.
In importFile the stream is the one the first version had, with importLine where the println was. The method returns it without compiling it, so the caller decides what happens to the results. Replace ImportBookings with a version that prints each result and counts the bookings:
object ImportBookings extends IOApp:
val logger: Logger[IO] = Slf4jLogger.getLogger[IO]
def report(result: Either[Rejected, Booking]): IO[Unit] =
result match
case Right(booking) =>
IO.println(label(booking))
case Left(Rejected(lineNumber, errors)) =>
IO.println(s"Line $lineNumber: ${errors.map(explain).mkString("; ")}")
def run(args: List[String]): IO[ExitCode] =
val path = Path(args.headOption.getOrElse("nightly.csv"))
val importing =
for
bookings <- Bookings.starting(List())
imported <- importFile(bookings, path).evalTap(report).filter(_.isRight).compile.count
_ <- logger.info(s"Imported $imported bookings from $path")
yield ExitCode.Success
importing.handleErrorWith: error =>
logger.error(s"Couldn't import $path: $error").map(_ => ExitCode.Error)
Each result is printed by evalTap(report), which runs an effect for each element and passes the element on unchanged. The filter and compile.count after the evalTap leave a single number, the bookings made.
This object extends IOApp where the others have extended IOApp.Simple. Its run receives the command-line arguments and returns an ExitCode for the process to exit with. With no argument it reads nightly.csv:
$ scala run . --main-class fullybooked.ImportBookings
18:30 table 3: Hughes, party of 4
19:00 table 1: Banerjee, party of 2
Line 3: 'six' isn't a number of people
19:30 table 4: Kowalczyk, party of 3
Line 5: we need a name for the booking
20:00 table 6: Mbeki, party of 8 (deposit taken)
Line 7: no table for 8 is free at 20:00
Line 8: '7pm' isn't a time, try 19:30; 'call the hotel' isn't a phone number
INFO ImportBookings: Imported 4 bookings from nightly.csv
Three lines failed validation, line 8 on two counts. The Mbekis got table 6 and the Tanakas, one line later, were refused it. The order of the file decides that, because evalMap finishes one line before it starts the next.
Add a test for the import to the end of StreamTests:
test("the import books what it can and reports the rest"):
val lines = List("Hughes,4,18:30,", "", "Olawale,six,19:15,", "Mbeki,8,20:00,", "Tanaka,8,20:00,")
val program = Files[IO].tempFile.use: path =>
for
_ <- Stream.emits(lines).through(Files[IO].writeUtf8Lines(path)).compile.drain
bookings <- Bookings.starting(List())
results <- importFile(bookings, path).compile.toList
yield results.map(_.left.map(_.lineNumber).map(_.party.name))
assertEquals(program.unsafeRunSync(), List(Right("Hughes"), Left(3L), Right("Mbeki"), Left(5L)))
The test writes five lines to a temporary file, imports it and checks which lines were booked. The temporary file is a Resource, so it’s deleted when use finishes. A stream does the writing too: Stream.emits(lines) makes a stream from the list and through sends it into a pipe. A pipe is what fs2 calls a function from one stream to another. The pipe from writeUtf8Lines writes each element as a line and emits nothing itself, hence the drain. Because the second line of the file is blank, the rejections are on lines 3 and 5, written 3L and 5L as Longs.
16.4. A file bigger than memory
Create bigfile.scala, which writes a CSV file too big to hold in a small heap and then reads it back in two ways:
package fullybooked
import cats.effect.{IO, IOApp}
import fs2.Stream
import fs2.io.file.{Files, Path}
object BigFile extends IOApp.Simple:
def line(n: Int): String =
if n % 10 == 0 then s"Party $n,lots,19:00," else s"Party $n,2,19:00,"
def run: IO[Unit] =
Stream
.range(0, 3000000)
.map(line)
.chunkN(4096)
.unchunks
.through(Files[IO].writeUtf8Lines(Path("big.csv")))
.compile
.drain
@main def checkAtOnce(): Unit =
val lines = java.nio.file.Files.readAllLines(java.nio.file.Path.of("big.csv"))
println(s"${lines.size} lines read")
object CheckFile extends IOApp.Simple:
def run: IO[Unit] =
Files[IO]
.readUtf8Lines(Path("big.csv"))
.filter(line => line.trim.nonEmpty)
.map(line => validate(requestFrom(line)))
.filter(_.isRight)
.compile
.count
.flatMap(valid => IO.println(s"$valid valid requests"))
Run BigFile with scala run . --main-class fullybooked.BigFile. Stream.range(0, 3000000) counts from 0 up to, but not including, three million. The run takes a few seconds and leaves a big.csv of three million lines, nearly 70 megabytes, in which every tenth line has a party size of “lots”.
The @main method checkAtOnce reads the file the way a short Java program might, all of it into a list of lines. It spells out java.nio.file.Files and java.nio.file.Path in full because the short names in this file belong to fs2’s own versions. Run it with the option --java-opt -Xmx64m to limit the JVM to 64 megabytes of heap:
$ scala run . --main-class fullybooked.checkAtOnce --java-opt -Xmx64m
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
A stack trace follows it. Now run CheckFile with the same limit. It puts every line through requestFrom and validate:
$ scala run . --main-class fullybooked.CheckFile --java-opt -Xmx64m
2700000 valid requests
No more than a few thousand lines were in memory at any moment, however long the file is. That only holds while nothing downstream collects the elements. Change count to toList in CheckFile and it runs out of memory as checkAtOnce did. Delete big.csv when you’ve finished with it.
16.5. Chunks
The two lines of BigFile that say chunkN(4096) and unchunks change how the stream’s elements are grouped. Add a test after the import test:
test("elements travel in chunks"):
assertEquals(Stream.range(0, 4).chunks.map(_.size).toList, List(1, 1, 1, 1))
assertEquals(Stream.emits(List(0, 1, 2, 3)).chunks.map(_.size).toList, List(4))
assertEquals(Stream.range(0, 10).chunkN(4).map(_.size).toList, List(4, 4, 2))
Elements move through a stream in batches called chunks, and the chunks method lets a test look at them. Stream.range produces its numbers one to a chunk, where Stream.emits puts everything it’s given into a single chunk. A file is read 64 kilobytes at a time, which comes to a few thousand of our lines. The third assertion regroups ten elements with chunkN(4) and gets chunks of 4, 4 and 2. BigFile follows its chunkN(4096) with unchunks, to get back from a stream of chunks to a stream of elements with the new grouping kept.
The grouping matters when something is done once per chunk, as writing is: writeUtf8Lines makes one write to the file for each chunk it receives. Without those two lines BigFile would make three million separate writes. I timed both versions writing 200,000 lines. The run with one line to a chunk made 200,000 writes. It took several times as long as the run with chunks of 4,096, which made about fifty.
evalMap and evalTap leave chunks of one behind them, since they run an effect for each element. A write that comes after one of them gets a chunk per element again unless a chunkN and an unchunks are put in between.
16.6. When something fails
Add two more tests to the end of StreamTests, about streams that fail:
test("a failure ends the stream unless each element handles its own"):
val numbers = Stream(1, 2, 0, 4)
val stopped = numbers.evalMap(n => IO.delay(12 / n)).compile.toList.attempt
assert(stopped.unsafeRunSync().isLeft)
val carriedOn = numbers.evalMap(n => IO.delay(12 / n).attempt).compile.toList
assertEquals(carriedOn.unsafeRunSync().map(_.toOption), List(Some(12), Some(6), None, Some(3)))
test("a file that isn't there fails the stream when it runs"):
val program =
for
bookings <- Bookings.starting(List())
results <- importFile(bookings, Path("no-such-file.csv")).compile.toList.attempt
yield results.left.map(_.getClass.getSimpleName)
assertEquals(program.unsafeRunSync(), Left("NoSuchFileException"))
The first test makes a stream of four numbers with Stream(1, 2, 0, 4) and divides 12 by each of them inside IO.delay. The third division throws, that effect fails and the stream fails with it. The 4 is never reached, and the IO that compile.toList made fails too, losing the two good answers.
In the second half attempt turns each element’s failure into a value and the stream carries on to the 4. The import never needed attempt for a bad line, because importLine already returns a Left for one and the stream goes on to the next line. An exception thrown inside importLine would still end it, as would a failure to read the file.
The last test gives importFile a file that doesn’t exist. Nothing looks for the file until the stream runs, so importFile returns a stream as usual and the NoSuchFileException appears when that stream is run. In ImportBookings the handleErrorWith around the whole program catches it:
$ scala run . --main-class fullybooked.ImportBookings -- tomorrow.csv
ERROR ImportBookings: Couldn't import tomorrow.csv: java.nio.file.NoSuchFileException: tomorrow.csv
The exit status of that run is 1, from the ExitCode.Error in the handler. Whatever runs the import every night can check that. A stream that fails or is cancelled part-way through still closes the file it was reading, because fs2 opens the file as a Resource.
16.7. Several at once
A stream can also run several effects at once, for work where one element at a time is too slow. One more test for the end of StreamTests:
test("parEvalMap runs several at once and keeps the order"):
val program =
for
busy <- IO.ref(0)
most <- IO.ref(0)
doubled <- Stream
.range(1, 9)
.parEvalMap(3): n =>
for
now <- busy.updateAndGet(_ + 1)
_ <- most.update(_.max(now))
_ <- IO.sleep(50.millis)
_ <- busy.update(_ - 1)
yield n * 2
.compile
.toList
atOnce <- most.get
yield (doubled, atOnce)
assertEquals(program.unsafeRunSync(), (List(2, 4, 6, 8, 10, 12, 14, 16), 3))
With parEvalMap(3), up to three effects run at a time. Each effect in the test adds itself to busy, a count of those in progress, and keeps the highest count seen so far in most. Then it sleeps for a twentieth of a second. The highest count was three. The results still came out in the order the elements went in. parEvalMapUnordered gives them in the order they finish, which can be quicker when one slow element would hold up the rest.
I haven’t used parEvalMap in the import. With several lines in progress at once, which of the two parties got table 6 would depend on timing and no longer on the order of the file. I’d use parEvalMap where the elements have nothing to do with one another, for example to send a confirmation for every booking the import made.
16.8. Exercises
Solutions are in the book’s downloadable code, under ch16/exercises.
-
Write
seatings, a stream of every quarter of an hour fromopenstolastOrders, usingStream.iterate. A stream hastakeWhile, andLocalTimehasplusMinutesandisAfter. -
Write
rejections(bookings: Bookings, path: Path): IO[Map[String, Int]], which counts how many times each reason for rejecting a line came up, in one pass over the file.collectandflatMapwork on a stream, andcompile.foldtakes a starting value and a function asfoldLeftdoes. -
Write
writeRejected(bookings: Bookings, from: Path, to: Path): IO[Unit], which puts a line of text for each rejected line into a second file. -
Write
importFileInParallel, which isimportFilewithparEvalMap(4), and importnightly.csvwith it. The Mbekis will probably still get table 6. Why can’t you rely on that? -
Take
chunkNandunchunksout ofBigFile, reduce the range to 200,000 and time it with and without them.IO.monotonicreads a clock. There’s no solution for this one, but my figures are in the chapter.