22. Shipping it
Starting the service with sbt "service/runMain fullybooked.Server" needs sbt, the source code and a compiler on the machine. It also needs a database password that’s sitting in a file in the repository. In this chapter we package the service so that a machine with only Java 25 can run it, with the password taken out of the file. We then add endpoints that tell whoever operates the service how it’s doing, and finish with the service in a container beside its database.
22.1. A package
An sbt plugin called sbt-native-packager does the packaging. Add it to project/plugins.sbt:
addSbtPlugin("com.github.sbt" % "sbt-native-packager" % "1.13.0")
In build.sbt, switch it on for service and add two settings at the top of that subproject’s list:
lazy val service = project
.dependsOn(core)
.enablePlugins(JavaAppPackaging)
.settings(
Compile / mainClass := Some("fullybooked.Server"),
bashScriptExtraDefines += """addJava "--sun-misc-unsafe-memory-access=allow"""",
The service subproject has eleven entry points left over from earlier chapters, so mainClass tells the plugin which one to start. The other setting adds a line to the start script that the plugin writes, to pass the JVM option we’ve been using since chapter 6. It’s in triple quotes because it has quotes of its own. Add one more line near the top of the file, under version:
Compile / doc / sources := Seq()
Without it, packaging also builds the API documentation for our code, which takes a little longer and prints a warning about a comment in somebody else’s library. Run sbt service/stage to build the package into a directory:
$ ls target/out/jvm/scala-3.9.0/service/universal/stage
bin
lib
The bin directory has a start script for each entry point, with a .bat version for Windows. The one called bin/service starts the server on a JVM with every jar in lib on its classpath. There are eighty of those jars: our code and every library it depends on.
22.2. Secrets from the environment
The database block that we added to application.conf in chapter 20 has a password in it. Give user an override and replace the password, so that the block reads:
database {
url = "jdbc:postgresql://localhost:5432/fullybooked"
url = ${?DATABASE_URL}
user = "postgres"
user = ${?DATABASE_USER}
password = ${DATABASE_PASSWORD}
}
With no default and no question mark on the password line, the service can’t start unless DATABASE_PASSWORD is set. Run sbt service/stage again so that the package has the new file, then start the packaged service:
$ target/out/jvm/scala-3.9.0/service/universal/stage/bin/service
pureconfig.error.ConfigReaderException: Cannot convert configuration to a fullybooked.AppConfig. Failures are:
- (application.conf @ .../application.conf: 11) Unable to parse the configuration: Could not resolve substitution to a value: ${DATABASE_PASSWORD}.
Those two lines come with a stack trace under them. Set the variable in your terminal with export DATABASE_PASSWORD=table-for-two. From now on every program that loads the configuration needs it, including LastTableDb and the server when they’re run through sbt. The tests don’t load the configuration, so they pass without the variable. In production, a platform such as Kubernetes runs the service and sets the variable from its own store of secrets.
22.3. Health and readiness
The platform that runs a service asks whether its process is alive, so that it can restart a dead one. It also asks whether the service can do its job right now, which for ours means being able to reach the database. When there are several copies of a service, requests go only to those that can. Add a method to the Bookings trait in concurrency.scala:
def ready: IO[Boolean]
Add this to InMemoryBookings, below all, since that class is always ready:
def ready: IO[Boolean] = IO.pure(true)
For PostgresBookings the answer comes from the database. Add import scala.concurrent.duration.* to the imports of postgres.scala, and add this method to the class, below the method called all:
def ready: IO[Boolean] =
sql"SELECT 1".query[Int].unique.transact(xa).timeout(2.seconds).attempt.map(_.isRight)
The database gets two seconds to answer SELECT 1, a query that reads no table. With attempt, a failure or a timeout becomes a Left, and ready answers false.
The two seconds in ready don’t cover the wait for a connection. Getting one from the pool is a blocking call, and chapter 15 showed that a timeout can’t interrupt one of those. With the database gone, the pool goes on trying to open a connection for thirty seconds by default, and /ready would take that long to answer. In transactor in database.scala, add a line under the one that sets the size of the pool:
pool.setConnectionTimeout(2000)
The figure is in milliseconds. It covers every wait for a connection, so a booking made while the database is down no longer keeps the client waiting for thirty seconds.
Two endpoints go at the end of api.scala:
val healthEndpoint = endpoint.get.in("health").out(stringBody)
val readyEndpoint =
endpoint.get
.in("ready")
.out(stringBody)
.errorOut(statusCode(StatusCode.ServiceUnavailable))
.errorOut(stringBody)
and their logic at the end of routes.scala:
def opsRoutes(bookings: Bookings): List[ServerEndpoint[Fs2Streams[IO], IO]] =
List(
healthEndpoint.serverLogicSuccess(_ => IO.pure("ok")),
readyEndpoint.serverLogic: _ =>
bookings.ready.map(up => if up then Right("ready") else Left("the database isn't answering"))
)
Both answer in plain text, which is what stringBody describes. /health doesn’t touch the database, because a slow database is no reason to restart the service. /ready answers 503 when the database can’t be reached.
22.4. Metrics, a time limit and a clean stop
Add two more libraries to the service list in build.sbt, after the line for Flyway:
"org.http4s" %% "http4s-prometheus-metrics" % "0.25.0",
"net.logstash.logback" % "logstash-logback-encoder" % "9.0",
Replace server.scala once more:
package fullybooked
import cats.effect.{IO, IOApp}
import cats.syntax.all.*
import com.comcast.ip4s.{ipv4, Port}
import org.http4s.{HttpApp, HttpRoutes}
import org.http4s.ember.server.EmberServerBuilder
import org.http4s.metrics.prometheus.{Prometheus, PrometheusExportService}
import org.http4s.server.middleware
import org.typelevel.log4cats.Logger
import org.typelevel.log4cats.slf4j.Slf4jLogger
import scala.concurrent.duration.*
import sttp.tapir.server.http4s.Http4sServerInterpreter
import sttp.tapir.swagger.bundle.SwaggerInterpreter
def httpRoutes(title: String, bookings: Bookings): HttpRoutes[IO] =
val routes = bookingRoutes(bookings)
val docs = SwaggerInterpreter().fromServerEndpoints[IO](routes, title, "0.1.0")
Http4sServerInterpreter[IO]().toRoutes(routes ++ docs ++ opsRoutes(bookings))
def httpApp(title: String, bookings: Bookings): HttpApp[IO] =
httpRoutes(title, bookings).orNotFound
object Server extends IOApp.Simple:
val logger: Logger[IO] = Slf4jLogger.getLogger[IO]
def run: IO[Unit] =
val service =
for
config <- loadConfig
applied <- migrate(config.database)
_ <- logger.info(s"$applied migrations applied")
port <- portFrom(config)
_ <- transactor(config.database).use(xa => serve(config, port, PostgresBookings(xa)))
yield ()
service.guarantee(logger.info("Fully Booked has stopped"))
def portFrom(config: AppConfig): IO[Port] =
IO.fromOption(Port.fromInt(config.port))(
IllegalArgumentException(s"${config.port} isn't a port")
)
def serve(config: AppConfig, port: Port, bookings: Bookings): IO[Unit] =
val server =
for
exporter <- PrometheusExportService.build[IO]
ops <- Prometheus.metricsOps[IO](exporter.collectorRegistry, "fully_booked")
measured = middleware.Metrics[IO](ops)(httpRoutes(config.restaurant, bookings))
app = (exporter.routes <+> measured).orNotFound
limited = middleware.Timeout.httpApp[IO](10.seconds)(app)
server <- EmberServerBuilder
.default[IO]
.withHost(ipv4"0.0.0.0")
.withPort(port)
.withHttpApp(middleware.Logger.httpApp(logHeaders = false, logBody = false)(limited))
.withShutdownTimeout(5.seconds)
.build
yield server
server.useForever
Most of httpRoutes is the old httpApp. The two new endpoints join the list after the documentation has been built, and I leave them out of it because the people who call /health and /ready are operating the service. The new function returns the routes before orNotFound has turned them into an application, since routes can still be joined to other routes and an application can’t. The tests still call httpApp, which now only adds orNotFound.
The for in serve runs over three resources: the exporter, the ops value that records the numbers, and the server. A for over Resources makes a single Resource, which releases the server first and the exporter last, the reverse of the order they were acquired in. The first two lines are for Prometheus, a widely used monitoring system that collects a service’s numbers by fetching a page of them at regular intervals. The Metrics middleware counts every request to our routes and times every response. The exporter serves those numbers at /metrics from routes of its own, which are joined to ours with <+>. That operator tries the routes on its left first. I’ve put the exporter’s there to keep Prometheus’s own fetches out of the counts.
The Timeout middleware goes round the whole application, metrics included. A request that hasn’t been answered in ten seconds gets a 503, and the fiber working on it is cancelled. When a fiber is cancelled doobie cancels its query, so a stuck query can’t hold a connection for ever.
When the process is told to stop, IOApp cancels run, and that releases every Resource, the server first. The server gives connections that are still open five seconds to finish, the figure passed to withShutdownTimeout. That’s half of the ten seconds Docker waits before it kills a container that hasn’t stopped. Then the pool closes and the guarantee in run logs a line.
Add a test for the new endpoints to the end of HttpTests:
test("the service says it's alive and ready"):
for
bookings <- Bookings.starting(List())
app = httpApp("Fully Booked", bookings)
health <- app.run(Request[IO](Method.GET, uri"/health"))
ready <- app.run(Request[IO](Method.GET, uri"/ready"))
answer <- ready.as[String]
yield
assertEquals(health.status.code, 200)
assertEquals((ready.status.code, answer), (200, "ready"))
$ sbt "service/test fullybooked.HttpTests"
[info] Passed: Total 4, Failed 0, Errors 0, Passed 4
[success] elapsed time: 2 s
22.5. Logs for machines
Our log lines are laid out for a person reading a terminal. In production they usually go to a program first, and it wants one JSON object per line. The second of the two libraries we added to build.sbt in the last section, the Logstash encoder for Logback, writes them in that form. Create logback-json.xml beside logback.xml:
<configuration>
<statusListener class="ch.qos.logback.core.status.NopStatusListener"/>
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
</appender>
<logger name="org.flywaydb" level="WARN"/>
<logger name="com.zaxxer.hikari" level="WARN"/>
<root level="INFO">
<appender-ref ref="console"/>
</root>
</configuration>
Logback goes on using logback.xml unless the system property logback.configurationFile names another file. We’ll set that property only in the container.
22.6. An image
Run sbt service/stage again to pick up the changes. Then create a file called Dockerfile in the project’s top directory:
FROM eclipse-temurin:25-jre
RUN useradd --system --no-create-home fullybooked
WORKDIR /app
COPY --chmod=755 target/out/jvm/scala-3.9.0/service/universal/stage/ /app/
USER fullybooked
ENV JAVA_OPTS="-Dlogback.configurationFile=logback-json.xml"
EXPOSE 8080
ENTRYPOINT ["/app/bin/service"]
The image starts from one that has a Java 25 runtime and nothing to compile with. The RUN line makes a user for the service so that the process doesn’t run as root, and COPY brings the staged directory in. --chmod=755 is there because the packager makes its scripts executable for their owner alone, and in the image the owner is root. The bin/service script reads JAVA_OPTS, which here holds the system property that names the JSON logging file. Create a .dockerignore file beside the Dockerfile to keep everything except the staged directory out of the build:
**
!target/out/jvm/scala-3.9.0/service/universal/stage/**
sbt-native-packager can write a Dockerfile and build the image itself. I write the file by hand, because it’s a few lines and the people who run the service can read it.
22.7. The service and its database together
Docker Compose starts several containers from one description. Create compose.yaml in the same directory:
services:
db:
image: postgres:18
environment:
POSTGRES_DB: fullybooked
POSTGRES_PASSWORD: ${DATABASE_PASSWORD}
healthcheck:
test: ["CMD", "pg_isready", "-U", "postgres", "-d", "fullybooked"]
interval: 2s
retries: 15
service:
build: .
environment:
DATABASE_URL: jdbc:postgresql://db:5432/fullybooked
DATABASE_PASSWORD: ${DATABASE_PASSWORD}
ports:
- "8080:8080"
depends_on:
db:
condition: service_healthy
Both passwords come from the DATABASE_PASSWORD in your terminal, so export the variable again if you’ve opened a new one. Inside Compose’s network the database’s host name is db, the name in the URL. Because of depends_on, the service isn’t started until Postgres answers pg_isready. Port 8080 has to be free, so stop any server that’s still running from an earlier chapter. Then build and start everything:
$ docker compose up -d --build
Give the service a couple of seconds to start, then ask it how it is:
$ curl localhost:8080/health
ok
$ curl localhost:8080/ready
ready
Book a table with one of chapter 19’s curl commands, then look at what the service has counted:
$ curl -s localhost:8080/metrics | grep ^fully_booked_request_count_total
fully_booked_request_count_total{classifier="",method="get",status="2xx",} 2.0
fully_booked_request_count_total{classifier="",method="post",status="2xx",} 1.0
The 2 beside get counts the requests just made to /health and /ready. The logs are JSON now:
$ docker compose logs --no-log-prefix service
...
{"@timestamp":"2026-11-14T19:30:00.000Z","@version":"1","message":"HTTP/1.1 POST /bookings","logger_name":"org.http4s.server.middleware.Logger","thread_name":"io-compute-1","level":"INFO","level_value":20000}
{"@timestamp":"2026-11-14T19:30:00.000Z","@version":"1","message":"HTTP/1.1 201 Created","logger_name":"org.http4s.server.middleware.Logger","thread_name":"io-compute-1","level":"INFO","level_value":20000}
...
Stop the database underneath the service, wait a few seconds and ask again:
$ docker compose stop db
$ curl -w '\n%{http_code}\n' localhost:8080/ready
the database isn't answering
503
A request to /health still gets ok, because that endpoint never asks the database. Stop the service and read the end of its log:
$ docker compose stop service
$ docker compose logs --no-log-prefix service
...
{"@timestamp":"2026-11-14T19:30:00.000Z","@version":"1","message":"Fully Booked has stopped","logger_name":"fullybooked.Server","thread_name":"io-compute-1","level":"INFO","level_value":20000}
Run docker compose down to remove the containers.
22.8. Exercises
Solutions to the first two are in the book’s downloadable code, under ch22/exercises.
-
The pool has eight connections because
database.scalasays so. Add apool-sizesetting to the database block, with a default of 8 and an environment variableDATABASE_POOL_SIZEto override it, and use it intransactor. The fixture inpostgres.test.scalabuilds aDatabaseConfigas well and will need the new field. -
Write a test that gives
httpAppaBookingswhosereadyis false, and check that/readyanswers 503 while/healthstill answers 200.new Bookings:followed by the three methods makes one on the spot. -
With everything running under Compose, stop the database and try to book a table. What does the client get, and how long does it wait? Which of chapter 15’s tools would improve on that, and where would it go?