Developing software is about more than just writing code - you have to debug as well. So why not make debugging as painless as possible?

With some errors, we write the stack trace to the log. The IDE we use (Idea, GoLand) allows for convenient file navigation using a copied call stack (Analyze external stack traces). Unfortunately, this feature only works well if the binary is built on the same host that the IDE is running on.

This post is about how we tried to create synergy between the call stack format and the IDE.

What kind of stack display options does Go Build provide?

There are two controls in Go Build for influencing the stack output format:

• the -trimpath flag: causes the display of the call stack to be the same, regardless of the local location of the files;
• the GOROOT_FINAL environment variable: lets you replace the prefix to system libraries on a stack when the -trimpath flag is off.

A program for comparing stack trace output

Let’s consider stack mapping using the example of a small program. The source code can be downloaded here, and here’s the program (stacktrace/main.go):

package main

import (
	"fmt"

	"github.com/Masterminds/cookoo"
	"github.com/pkg/errors"
)

func main() {
	// Build a new Cookoo app.
	registry, router, context := cookoo.Cookoo()
	// Fill the registry.
	registry.AddRoutes(
		cookoo.Route{
			Name: "TEST",
			Help: "A test route",
			Does: cookoo.Tasks{
				cookoo.Cmd{
					Name: "hi",
					Fn:   HelloWorld,
				},
			},
		},
	)
	// Execute the route.
	router.HandleRequest("TEST", context, false)
}

func HelloWorld(cxt cookoo.Context, params *cookoo.Params) (interface{}, cookoo.Interrupt) {
	fmt.Printf("%+v\\n", errors.New("Hello World"))
	return true, nil
}

And then let’s add a little go.mod:

module github.com/bozaro/go-stack-trace

go 1.20

require (
	github.com/Masterminds/cookoo v1.3.0
	github.com/pkg/errors v0.9.1
)

The tried and true GOPATH

To keep things in perspective, let's start with some good old GOPATH.

Sample output:

➜ GO111MODULE=off GOPATH=$(pwd) go get -d github.com/bozaro/go-stack-trace/stacktrace
➜ GO111MODULE=off GOPATH=$(pwd) go run github.com/bozaro/go-stack-trace/stacktrace 
Hello World
main.HelloWorld
	/home/bozaro/gopath/src/github.com/bozaro/go-stack-trace/stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	/home/bozaro/gopath/src/github.com/Masterminds/cookoo/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	/home/bozaro/gopath/src/github.com/Masterminds/cookoo/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	/home/bozaro/gopath/src/github.com/Masterminds/cookoo/router.go:131
main.main
	/home/bozaro/gopath/src/github.com/bozaro/go-stack-trace/stacktrace/main.go:27
runtime.main
	/usr/lib/go-1.20/src/runtime/proc.go:250
runtime.goexit
	/usr/lib/go-1.20/src/runtime/asm_amd64.s:1598

Everything here is straightforward - we see the full paths to each file. In this case, all paths are located either in the src directory of GOROOT or in the GOPATH directory.

Unfortunately, such a stack only points to existing files if the executable is built in an environment with the same directory layout. In our case, where some developers use MacOS, and the product build environment uses Linux, this requirement is not feasible.

Luckily, there’s a -trimpath flag that removes the troublesome part of the call stack:

➜ GO111MODULE=off GOPATH=$(pwd) go run -trimpath github.com/bozaro/go-stack-trace/stacktrace
Hello World
main.HelloWorld
	github.com/bozaro/go-stack-trace/stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	github.com/Masterminds/cookoo/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	github.com/Masterminds/cookoo/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	github.com/Masterminds/cookoo/router.go:131
main.main
	github.com/bozaro/go-stack-trace/stacktrace/main.go:27
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1598

In this case, all paths will relative to either GOPATH or src in the GOROOT directory, and it ended up being a completely portable call stack format.

Go Modules

When using Go Modules, the behavior of the -trimpath flag changes dramatically. Let’s compare the output of the call stack without it:

➜ git clone <https://github.com/bozaro/go-stack-trace.git> .
➜ go run ./stacktrace 
Hello World
main.HelloWorld
	/home/bozaro/github/go-stack-trace/stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	/home/bozaro/go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	/home/bozaro/go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	/home/bozaro/go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:131
main.main
	/home/bozaro/github/go-stack-trace/stacktrace/main.go:27
runtime.main
	/usr/lib/go-1.20/src/runtime/proc.go:250
runtime.goexit
	/usr/lib/go-1.20/src/runtime/asm_amd64.s:1598

And a similar output with -trimpath:

➜ go run -trimpath ./stacktrace
Hello World
main.HelloWorld
	github.com/bozaro/go-stack-trace/stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	github.com/Masterminds/cookoo@v1.3.0/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	github.com/Masterminds/cookoo@v1.3.0/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	github.com/Masterminds/cookoo@v1.3.0/router.go:131
main.main
	github.com/bozaro/go-stack-trace/stacktrace/main.go:27
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1598

Without -trimpath we still see full paths to each file, while clearly tracing three types of source files:

• a working copy directory (in this example: $HOME/github/go-stack-trace);
• GOROOT system libraries from $GOROOT/src (in this example: /usr/lib/go-1.20/src);
• third-party libraries from $GOMODCACHE (in this example: $HOME/go/pkg/mod);

At the same time, unlike GOPATH, the -trimpath flag does not cut off the prefix in file names:

• files from the current module in the working copy directory are named with the module name from go.mod as a prefix (in this example: $HOME/github/go-stack-trace → github.com/bozaro/go-stack-trace);
• GOROOT system libraries from $GOROOT/src get filenames without a prefix;
• third-party libraries get the module name with the version as a prefix (in this example: /home/bozaro/go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0 → github.com/Masterminds/cookoo@v1.3.0, and it’s worth noting that the word Masterminds in the file path and module name is spelled differently).

Which stack trace is IDE-friendly?

If you happen to open a project from the repository in Idea / GOROOT and try to analyse any of the above call stacks, then there won’t be any navigation through the source files:

• call stack options for GOPATH aren’t suitable because this mini-project uses Go Modules and has a different file layout;
• the option for Go Modules without -trimpath won't work because your home directory will most likely be different from /home/bozaro;
• the option for Go Modules with -trimpath won't work as it’s not supported in IDE (https://youtrack.jetbrains.com/issue/GO-13827), and of all the paths that are visible on the stack, only files from the Go SDK will be the suffixes of existing files.

It seems like the IDE in our case is looking for source files along the paths relative to the project directory and its parents. As a result, an acceptable format for a portable call stack is as follows:

main.HelloWorld
	stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:131
main.main
	stacktrace/main.go:27
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1598

As a result:

• paths to project files are displayed relative to the project root;
• as paths to third-party dependencies, the path to the module relative to $GOMODCACHE is used, but with the prefix go/pkg/mod (IDE will find this path when the project is in the home directory, and the environment variables GOPATH and GOMODCACHE have a default value);
• we use GOROOT as Go SDK file prefix. This marker allows you to visually identify these files, but we failed to get the IDE to provide navigation on them.

With this call stack format, the IDE recognizes all files except those from the Go SDK. The whole thing breaks if a developer locally overrides the GOPATH or GOMODCACHE environment variables, but there’s generally no need to do this.

Getting the call stack in the right format

This can be done using the following methods:

• modify debug infromation in compile time;
• before output, convert the call stack to the desired format;
• make an external utility that converts the call stack to the required format.

Modify debug information in compile time

With Go Build, we cannot modify debug information in compile time to get the desired format of the stack trace.

Stack conversion before output

In our case, we use the library github.com/joomcode/errorx across the board, and it has a method for converting the call stack to the desired format before output (you can find that here).

Converting a path from a view without -trimpath is trivial, but this method has a number of disadvantages:

• if the call stack is passed by this filter, then it will remain in the original format;
• some places, like pprof, are guaranteed to be transmitted in their original format.

External utility

Using an external utility greatly complicates the overall call stack parsing scenario. In our case (and in most cases), the stack was taken from the logs where it was already in the necessary format, so we didn’t seriously consider this option.

Migration to Bazel forced me to tackle this problem again

In general, converting a stack before log write was suitable to the Bazel build migration. But building by Bazel brought the problem to a new level.

Stack from the binary built by Bazel

➜ bazel run //stacktrace 
...
INFO: Running command line: bazel-bin/stacktrace/stacktrace_/stacktrace
Hello World
main.HelloWorld
	stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	external/com_github_masterminds_cookoo/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	external/com_github_masterminds_cookoo/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	external/com_github_masterminds_cookoo/router.go:131
main.main
	stacktrace/main.go:27
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	src/runtime/asm_amd64.s:1598

We don't require developers to to use Bazel to build and run tests for a number of reasons, some of which are:

• we generate BUILD files with our utility and don’t want to require regeneration of files for every change (it’s fast, but not instantaneous);
• Synchronization of IDE and BUILD files is rather slow.

At the same time, in the call stack from Bazel:

• third-party libraries begin to refer to the external directory, which the IDE doesn’t see;
• it’s impossible to get the path to the module in GOMODCACHE in a straightforward way - information about the module version is lost;
• generated files can get a completely unexpected prefix like bazel-out/k8-fastbuild-ST-2df1151a1acb/....

All of these paths refer to real files and are quite meaningful in the context of Bazel, but without full integration they only complicate the matter.

Stack conversion before output

Initially, they tried to assemble a set of rules that allow you to form something acceptable from the existing call stack.

To do this (through x_defs and then embed ), a separately generated file was passed to the program, which contained the correspondence of the external name to the desired prefix in the call stack.

We also made a number of transformations to process the paths of generated files. The problem became less acute, but the results were still unsatisfactory:

• pprof became a complete nightmare;
• some of the paths were converted incorrectly;
• the structure as a whole was quite complex and fragile.

External utility

I didn’t want to have to go down this path: in addition to all the complexity and fragility when converting the stack before output, there was also the problem of putting information to this utility that we integrated into the executable file, namely, matching the external name to the desired prefix in the call stack.

That is, it seemed necessary to do a stack trace deobfuscator. The problem is that we would obfuscate the code only from ourselves.

Modify debug information in compile time

When using Bazel, the build is at a lower level than Go Build. We had hoped to fix the assembly in order to immediately have convenient paths to files. The $(go env GOTOOLDIR)/compile utility also has a -trimpath option, but this parameter is no longer a boolean flag, but rather a list for replacing prefixes.

As a result, we added additional attributes to the go_library and go_repository rules so that we could set stack trace prefix for the library:

• https://github.com/bazelbuild/rules_go/pull/3307
• https://github.com/bazelbuild/bazel-gazelle/pull/1379

After these changes, you can override the path of files in the call stack, for example:

diff --git a/deps.bzl b/deps.bzl
index ffe4981..d917282 100644
--- a/deps.bzl
+++ b/deps.bzl
@@ -5,6 +5,7 @@ def go_dependencies():
         name = "com_github_masterminds_cookoo",
         importpath = "github.com/Masterminds/cookoo",
         sum = "h1:zwplWkfGEd4NxiL0iZHh5Jh1o25SUJTKWLfv2FkXh6o=",
+        stackpath = "go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0",
         version = "v1.3.0",
     )
     go_repository(
@@ -12,4 +13,5 @@ def go_dependencies():
         importpath = "github.com/pkg/errors",
         sum = "h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=",
         version = "v0.9.1",
+        stackpath = "go/pkg/mod/github.com/pkg/errors@v0.9.1",
     )

A sample output in bazel branch:

➜ git checkout bazel
➜ bazel run //stacktrace
INFO: Running command line: bazel-bin/stacktrace/stacktrace_/stacktrace
Hello World
main.HelloWorld
	stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	go/pkg/mod/github.com/!masterminds/cookoo@v1.3.0/router.go:131
main.main
	stacktrace/main.go:27
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	src/runtime/asm_amd64.s:1598

NOTE: For some reason, the Gazelle patch is not picked up by itself. If an error like flag provided but not defined: -stack_path_prefix occurs while running the example, then to fix it, you need to rebuild Gazelle itself. In this case, the easiest way to reset the Bazel cache is: bazel clean --expunge && bazel shutdown.

Conclusion

As a result, we were able to ensure that the stack trace copied from the log was correctly recognized in the IDE. At the same time, this extended to all sources of stack traces, including, for example, prof full goroutine stack dump.

To do this, we had to patch up rules_go and gazelle a little, but we hope that these changes will someday get into the upstream.

Any comments, suggestions and words of support (or condemnation) are welcome, and stories about your own experience are more than welcome!

The battle for an IDE-friendly stack trace in Go (with and without Bazel) was originally published in JoomTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi, My name’s Artem, and I’m a backend developer in the client backend team. One of the important parts of my job is reducing the latency of our backend. That exact issue is what inspired this article, so let’s get into it.

In one of our checkout endpoints, the 99th percentile latency breaks the SLO. We need to fix this.

Accordingly, the question arises: How can we quickly and accurately find the cause of slowdowns on a very low-frequency request at the 99th percentile, and what can we do to eliminate it? The answer turned out to be a library for semi-automatically searching for bottlenecks in distributed systems. You can find the relevant GitHub link at the end of the article.

What are the standard tools for finding performance bottlenecks?

Several of these tools are already available in the public domain.

• Profiling, e.g. with pprof.
• Building dashboards with performance metrics. For example, Prometheus + Grafana.
• Studying the execution traces of the requests we are interested in. We obtain these, for example, using Jaeger.

Let’s take a closer look at these three paths , including their advantages and disadvantages.

Profiling won’t help in a distributed system

Profiling helps with find runtime, CPU, and memory bottlenecks. In the case of the Go ecosystem, the profiler is generally available right out of the box. Therefore, if we have a dominator in the code that’s not only going to the database, but also mining crypto for a neighboring team, then they can easily be found in this way.

But a profiler will not help if we are talking about a distributed system with dozens (or even hundreds) of services written in different languages. In such systems, it’s not always possible to uniquely identify the problematic service that needs to be profiled. Therefore, we want to have a tool that can analyse the performance of the entire system as a whole and identify problems across all services at once.

Such a tool is super useful if you need to speed up the backend, but the problem becomes responsibility of some other team. In this scenario, the work goes much more smoothly if the task is formulated as “John, your service slows down on request X to base Y — try to speed it up, please”, and not “John, it seems that your service is slowing down somewhere — make it faster.”

In our case, when accelerating a request, it turned out that the neighboring service was slowed down due to the fact that the data was in an overloaded database. Migration to a nearby base should help, and since we already knew exactly what to do, the team responsible for this service was able to complete the migration. As a result, optimisation was completed fairly quickly.

Another problem with profilers is that we only get information about how the service performs in general. That is, if we have some critical, but low-frequency request that doesn’t make a serious contribution to the total resource consumption, we won’t be able to understand how it can be accelerated using just the profiler.

For example, in our backend, some of the most frequent requests are requests for a product card and the main page. Any bottlenecks in these areas will stand out and easily be found! On the other hand, shopping carts, checkouts and user profiles are requested thousands of times less often, meaning even if these queries have very serious performance problems, they will be almost invisible on profiles. This is especially the case if these problems appear only at higher percentiles (for example, p95 or p99). In this case, getting the profiler to answer the question “What slows down in a certain query at a certain percentile?” is not possible at all.

Long story short: Profiling can help find global bottlenecks in a particular service, but this tool is not suited to identifying problems in specific user scenarios in distributed systems.

Dashboards with performance metrics — you can’t please everyone

If you already have a dashboard that shows the latency of the problem area, that’s great! But the fact is that in order to build such a dashboard, you first have to find the problem. Building each dashboard requires some effort, and building them in advance for all occasions is more or less unrealistic.

Returning to the example from the previous section, it’s highly improbable that someone would build a dashboard for a very specific request to the exact collection in the database given there are hundreds of collections there. But even if such a dashboard suddenly appeared, how would you know to check it?

Therefore, instead of a bunch of disparate dashboards, it’s better to have a tool that would passively do the working of collecting all the information (more or less) necessary for performance analysis, and then would allow us to make queries of arbitrary complexity to the collected data and highlight probable problems.

Manual viewing of Jaeger traces is great, but something is missing

Jaeger traces are tree structures, each node of which contains the execution time of the corresponding method in the process of processing a request.

Collecting such traces allows you to eliminate all the previously mentioned disadvantages of profilers and dashboards.

• The trace has parameters such as endpoint and execution time. We can focus on just the requests we really need.
• The trace may contain information about all services involved in processing a request. We can at a glance cover a distributed system with an arbitrary number of components.
• Traces are collected passively. If we decide to see why a shopping cart slows down for us, we don’t need to start wrapping it with metrics and building a new dashboard — simply ask Jaeger for the appropriate traces.

But unfortunately, this tool also has problems.

• Long traces (up to several thousand spans). Problems can be difficult to spot by eye.
• On the higher percentiles (p95, p99) things are tricky. From single traces it’s difficult to understand what slows down the most — the slowdown can be different for different traces, such as the database, the cache, the search, etc. It can be difficult to understand where to concentrate your efforts.
• Jaeger’s UI does not allow you to conveniently look at the latency of individual subsystems. You have to open the entire trace, and only then look for the necessary calls.
• There is no way to estimate how much the proposed optimisations will improve the performance of the entire system as a whole.

Returning to the example from the section on profiling — although the slowdowns in that collection were the main problem, we didn’t find long queries on every trace. It was also difficult to unequivocally decide if it was worth focusing efforts on collection migration as opposed to, for example, optimising requests to Redis.

Our solution — Jaeger + Spark

Of all the ways to find bottlenecks, the analysis of Jaeger traces seemed to be the most promising. But in order not to have to deal with viewing single traces manually, we decided to upload them and analyse them using third-party tools.

When the question arose of how to build some kind of aggregated statistics on top of traces, the first thing that came to mind was Spark (spoiler: we stopped there). At Joom, we use Spark for many different tasks, such as:

• building dashboards based on client analytics events,
• processing of historical logs,
• building pipelines for our ML teams.

So, in order to analyse traces in Spark, you must first upload them. To do this, we used Jaeger-collector (the service responsible for forwarding traces to long-term storage) in order to use Kafka as secondary storage. Next, the periodic job extracts traces from Kafka and stores them in S3, and from there we can read them in our spark jobs.

Now the traces are at our disposal, and all that remains is to decide what to do with them.

A search on the web for ready-made solutions for batch analysis of execution traces led us only to a single Microsoft article. But despite its effectiveness, the algorithm proposed in the article is quite difficult to implement, and also needs to be adapted to our needs.

In this regard, we set ourselves the task of writing a tool that, based on the collected traces, would be able to answer the following questions.

1. I have a set of requests of a certain endpoint with a certain duration. Which calls in this slice takes the most time?
2. If I manage to speed up some calls (by a percentage or by a fixed amount), then what will the latency percentiles of the corresponding slices that I am interested in be?

To solve the first question, we decided to follow the simplest path — we select the required trace slice and inside it summarised the durations of all spans by the name of the operation. An arbitrary string can appear as an operation name. For example, the full name of the method. After that, we can sort in descending order the execution time of all the spans and understand which one is potentially the most difficult. This is a rather rough estimate, but for our purposes, it still turned out to be quite useful.

With the second question, everything turned out to be somewhat more complicated, since calls can be performed both sequentially and in parallel. Let’s say that a call to a third-party service takes 100 ms. At best, a 95% speedup here can at best speed up the entire query by 95 ms overall.

And at worst, it won’t speed up at all if the trip to this service was done in parallel with an even longer request.

In a trivial case, we can just look into the codebase and find out exactly how the request is made in relation to all other requests in the trace. But if our codebase is big enough, it’s easy to miss running a new goroutine six frames up the stack and waste time optimising something that won’t speed things up overall.

Therefore, when writing our library, we took a more reliable path — using Scala code, we apply hypothetical optimisation to each trace from the slice. Hypothetical optimisations can take the following forms:

• operation X is accelerated by Y%;
• operation X is accelerated by Y ms;
• all X operations after the first one are accelerated by 100% (this formulation is useful in case of caching repeated calls within one trace);
• …and potentially many more.

After that, based on the obtained synthetic traces, the latency distribution is constructed over the percentiles of interest for each of the synthetic optimisations. Based on the results of such a simulation, we have a fairly reliable estimate of what we’ll get from optimising the operation of one or another system component. Now you can start actually refactoring the code!

Let’s get to work!

Let’s use a real example to see how the approach described above helped us find a problem in one of our checkout flow requests.

Searching for optimisation

First of all, we look at the details of what the service does during the execution of the request.

On this graph, operations are plotted along the horizontal axis, and the ratio of their total execution time to the total execution time of all traces in the slice is shown along the vertical axis. That is, on average, each call to the checkoutdata.GetBundle method takes 15% of the total request time. Sometimes this is enough information to start thinking about where to look for bottlenecks. If this isn’t the case, you can request the same diagram for any of the child operations. For example, let’s see what happens inside a checkoutdata.GetBundle call.

So, we see that in the GetBundle request, a significant part of the time is spent going to the DaData address suggestion service.

Looking closely at the code, we noticed that in this request we’re not interested in the entire address as a whole — just the region is enough. That is, this query can be efficiently cached. But before jumping into the fray and screwing up caches, let’s find out how much that would actually speed up the request.

Simulation optimisations

Let’s say we can cache a call to the address suggestion service, and the cache cunning will be around 95%. It is necessary to evaluate how much such a cache will speed up the analysed query as a whole. For simplicity, we will assume that in all traces we speed up the call to DaData by 95% (if we want to simulate the behavior of the cache more realistically, we need to speed up the call to DaData by 100% with a probability of 95%, with a 5% probability of not speeding it up at all) . We apply this synthetic optimisation to all traces of interest to us and get the following graph.

The query latency percentiles are plotted along the horizontal axis, and along the vertical axis, by what percent the corresponding percentile will accelerate when hypothetical optimisation is applied.

We see that the greatest influence of such optimisation is concentrated in the zone of lower percentiles (p75, p85). At the same time, p99 reacts relatively weakly — only by 5%. However, a 10% improvement in p50 looks very promising. Therefore, it is worth trying to add caching to this request. In reality, after we added such caching, the latency of this request on p50 only fell by about 6–7%.

Other examples

The example described above is just one of many where a Jaeger trace analysis library helped us.

These were some more successful optimisations:

• Found and neutralised an extra trip to a third-party service in the pricing module. The result was minus 6% latency on the p50 of the main screen.
• Found an extra trip to the ranking system on the shares screen. The result is minus 40% of the p99 latency of this screen.
• We found a trip to an overloaded base at checkout (the same one that we talked about when comparing different methods of searching for bottlenecks). The result is minus 12% of the p99 latency of this query.

Conclusion

As a result, by applying the full power of Spark to the traces unloaded from Jaeger, it was possible to create a convenient tool that lets us search not only for potential problems in large distributed systems, but also to give a detailed assessment of the potential of optimisations before their implementation. If you also use Spark and Jaeger in your work, you may also find it useful. You can find the open code here on GitHub. If you have any questions, please leave a comment below — I’ll be happy to answer!

Find bottlenecks in just 30 minutes using Jaeger traces was originally published in JoomTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, we’re going to discuss Churn Rate, as well as look into the following questions:

• What is Churn Rate?
• Why do you need this metric?
• How can you calculate it using SQL?
• How can you calculate it using Python?

Our goal is to provide you with answers as quickly and clearly as possible as to why this metric is needed and how to calculate it correctly using various software. This article also gives examples of code that you’re free to use to generate data and calculate Churn Rate on your own.

What is Churn Rate?

Churn Rate is a metric that allows you to answer the question: What proportion of users have stopped using a service, or in other words, have fallen off? A churned (or dropped) user is one who, since the day of their previous activity, has not logged into the service for some fixed period of time (10 days, 2 weeks, 1 month, etc.). Why is a time frame necessary? The answer is simple — we cannot wait forever to say for sure whether a particular user will return or not. For this reason, we calculate within a certain period, and:

• The larger the time frame, the more accurate the metric.
• The smaller the time frame, the sooner we will know the result.

Example: On December 1, 100 users logged into our app, and we are interested in what proportion of them will not log into the service again within 28 days. Let’s say there were 17 such users → Churn Rate = 17%. (in this case, we can calculate the metric only after 28 days from the starting date).

Calculation formula

Churn Rate = [number of dropped users] / [number of users who were active on the selected day]

Returning to the previous example, we have:

• Day X: December 1
• Time frame: 28 days
• Date X + Time frame: Current date + 28 days (until December 29)
• Users who were active on the selected day: 100 people who logged into the app on December 1
• Churned users: 17 people who logged into the app on December 1 but didn’t log in again for 28 days, i.e. until December 29

It’s important to understand that Churn Rate in most cases will not be equal to 1 — Retention Rate (the proportion of users who were active on date X and returned to the service after a certain period of time).

Why is it that Churn Rate ≠ 1 — Retention Rate? It’s because within a given period, users don’t fall off in one big group at the same time. Here is a Retention Rate chart for different periods on date X:

We see that some of users were active during the Churn time frame → if a user was inactive on day 28, this does not mean that they were also absent throughout the previous period, so Churn Rate ≠ 1 — Retention Rate (but if Retention Rate were cumulative, then the equality would hold).

Metric calculation in practice

Software:

• SQL — Google BigQuery
• Python — Jupyter Notebook
• BI — Data Studio

To calculate the metrics, we will use synthetic data. We generate it like this:

1. Import libraries

import names
import random
import datetime as dt
import pandas   as pd

2. Generate an array of values for the future dataframe (may take several minutes)

# this will store all the usernames we will use
# we will have no more than 10000 such names (they will not necessarily be unique)
names_list = []
for i in range(10000):
    names_list.append(names.get_full_name())
    
# values to be substituted randomly
platforms = ['ios', 'android']
columns   = ['event_date', 'platform', 'name', 'event_type']
events    = ['purchase', 'productPreview', 'addToCart', 'productClick', 'search', 'feedbackSent']

# form the correspondence of the name and platform of the user
# (for simplicity, we assume
# that each user only uses one of the two possible platforms)

names_platform_dict = {}
for name in names_list:
    names_platform_dict[name] = random.choice(platforms)
    
# form arrays of users according to the degree of uniqueness
regular_users     = names_list[:2000]
not_regular_users = names_list[2000:]

# rows for the future dataframe will be stored here
df_values = []

# main loop
# go through all dates from December 1, 2021 to May 29, 2023

for date_index in range(180):
    date = dt.date(2022, 12, 1) + dt.timedelta(days=date_index)
    
    # how many lines will correspond to one day
    rows_number  = random.randrange(2000, 3000)
    
    for name_index in range(rows_number):
        # here we set the weekly seasonality
        # on weekdays the proportion of regular users will be higher

        if (date_index + 2) % 7 == 0 or (date_index + 3) % 7 == 0:
            list_of_names = random.choices([regular_users, not_regular_users], weights = [4, 1])[0]
        else:
            list_of_names = random.choices([regular_users, not_regular_users], weights = [7, 1])[0]
        
        name       = random.choice(list_of_names)
        event_type = random.choice(events)
        platform   = names_platform_dict[name]
        
        # add a row to the array of all rows for the future dataframe
        df_values.append([date, platform, name, event_type])

3. Received dataframe

# create a dataframe
events            = pd.DataFrame(data=df_values, columns=columns)
events.event_date = pd.to_datetime(events.event_date)
events

In our case, the original dataframe looks like this:

You can also find basic information about the dataframe

4. Data upload

# In the source directory where the file with the code is located, you can find the csv file
events.to_csv('./example_events.csv')

Context

We work for a company that sells goods via a smartphone app. The app is available on two platforms: Android and iOS. The company considers a churn metric with a time frame of 28 days.

How to calculate Churn Rate using SQL

We upload the data to Google BigQuery (any environment can be used).

Let’s move on to calculating the metrics.

• Since it’s customary to calculate Churn Rate with a time frame of 28 days (in the described case), it would be incorrect to calculate it for those dates from the moment of which the 28th day has not yet come. For this reason, we will cut these lines at the last stage of the query.
• Since a user can have multiple events on the same day, it’s a good idea to leave only the unique matches for platform, user and date beforehand.
• For each user on the date in question, we find the date of their next event, then calculate the number of days before this date. The resulting number of days will be compared with the time frame of the metric. If a user has been absent for ≥ 28 days from the date in question, they are considered “churned”.

This results in the following code:

-- find unique matches for the name, platform and date of activity

-- we only care if a user was active on a particular day
-- it doesn't matter what actions they performed

WITH names_days AS (
    SELECT DISTINCT
        platform,
        event_date,
        name,
        -- with the window function, find the date of the next user activity
        LEAD(
            event_date
        ) OVER(PARTITION BY name ORDER BY event_date) AS next_event_date
    FROM example_events
),

main AS (
    SELECT
        *,
        -- find how many days later the next user event happened
        DATE_DIFF(next_event_date, event_date, DAY) AS days_till_next_event
    FROM names_days
)

SELECT
    event_date,
    -- find the share of users:
       -- who have been absent for at least 28 days from this day out of all users
       -- and who were active that day

     -- you can also add a slice by platform here

     -- `distinct` below is not needed in this case,
     -- but with a different format of the source data, it may be needed

    COUNT(DISTINCT
        IF(
            days_till_next_event < 28,
            NULL,
            name
        )
    ) / COUNT(DISTINCT name) AS churn_rate
FROM main
WHERE 1 = 1
    -- cut rows where 28 days have not passed since the current date 
      AND event_date <= (SELECT DATE_SUB(MAX(event_date), INTERVAL 28 DAY) FROM example_events)
GROUP BY 1
ORDER BY 1

If there are no window functions in the version of SQL you are using, you can:

• Access the source table sorted by name, event_date through a subquery and apply the LEAD / LAG function without a window (providing in the SELECT clause a condition for matching the name. Example: IF(LAG(name) = name, LAG(event_date), NULL)
• Number the rows in the source table sorted by name, event_date and join it to itself through table.index = table.index + 1 AND table.name = table.name

At the output, we get a table in which the Churn Rate indicator is calculated for each date with a time frame of 28 days:

The result of the received query is not very convenient to analyse if we’re looking at the result of the SQL query in a table format. Therefore, we visualise the result using a BI tool:

We may also be interested in how the metric behaves depending on the chosen platform. In order to display such a graph, you need to add the additional grouping field platform to the original SQL query (also, do not forget to specify it in SELECT)

In both cases, seasonality is observed: on weekends, the outflow of users is higher.

How to calculate Churn Rate using Python

Import libraries

import numpy             as np
import datetime          as dt
import pandas            as pd
import seaborn           as sns
import matplotlib.pyplot as plt
import matplotlib.ticker as mtick

Load data from the csv file

events = pd.read_csv('./example_events.csv', parse_dates = ['event_date'])

Let’s perform similar transformations using Python. To begin with, let’s leave only unique combinations of name, platform, event_date

events_grouped = events.groupby(['event_date', 'platform', 'name'])\
                       .agg({'event_type' : 'count'})\
                       .reset_index()

events_grouped.columns = ['event_date', 'platform', 'name', 'events_number']

Now for each user, we’ll find the date of their next event. We’ll also find how many days later their next event took place

events_grouped['next_event_date']      = events_grouped.groupby(['name', 'platform'])['event_date'].shift(-1)
events_grouped['days_till_next_event'] = (events_grouped['next_event_date'] - events_grouped['event_date']) / np.timedelta64(1, 'D')

For simplicity, let’s add a marker for whether we consider the user to be churned on the date in question

events_grouped['is_churned_int'] = events_grouped.days_till_next_event.apply(lambda x: 0 if x < 28 else 1)

After all the transformations, the data looks like this:

We now have everything we need to calculate the Churn Rate. Let’s find the maximum date from which 28 days have passed, then cut off unnecessary lines. After that, for each date, we calculate the number of unique users (DAU) and the number of those who are in the outflow. Then we calculate the share of those who are in the outflow of the total.

# we look for the maximum date for which the metric calculation can be considered correct
max_relevant_date = events_grouped.event_date.max() - dt.timedelta(days=28)

churn_rate = events_grouped.query('event_date <= @max_relevant_date')\
                           .groupby(['event_date'])\
                           .agg({'is_churned_int' : 'sum', 'name' : 'nunique'})

churn_rate.columns = ['churned', 'dau']

# calculate the metric
churn_rate['churn_rate']      = churn_rate['churned'] / churn_rate['dau']
# convert it to a percentage
churn_rate['churn_rate_perc'] = churn_rate['churn_rate'] * 100

We get the following table with our Churn Rate:

We visualise the results obtained using Python

palette = sns.color_palette(['#ff6063', '#32b6aa', '#ffcc53'])

sns.set_palette(palette)

plt.figure(figsize=(20,10))
plt.grid()

ax = sns.lineplot(data=churn_rate, y='churn_rate_perc', x='event_date')

ax.set_title ('Churn Rate', fontsize=15)
ax.set_xlabel('Date', fontsize=15)
ax.set_ylabel('Outflow, %', fontsize=15)

ax.set(ylim=(0, 50))
ax.yaxis.set_major_formatter(mtick.PercentFormatter())

plt.show()

If we want to calculate the metric separately for each platform, then that platform field must be added to the grouping fields, and also placed in the legend when plotting charts

churn_rate_platform = events_grouped.query('event_date <= @max_relevant_date')\
                                    .groupby(['event_date', 'platform'])\
                                    .agg({'is_churned_int' : 'sum', 'name' : 'nunique'})

churn_rate_platform.columns = ['churned_names', 'dau']

churn_rate_platform['churn_rate']      = churn_rate_platform['churned_names'] / churn_rate_platform['dau']
churn_rate_platform['churn_rate_perc'] = churn_rate_platform['churn_rate'] * 100

plt.figure(figsize=(20,10))
plt.grid()

ax = sns.lineplot(data=churn_rate_platform, y='churn_rate_perc', x='event_date', hue='platform')
ax.set(ylim=(0, 50))
ax.yaxis.set_major_formatter(mtick.PercentFormatter())

ax.set_title ('Churn Rate', fontsize=15)
ax.set_xlabel('Date', fontsize=15)
ax.set_ylabel('Outflow, %', fontsize=15)

plt.show()

Conclusion

We’ve analysed a traditional method of calculating Churn Rate. In some cases, it’s more efficient to calculate this metric in a different way — it all depends on the business context and the task being solved.

Usually we are not interested in specific events, like product clicks or search queries. We just need to know whether a user was active during the time frame. Hence, Churn Rate may be calculated using a wide range of events that may be attributed to user activity. However, there are situations where certain events should be disregarded. For example, if we’re talking about search engines, which are often set in the browser as the “home page”, taking into account something from the “user opened the page” category when calculating Churn would not be very useful, because this can hardly be considered as genuine user activity.

Possible questions

1. There are dates for which we still do not know whether the user logged in in the subsequent period or not (we’re talking about the metric calculation time frame). What happens if you don’t cut such lines?

Starting from your current (or last available) date, minus the time frame, the metric will skyrocket. For the most recent day, its value will hit 100% since we won’t know about the subsequent activities of these users — in the calculation they will be considered as part of the outflow. The graph in this case would look something like this:

2. But the metric doesn’t tell us who the service has truly lost forever (Actual Churn)!

That’s true, but what is “Actual Churn” anyway? We can never know for sure whether a user will log in to our service again (even if they’ve been absent from it for several years). The metric is needed in order to track a trend in dynamics, thus it will not tell us how many users we have lost forever.

3. We noticed an anomalous change in Churn Rate, what are the causes?

There can be infinitely many reasons for changes in Churn Rate, and for each area they will be different. Let’s try to cover the main ones:

1. Advertising

• budget reallocation;
• advertising campaigns have been adjusted;
• new ad channels were added.

2. Product changes

• prices have been changed;
• switched to a different type of product (application, website, widget, etc.);
• Certain features have been added or removed (the app now has the ability to make calls or now it is impossible to pay for goods with certain kinds of electronic wallets).

3. Other

• Competitors’ activity;
• Influence of substitute goods;
• Current events and news.

It’s more important to be able to answer the following questions:

1. On the basis of what audience is the metric calculated?
2. What could have caused the negative experience?
3. What can disturb the audience now?
4. How can I choose a time frame for calculating the metric?

The size of the time frame used depends on your specific business and expectations. We need to understand that using a larger time frame, we will wait longer, but we will get more accurate results.

Thank you for your attention!

We appreciate you reading till the end! We hope the article was useful.

Should you have any questions, feel free to ask them in the comments below — we’ll be happy to answer them. Also, if you think it’s worth considering other topics related to metric construction or business analytics, let us know. We’ll try to touch on them in a future post.

Churn Rate: What it is and how to calculate it was originally published in JoomTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Although, even if we forget about a traditional bunch of enterprise tweaks like analytics, backward compatibility support and A/B testing and focus purely on the code in direct relation to the implemented functionality, we may see that its complexity often grows out of control.

In this article, I’m going to talk about several features that my colleagues and I have implemented in Joom at different times, from the problem definition to the details of implementation, and show how easily the seemingly simple things turn into a tangle of endlessly complex logic, requiring many iterations of development.

Profile search

One of the big sections of the Joom app is an internal social network where customers can write reviews of products, like and discuss them, and subscribe to each other. And what kind of a social network would it be without profile search!

Of course, search as a feature is not such a seemingly easy thing. But I already had all the necessary knowledge, and we had a ready joom-mongo-connector component which could transfer data from a MongoDB collection to Elasticsearch index, adding additional data and doing some post-processing. The task sounded pretty simple.

Task. Implement an API for searching by social network profiles. No search filters — sorting by the number of followers will do for a start.

Okay, that sounds easy enough. We set up a transfer from the socialUsers collection to Elasticsearch by writing a YAML config. On the backend, we add a new API endpoint similar to the product search API, but without filters and sorting support for now (just query text and pagination). In the handler we make a simple request to Elasticsearch-cluster (the key is to get the right cluster!), then we take IDs of found documents — they are also user IDs — and convert them to the client JSON, hiding private information from prying eyes. That’s it — or is it?

The first problem we faced was transliteration. The user names were taken from social networks, where users from Eastern Europe (they were the majority at the time) often wrote them in Latin letters while their languages used Cyrillic. You may try to find “Юля”, but she’s “Julia” on Facebook, so she’s not in the search results. Similarly, you cannot find “Иван” by “Ivan”, though you’d really like to!

Here goes the first complication — during indexing, we would use Microsoft Translator API for transliteration and store two versions of first and last names. As an unpleasant side effect, the common indexing component became dependent on the transliterator client (and still is).

And the second problem, which is easy to anticipate if your native language is Russian, but it exists in other European languages as well, is the diminutive forms and abbreviations of names. If “Ivan” decides to call himself “Vanya” on Facebook, you won’t find him by querying “Ivan”, no matter how much you transliterate it.

So the next complication was that we found an index of diminutive names, added it to the code base as a hardcoded table (as easy as two thousand lines), and began to index not only the names and their transliterations, but also all the obtained diminutive forms (fun fact: in English they are officially called “hypocorisms”). We took every word in the username and looked it up in our humble table.

Then we put the word out to Joom’s regional managers and asked them to find us reference books of national name abbreviations in their countries of operation. If not academic, then at least whatever they could provide. It turned out that in some languages, in addition to the tradition of having a compound name (“Juan Carlos”, “Maria Aurora”), there are also abbreviations of two, three or even four words into one (“María de las Nieves” → “Marinieves”).

This new fact made it impossible for us to make a one-word lookup. Now we had to break a sequence of words into fragments of arbitrary length, and what’s more, different breaks can lead to different abbreviations! We didn’t want to delve into the depths of linguistics and write an AI that would abbreviate a Spanish name the way it’d be abbreviated by a real Spaniard, so we sketched out (I’m so sorry, Dr. Knuth) a combinatorial search.

And, as always happens with combinatorial search, it exploded on one of the users and we had to urgently add a limit on the maximum number of generated spelling variants. This further complicated the code (which was already surprisingly complex for this kind of task).

Machine translation of products

Task. Translate names and descriptions of products provided by the sellers in English to the user’s language.

You might have seen memes about the bizarre translation of Chinese product names. We’ve seen them too, but the desired time to market didn’t let us come up with anything better than using some existing API for translation.

It is easy to write an HTTP-client, create an account, and translate the product into the required language when it’s displayed to a user. But translations aren’t cheap, and it would be wasteful to translate the same popular product into French in each one of tens of thousands of views. Therefore, we’ve implemented caching: for each product we saved translations to the database, and when there were translations already available there, we didn’t use the translator.

But there was still room for improvement. We figured that a reasonable trade-off between translation quality and price would be to break the descriptions into sentences and cache them. After all, products often have the same patterned phrases, and constantly translating them is just wasteful. So we added another layer of abstraction to our translator component — a layer between the HTTP client and the cache (holding entire products in different languages) that breaks the text into fragments.

After release, the quality of translations was, of course, a great concern. We thought: what if we used a more expensive translation API? Would it work better with our specific texts? You can’t compare them with the naked eye, so we had to conduct an A/B test. This way we added a translation API name to our translation cache key in addition to the product ID and started requesting a translation with translation API choice depending on which A/B test group the user was in.

The expensive translator performed well, but it was still too wasteful to use it on all products. At some point, however, Joom was released in new countries where national languages were so poorly handled by our primary translator that we were ready to spend more for a successful launch; this way the logic of choosing translation API became even more complicated.

Then we decided that some of the stores on the platform are so good and the platform cares so much for their success that it’s OK to translate their products with the more expensive translator. So the logic for choosing a translator became dependent on the user, the country and the store ID as well.

Finally, we decided that our primary translator might have improved over the few years of Joom’s existence, and it could make sense to update the translation cache with some periodicity. But how to go without the A/B test? To that end, we got a “freshness” field in our cache, and things got even more complicated once again. As a result, our translation component got so incredibly complex, and this despite the fact we haven’t even enabled any homemade computational linguistics… Yet.

Converting clothing sizes

Perhaps one of the most painful problems when you buy clothes and shoes online is choosing the right size. When shipping from local warehouses, domestic businesses like Lamoda can simply deliver several sizes at once and just as easily take back what didn’t fit, but it doesn’t work that way with cross-borders. Parcels take a long time to deliver, the cost of each extra kilogram is high and their senders don’t expect a large flow of returning mail.

The problem gets further complicated by the fact that sellers from different countries may have completely different ideas of size. A Chinese “M” can easily turn out to be a European “XS”, and the horrendous-sounding “9XL” may not be all that different from an “XXL”. Experienced users rely on measurements, but even these are not always correct. For example, the user expects to see a measurement for chest girth, but the seller shows measurements of the garment itself — these differ by 5–10%. We don’t want the user to bother that much to shop at Joom!

Task. Instead of sizes provided by sellers, show users the sizes calculated by us using some uniform conversion table based on measurements.

Okay. We take the size table, which is parsed from the product description (there’s a dedicated rocket science of 5 KLOC component to handle this) and stored in a separate field, and substitute the sizes in it with the calculated ones. Then we hardcode the table to convert the girth to size (the one we simply find on the Internet) and rejoice.

But if there is no size table in a product or there are not enough rows in it, it doesn’t work. There goes the first reason for implicitly turning off the feature on a particular product.

Hmm, the size table is supposed to show sizes for real body measurements, but most sellers provide them by measuring clothes rather than mannequins. OK, let’s tweak them by a difference factor. Product manager Rodion, the lucky owner of the perfect size “M” shirt, goes to a nearby mall, tries on a bunch of different clothes, and comes up with factors — they’re similar, but vary significantly for different kinds of clothing. For a tight turtleneck the difference is almost 0%, but for a sweater it’s 10%. Also, outerwear of the same kind may vary in fit (“slim fit”, “normal fit” and “loose fit”), and that gives a spread of another ±5%. Now our difference factor (carved in the code as the Rodion coefficient) consists of two multipliers.

To define the fit, we make another parser that attempts to extract it from the product’s name or description. If an item does not fall into one of the categories checked by Rodion, the feature is implicitly turned off — for the second reason.

The final touch: in a large number of products, the chest girth is listed from armpit to armpit, that is, it’s only half the girth, which leads to ridiculously small sizes. We add the logic that if the girth is less than X, well, it just can’t be, it’s obviously half the girth, and we multiply it by two. We’re lucky that adults don’t usually differ from each other in size by more than 100%.

Now everything is so complicated that when testing a feature, it’s impossible to understand why it didn’t turn on or worked out one way rather than another just by looking at the product. We’re adding a large layer of logic to the code which logs detailed reasons why the conversion is turned off. To be able to fully trace the reason for it turning off on a particular product, we have to forward the error messages up the stack, adding details for each instance, several times. The code becomes dreadful.

And it all works in various ways depending on the A/B test group, of course.

Beware of G̶r̶e̶e̶k̶s̶ ̶b̶e̶a̶r̶i̶n̶g̶ ̶g̶i̶f̶t̶s developers optimistically estimating deadlines. Estimating development time is very difficult, no matter how simple the task sounds, and surprises await at every turn!

The unexpected complexity of simple software was originally published in JoomTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

However, everything’s changed after working on the bottom sheet in the Joom app, where it‘s used everywhere. Even in such critical scenarios as payment flow, so we’re truly confident in this component that I shared our experience on Podlodka iOS crew #7. As part of the workshop, I showed how to make a bottom sheet adapt to the content size, dismiss interactively, and support UINavigationController.

Wait, Apple has already presented a native way to present the bottom sheet. Why write your own? Though it’s true, this component is only supported from iOS 15, which implies that it will be available only in 2–3 years. Besides, designer requirements often change and go beyond native iOS elements, so you end up with your own implementation anyway.

In this article, I want to clear the air over Bottom Sheet, answer the questions I’ve faced myself, and suggest one of the possible implementations. So after you’re through with this article, you could deservedly add the “proficient with Bottom Sheets” line to your CV.

If you’re interested, let’s get started! We’ll create a simple bottom sheet and upgrade it step by step. Along the way, you’ll:

1. Learn how to adapt the bottom sheet to the content size and how to dismiss it.
2. Add interactive dismissal, taking scrollable content into account.
3. Support UINavigationController with navigation inside the bottom sheet.

Part 1. Adapting to content size. Bottom Sheet dismissing. Basic design

First of all, let’s make sure we’re on the same page with the term “Bottom Sheet”. Bottom Sheet is a component that sits at the bottom and adapts to the size of the content. Here are some examples of its use in some system applications: Apple Maps (search), Stocks (news), Voice Memos (voice recording), etc.

The starter project

Download the starter project via the Github link. Once downloaded open BottomSheetDemo.xcodeproj and look around. You’ll see two targets in the project: BottomSheetDemo and BottomSheet — an application and a library with the bottom sheet.

RootViewController is the first screen in the application. It has only one “Show Bottom Sheet” button which presents ResizeViewController on tap.

ResizeViewController takes the height of the content in the initialiser. There are also four buttons that change the height of the content: by +100 and -100, by 2 and 0.5 times.

Let’s see it in action.

Theory. How do I present a Bottom Sheet?

We need an entity to control the presentation, which will add a bottom sheet to the UI hierarchy, position it on the screen, account for the size of the content, respond to changes in it, take care of animation, and enable the interactive closing.

This sounds like a UIPresentationController’s responsibility:

From the time a view controller is presented until the time it is dismissed, UIKit uses a presentation controller to manage the presentation process for that view controller.

In order to use it, we need to override modalPresentationStyle and transfer the presentation controller through transitioningDelegate.

Armed with this knowledge, let’s start making the bottom sheet!

Let’s create a presentation controller

To show a bottom sheet, let’s override modalPresentationStyle and transitioningDelegate. Don’t forget that transitioningDelegate is a weak reference, and we need a strong one for not losing the object.

Create BottomSheetTransitioningDelegate, an implementation of transitioningDelegate.

And presentation controller.

Finally, go back to RootViewController and resolve TODO.

Let’s run the application.

Looks like it just got worse. The view controller’s opened up in full screen and hidden behind the status bar. We’ve overridden the system presentation controller that showed the view controller pretty well and positioned it according to Safe Area. There is nothing like this in our presentation controller — we haven’t specified the view controller’s position in any way, not to mention Safe Area, so let’s fix that.

Taking content size into account

Let’s go back to ResizeViewController. The currentHeight property is self explanatory and accounts for the current height. To avoid creating excessive protocols, we’ll use preferredContentSize to represent the Bottom Sheet’s desired size.

Now let’s override frameOfPresentedViewInContainerView, which accounts for the position of presentedView in the presentation controller. In our case, presentedView is a ResizeViewController’s view and containerView holds it.

Additionally, we set shouldPresentInFullscreen to false, because Bottom Sheet does not cover the whole screen.

Now, let’s see the result

The original size is now considered, but there is no reaction to its change.

Reacting to content change

Let’s see UIPresentationController. It implements UIContentContainer, and thus preferredContentSizeDidChange(forChildContentContainer:), which is invoked after preferredContentSize is changed in one of the child view controllers.

Here we check the current frame and the one we believe to be correct. If they are different, we refresh the presentedView frame. Let’s run the application.

The size changes abruptly and without animation. Why? Because we didn’t include this animation in any way. Let’s add an animation on preferredContentSize changes in ResizeViewController.

Now, check it again.

It works! But here’s another problem — we can’t close it.

Bottom Sheet dismissal

To dismiss it, we’ll need a shadow which’ll close the bottom sheet on tap. Also, we’ll need a dismissal handler by which the presentation controller will report that it’s ready to be closed.

Let’s inject it into the presentation controller initialiser.

For convenience, let’s add a presentation controller factory.

It will be used within BottomSheetTransitioningDelegate,

We’ll implement the factory with the dismiss handler in RootViewController.

Now let’s configure a shadow with dismiss handler in the presentation controller. We should add the shadow before the transition starts and remove it when it’s over.

To begin, we need to track the presentation controller’s state. Let’s introduce a state property, which will be in charge of the Bottom Sheet’s current state. Let’s override the transition lifecycle methods and change state accordingly.

Then the question arises as to when we should add and remove the shadow. We’ll add the shadow before presenting the bottom sheet, and remove it right after it’s hidden

Finally, we can implement addSubviews() and removeSubviews()

Now, let’s see how it works.

Great! Now we got the shadow, and the bottom sheet’s dismissed by tap on it! But the shadow appears and disappears without any animation.

Animated transition

What should we do? The shadow relates to the transition, so it should be animated along. Therefore, we need to inline it into the transitioning delegate.

Let’s implement the UIViewControllerAnimatedTransitioning protocol the same way iOS already does it, but we’ll also include a fade-animation for the shadow.

Also, don‘t forget to implement the relevant methods in the BottomSheetTransitioningDelegate.

Let’s make sure that the animation works.

And it does! To finish our bottom sheet, we need to round the top corners.

Rounding the corners

We can make it via presentedViewController’s cornerRadius in the presentation controller. It needs to be done before the transition starts in presentationTransitionWillBegin().

Finally, let’s have a look at the corners.

What we’ve accomplished so far:

1. We overridden the system transitioning delegate.
2. We created a presentation controller.
3. We added a shadow to hide the bottom sheet via the dismiss handler.
4. We implemented an animated transition via the transitioning delegate.
5. We made the basic design.

Part 2. Interactive Bottom Sheet dismissal

Like in the first part, let’s begin with the starter project. A pull bar has been added to indicate that the bottom sheet can be closed by the swipe-down gesture. Also, a scrollView has appeared full screen in ResizeViewController. We will need it for list-based screens. The rest is from part one.

Let’s see the application.

Theory. Specifics of Interactive dismissal

UISwipeGestureRecognizer is an obvious choice for detecting swipe gestures. It will initiate the bottom sheet dismissal.

But what if the presented controller already has this gesture? It may cause a conflict of gestures since it’s not clear which one to process first.

Is it common for the presented controller to have this gesture? In fact, all the time. In modern applications, 99 % of screens are list-based, which means that each one has a UIScrollView or its descendants, UITableView or UICollectionView, which have exactly the same gesture. What should we do?

Let’s break it down into two cases, without and with UIScrollView.

1. If there’s no UIScrollView, we simply add a swipe gesture.
2. If it’s available, the content may fit:

• Completely. Then the bottom sheet’s size will be less than the screen, so it can be closed with a swipe right away.
• Partially. Then a swipe may also mean content scrolling. Let’s assume that the user scrolls to close the bottom sheet when contentOffset is zero. Otherwise, their intention is content-scrolling.

If UIScrollView exists, we subscribe to contentOffset changes and judge by them when interactive dismissing can be started.

Now that we have the plan, let’s implement it!

If there is no UIScrollView

Then we add the pan gesture to presentedView. At what point should we do this? The gesture initiates interactive dismissing and the bottom sheet can be closed only if it’s fully appeared. So it makes sense to add the gesture at the end of the presentation.

And write the function that adds the pan gesture to the given view.

Let’s break down each state of the gesture.

began — the user has just started sliding the finger, and the gesture has been recognised as the pan gesture. We initiate the bottom sheet dismissing.

changed — the user continuously slides the finger across the screen. We hide the bottom sheet proportionally to the distance the user’s finger has travelled across the screen.

ended — the user hass taken fingers off the screen. We decide if we dismiss the bottom sheet or return it to its original position.

cancelled — the gesture is cancelled. We return the bottom sheet to its original position.

Additionally, we will use UIPercentDrivenInteractiveTransition to pass the transition state to the transitioning delegate.

Let’s start with the began state. This is the right moment to initialize interactive dismissing because this state occurs only once. We also call dismiss from presentingViewController to notify UIKit of the intention to close the bottom sheet.

Now let’s proceed with the changed state. Change the position of presentedView proportionally to the distance travelled by the user’s fingertip across the screen. We measure the distance from the starting point, where the gesture began, to the current position. Next, we calculate transition progress relative to the height of the content view controller, i.e., presentedView.

Once the user takes their finger off the screen, the gesture switches to the ended state. We need to figure out what the user wants — to scroll the content or to dismiss the bottom sheet. If the user moved their finger abruptly, travelling a very short distance, they most likely wanted to dismiss the bottom Sheet. Another matter is if the user’s finger travelled a long way across the screen and, at the very last moment, sharply took their finger off the screen, aiming upwards. In this case, the bottom sheet should return to its original position. This suggests the idea of calculating some kind of movement momentum that would account for both acceleration and direction.

A bit of physics. Imagine that there is a body that moves with a constant velocity v₀. Then, at a distance x₀, it comes under the influence of a deceleration a. The question is, where will the body stop?

The velocity formula is v(t)=v₀﹢a﹡t

With negative acceleration the velocity becomes zero, let’s denote this moment in time as t₁ and put it in the velocity formula:

Next let’s put t₁ in the distance formula x(t) = x + v₀ ﹡ t + a ﹡ t²/2

Then, we finally get the formula for the distance x₀−0.5 ﹡ v₀²/a, where x₀ is the body’s initial position, v₀ is its initial velocity and a is the deceleration.

Let’s assume that Bottom Sheet is the body from the problem above when the gesture is over. Then, the current speed and the travelled distance can be found through pan gesture. Let’s assume the deceleration to be a constant of 800. As we know the distance formula is x₀−0.5 ﹡ v₀²/a, so we can calculate where the bottom sheet will stop affected by deceleration. If the stopping point is closer to the starting point, we cancel the transition, otherwise we carry it out to the end.

If the gesture is cancelled, we return to the starting point.

Finally, we should return interactiveTransitioning to the transitioning delegate.

Now let’s run the application and see what happens.

Woohoo, we’ve taught our bottom sheet to close on a swipe down! However, if the content is larger than the screen, scrollView will intercept the gestures.

The list-based screens break down

Even though ResizeViewController was already a list-based type, it didn’t prevent us from adding a pan gesture. It’s because scrollView has no scroll when contentSize is equal to its size.

Therefore, let’s consider the opposite case when contentSize is larger than the bottom sheet, and scrolling works. First of all, we should subscribe to contentOffset. Then, if contentOffset is zero and the user is scrolling down, then we initiate dismissing. When the user releases their finger from the screen, we’ll either cancel or finalise the transition, just like we did before. If contentOffset is changed and the user does not touch the screen, then scrolling goes on by inertia and we shouldn’t do anything.

Firs of all, we need an indicator which would tell us whether a view controller has a scrollView. Let’s introduce a protocol for this.

To track contentOffset changes we subscribe to UIScrollViewDelegate. But what if someone has already subscribed to UIScrollView delegate? Then we’ll overwrite the previous delegate.

So we will use MulticastDelegate to forward UIScrollView’s delegate invocations to all interested parties and not just one. In a classic iOS delegate pattern, only one object at a time can subscribe, and it’s not really convenient. That’s why MulticastDelegate is introduced. By using it, firstly, we ensure that the delegate property is not overwritten. Secondly, we proxy all invocations to subscribers, and such an approach doesn’t require any changes in the existing codebase. It’s implemented in Objective-C because of its runtime capabilities and ability to use message dispatching. Though we could use Swift, it would be much more verbose and less universal — for each type with delegate new instance should be implemented with all the methods.

We subscribe to delegate after the end of the transition, just like we did with the pan gesture.

Then we should implement UIScrollViewDelegate. In the first place, let’s make some helpers

Next, let’s consider the moment when the user is about to scroll the content. At this moment, the user has just started a swipe gesture. Remember this state with the isDragging flag.

Next, let’s make the helper function, which will help us to determine whether transition progress needs to be started or carried on.

We check that the user is now sliding the finger across the screen. If so, we check if the transition should be carried on.

If the transition hasn’t started or has been just initiated, it should be continued if the user scrolls down and the content is at the top (check it via isContentOriginInBounds). If we’re in the middle of the dismissal, we just continue the transition.

Then let’s see when contentOffset is changed in scrollViewDidScroll(:_).

startInteractiveTransitionIfNeeded() will initiate the interactive transition if we haven’t already done so.

In scrollViewDidScroll(_:) we’re checking whether we can continue/start the interactive transition. If we can start or continue, we initiate the transition. If we can’t, we should remember the last content offset before the transition was activated. We’ll need it later.

💡 Remember how you were asked during interviews when a View’s bounds origin is not zero? And you probably replied: When the content offset of scrollView is non-zero. But it is not clear how this could be used in practice, is it? Below you will see how having this knowledge may be handy!

Now let’s make sure that content is tied to the top and equate contentInset to contentOffset. We change contentOffset via bounds.origin to avoid scrollViewDidScroll(_:) invocation. In the end, we update the transition progress.

Once the user takes the finger off the screen after scrolling, we should finalise or cancel the transition, which is the same as we did when the pan gesture is in the ended state.

Via didStartDragging, we check if the bottom sheet interactive dismissal was active before the end of scrolling.

If yes, then, just like with pan gesture, we use the impulse to decide if we cancel or finalise the transition.

If not, we cancel the transition. It’s also possible that the user started dismissing the bottom sheet and then returned to content scrolling. In this case, the transition’s progress is zero, and we definitely want to cancel it.

Now let’s see what we’ve got.

So, we’ve learned to work with all Bottom Sheet sizes.

What we’ve achieved in the second part

1. We’ve made truly interactive transition using pan gesture and scrollView.
2. We’ve implemented multi-subscriptions pattern to any delegate via MulticastDelegate.

Part 3. Maintaining UINavigationController

F inal starter project is already waiting for you. Two new buttons have been added to ResizeViewController, which are visible if a navigation controller is present. The first one pushes ResizeViewController with the current content height, and the second one pops to rootViewController. The rest is from part two.

In part three, our goal is to support the navigation controller inside the bottom sheet with standard push and pop operations, preserving interactive pop.

Theory. Will it work out of the box?

Can I use the system UINavigationController directly? Unfortunately, no.

The navigation controller does not fully support preferredContentSize. The original content size and its increase work as expected. But the navigation controller does not react to its downsizing. When tapping -100, the size doesn’t change.

So we will definitely need a subclass of UINavigationController, capable of tracking changes in the navigation stack and updating its own preferredContentSize according to topViewController.

When tracking scrollView in the presentation controller, we need to take into account that presentedViewController can be UINavigationController. Additionally, when changing the navigation stack, we need to extract scrollView from the current topViewController, if there is one.

And the final note. UINavigationController is packed within an iOS SDK version with its own features. As we will see later, these features will come to light and become a nudge. We will discuss what we can do about it later.

Adapting to content size

We’ve already made content size adaptation in the first part, but it doesn’t work with a navigation controller. The system navigation controller doesn’t fully respect preferredContentSize changes. So let’s create a descendant of UINavigationController and use the UIContentContainer feature.

In updatePreferredContentSize, we account for topViewController and additionalSafeAreaInsets.

Similar to the presentation controller, we react to content size changes via preferredContentSizeDidChange(forChildContentContainer:). Remember that we should take care of the animation ourselves when changing preferredContentSize. So let’s add animation to the navigation controller and remove it from ResizeViewController.

In the presentation controller, we should bear in mind that the presentedViewController can be a UINavigationController. In this case, we need to track scrollView inside the current topViewController. Let’s update setupScrollTrackingIfNeeded() accordingly.

To track changes in the navigation stack we subscribe to delegate via the same MulticastDelegate pattern we did before. We’ll keep an eye on scrollView when presenting the view controller.

Let’s check the result.

The navigation controller is now responsive to changes in content size, but when switching back, the size is not involved in the animation.

Animating the transition push and pop

Why does it happen? Because the system implementation of the navigation controller fails to account for preferredContentSize when changing the navigation stack. Therefore, we need to update content size every time changes are made to the navigation stack.

With this in mind, let’s introduce a helper function for updating the stack and preferredContentSize together. Where possible, we resize content with an animation via transitionCoordinator. It is vital to update the stack first and only then the content size. Otherwise, topViewController will not be up to date.

Finally, let’s implement UINavigationController’s methods that change the navigation stack via updateNavigationStack(animated:applyChanges:).

Now let’s run the application and see what we’ve got.

It got better and the content size is now considered, though with some artifacts. Let’s have a look at iOS 12.

Transition quality has worsened, and the content size is the same as the previous one after the pop.

It turns out that we can’t fully count on the system transition of UINavigationController, and we also have to implement it by ourselves. Let’s implement UINavigationControllerDelegate, in which we redefine the push and pop transitions.

Next, let’s implement UINavigationControllerDelegate and define the transition’s animation.

Let’s run the application and pay attention to the animations.

Everything works smoothly, as we expected!

Interactive pop

When we overrode push and pop transitions, we lost the system interactive pop transition, which had previously functioned out of the box. So we’ll have to implement it by ourselves.

But how do we replicate the system interactive pop transition? Let’s see what we have in UIKit… UIScreenEdgePanGestureRecognizer! It’s used by iOS for exactly this kind of gesture.

We’ll add this gesture for the view controller, which has been pushed into the navigation controller’s hierarchy. Below, I’m giving only critical segments of the code.

In the handler, we do the same as for the pan gesture. Then we initiate the interactive transition, updating progress proportionally to finger movement, and use the same criterion for cancelling and finalising the transition.

What remains is to account for the interactive transition in UINavigationControllerDelegate. We configure an interactive pop gesture on push operation.

Let’s check it now.

And yay, we’ve made it through our last challenge!

Let’s summarise what we’ve done in the last part:

1. We’ve adapted UINavigationController to content size.
2. We’ve achieved the right push and pop transitions in iOS 12+.
3. We’ve kept a unique interactive pop.

Conclusion

Thank you for reading to the end! Now you can update your resume. We started from scratch and ended up with a multifunctional bottom sheet. We answered the questions we asked at the beginning of the story, and even recalled school physics!

I hope you found this helpful and now the bottom sheet in your app will play out in fresh colours!

Bottom Sheet, shall we drop the formalities? was originally published in JoomTech on Medium, where people are continuing the conversation by highlighting and responding to this story.