Recently, I redesigned and reimplemented application monitoring for a couple of production services. I am happy about result, so here’s my key points:

#0: Collect enough data

Observe your system from in- and outside

Application can’t fully monitor itself by design. For example, web application router didn’t expect a special symbol and silently crashed without incrementing error counter. Unless you have something that can observe server from the outside (reverse proxy, load balancer, client-side), you won’t notice a problem. On the other hand, you can’t observe system only from the outside, as some details are inherently hidden inside.

Gather most important sensors in one place

Chances are high that in case of incident you’ll get an alert, which says little to nothing about cause, hence won’t help you to figure out how to stabilize your system. That’s why it‘s important to gather significant sensors in one place, where they can be easily viewed.

Take your time, think about possible problems, review previous incidents and assemble corresponding sensors. Don’t forget to revisit it as your system and knowledge about it grows.

#1: Send notifications when things are bad and immediate action is required

Increase alert’s reactivity

On incident, alert should be send ASAP — faster reaction means shorter incident’s duration. This heavily depends on how alert’s expression looks like, e.g. calculating average slows down the reaction, as well as calculating metric over a big window.

Percentiles over relatively small windows should be your go-to choice.

Alert should represent breached SLO/User experience

Don’t get alerted about some hidden potential precursors, like slight GC timing increase. GC itself is not harmful, unless your service SLO was breached because of it.

If your SLO alert is reactive and GC degraded enough to affect users — you’ll get notified anyway. But when users left unaffected, issue can wait for a while.

#2: Don’t send notifications when immediate action is not required

Over-information fallacy

Sometimes people decide they want to get notified about every single thing. Eventually you’ll start ignoring them and will miss an important one, that’s how brain works. Don’t fall for this.

Redesign false-negative alerts

Alert which bleeps without a serious reason is the worst. It distracts engineers and make them reluctant in reacting to alerts. This happened to me once and it was the dumbest incident of my life.

Usually alert is flaky, when at least one of these is true:

• Alert triggers on a single threshold violation
• Metric is first evaluated, then aggregated
• During off peak measurable value shifts towards the tail

Evaluating data over a window (which gets bigger off peak) or waiting for a few consecutive results should solve a problem.

Keep in mind it’s completely opposite to reactivity: bigger window means less reactivity, smaller window means more false-negative alerts. Hitting balance will take time, but it’s worth it.

Consider alert scope granularity

If you have a lot of stateless instances being deployed, like 3+ per availability zone or so, you don’t need to be notified when one of them become faulty. System itself should tolerate minor errors. Doing proper capacity planning, setting adequate timeouts, retries, and circuit breakers should make it.

Alerts configured per-instance are subjects for thorough review.

Separate alert channels

Although I criticized extensive alerting, it’s crucial to know what’s going on with your system (I can’t stress it enough!), you just don’t need to know it straightaway — use asynchronous channel.

#3: Ability to observe trends and take necessary actions in advance

Long distance trends such as cpu consumption, connection pool size, and amount of stored data at database doesn’t require immediate actions and can be examined in following ways:

1. Manual labour: revisit all dashboards on a daily/weekly schedule. Doesn’t scale well with increasing amount of sensors and applications
2. Semi-automatic: clone corresponding alert and set lower threshold. Then attach it to separate channel, where they will be deduplicated, aggregated into single report and sent daily/weekly

When I incorporated those, it turned out being an on-call engineer doesn’t have to be a nightmare :)

Thanks for reading, hope you learned something useful!

Application Monitoring I Wish I Had Before was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Definition

Let me coin a phrase for brevity:

Homogeneous API — API with equal computational cost of all requests.

What’s computational cost? It’s all resources, which are consumed to produce outcome, e.g. cpu time, memory, disk space, and network bandwidth. Let’s consider a service, which stores user-specific information, e.g. Medium. It has a reading list feature, which lets you save stories for later. If we design a backend to serve requests in a form of “give me all stories for user A”, the computational cost will vary from user to user, making API heterogeneous.

Now let’s take a look at the pros of making API homogeneous.

Advantages

State-of-the-art observability

Deviation of heterogeneous API timings depends on how users behave, how much data they save and consequently request. Thus, if given percentile grow, it’s hard to distinguish, whether the users distribution changed or the service itself slowed down.

Conversely, users can’t mess up homogeneous system, therefore there is a higher correlation between service metrics and code pushes.

SLOs are easier to enforce

Since metrics fairly represent a service’s health, it reduces false-positive alerts, hence bringing less burden to operations team.

Happy users

With homogeneous API you can be sure that no user struggles consistently with your service. For example you can monitor a p999 trend to prevent SLA violations, but a “heavy user” would fall in p9999. Their experience might degrade over time and you won’t notice the problem unless they complain explicitly.

So, no more “heavy user” tickets, which are harder to investigate and impossible to prevent. Amazing, right?

Fault-tolerance configuration

To provide sane experience for everyone, you’re obliged to set timeouts and deadlines as big as the heaviest request takes. Although it will allow everyone to be served, lightweight requests might be stuck for an unnecessarily long time. If a request usually completes in under 100ms and timeout is 1000ms, the client will waste an extra 900ms on a network bleep. Lesser timeout would’ve let the request be retried and succeeded earlier. Having all requests of similar sizes solves this issue, allowing us to apply those patterns effectively.

How to implement

Making API homogeneous requires two steps:

Pagination

Instead of returning all, return N entities at most. Sadly, necessity to change clients is not the only cost which comes with pagination.

Another one is consistent editing problem. It’s likely to occur with explicit pagination, when a user can see page numbers and navigate over them. The simplest approach to implement pagination is to calculate static indexes via “skip” and “limit”, thus page number P of size S would have indexes (math notation):

(S*(P — 1), S*P]

Now, imagine a situation, when a client deletes an entity from one page and goes forward:

Since the user deleted entity #3, the entities from page #2 shift to the left, effectively losing entity #4. To observe it user has to go backwards or reload a page after deletion. Instead of fetching pages by statically calculated indexes, retrieve S entities after N, where N is an identifier of last entity at current page. When jumping more than one page away, calculating indexes is fine.

Choose page size wisely, setting it to 1000 if median objects count is 25 won’t make an API homogeneous. Examine clients to figure out how many objects can be observed on the screen at the same time. Generally, multiplying this number by 2 to 5 gives a reasonable page size.

Getting back to the Medium “reading list”: if you take a closer look at it with network inspector, you’ll see an implicit pagination (pages are being requested as you scroll down) with page size = 10. So, their “reading list” API is homogeneous (others probably too, but I didn’t collect any evidence)

Isolation

It’s a paradigm where every service serves one and only one purpose. For example, if you have a service with 2 endpoints, restricting their thread pools won’t give perfect isolation. Today, 60/40 ratio represents user distribution, but tomorrow’s workload might change and one of the functions would starve. Plus, it leads to unnecessary underutilization.

Isolation is easier to achieve if you have an application template and matured CI/CD/deploy infrastructure.

I am aware of specific domains, where making api homogeneous is impossible or insignificant. To name a few — query engines and math evaluation programs. Nonetheless, I believe it’s useful and feasible for most web scenarios.

Resources

• https://landing.google.com/sre/sre-book/chapters/load-balancing-datacenter/, section “Varying query costs”
• https://landing.google.com/sre/sre-book/chapters/service-level-objectives/
• https://itnext.io/5-patterns-to-make-your-microservice-fault-tolerant-f3a1c73547b3

How APIs Can Benefit From Computational Cost Equality was originally published in codeburst on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article I’ll cover fault tolerance in microservices and how to achieve it. If you look it up on wikipedia, you will find following definition:

Fault tolerance is the property that enables a system to continue operating properly in the event of the failure of some of its components.

For us, a component means anything: microservice, database(DB), load balancer(LB), you name it. I won’t cover DB/LB fault-tolerance mechanisms, because they are vendor-specific and enabling them ends up setting some property or changing deploy policy.

As a software engineers, applications is where we have all the power and responsibility, so let’s take care of it. Here’s the list of patterns, I am going to cover:

• Timeouts
• Retries
• Circuit Breaker
• Deadlines
• Rate limiters

Some of the patterns are widely known, you might even doubt they worth mentioning, but stick to the article — I’ll cover basic forms briefly, then discuss their flaws and how to overcome them.

Timeouts

Timeout is a specified period of time which is allowed to wait for some event to occur. There is a problem if you are using SO_TIMEOUT (also known as socket timeout or read timeout) — it represents timeout between any two consecutive data packets, not for the whole response, so it’s harder to enforce SLA, especially when response payload is big. What you usually want is timeout, which covers whole interaction from establishing connection to the very last byte of the response. SLA usually described in such timeouts, because they are humane and natural to us. Sadly, they doesn’t fit SO_TIMEOUT philosophy. To overcome it in JVM world you can use JDK11 or OkHttp client. Go has a mechanisms in std library too.

If you want to dig in — check my previous article.

Retries

If your request failed — wait a bit and try again. That’s basically it, retrying makes sense, because network might degrade for a moment or GC hit that particular instance your request came to. Now, imagine having chain of microservices like that:

What happens if we set number of total attempts to 3 at every service and service D suddenly starts serving 100% of errors? It will lead to a retry storm — a situation when every service in chain starts retrying their requests, therefore drastically amplifying total load, so B will face 3x load, C — 9x and D — 27x! Redundancy is one of the key principles in achieving high-availability, but I doubt you would have enough free capacity on clusters C and D in that case. Setting total tries to 2 doesn’t help much either, plus it makes user experience worse on small blips.

Solution:

• Distinguish retryable errors from non-retryable. It’s pointless to retry request, when user doesn’t have permissions or payload doesn’t structured properly. Contrary, retrying request timeouts or 5xx is good.
• Adopt error budgeting — technique, when you stop making retries if rate of retryable errors exceeds threshold, e.g. if 20% of interactions with service D results in error, stop retrying it and try to degrade gracefully. Amount of errors might be tracked with rolling window over N last seconds.

Circuit Breaker

Circuit breaker can be explained as a stricter version of error budgeting — when errors rate is too high, function won’t be executed at all and will return fallback result, if provided. Very small portion of requests should be executed anyway in order to understand if 3rd party recovered or not. What we want is to give 3rd party a chance to recover without any manual work done.

You might argue, that it doesn’t make sense to enable circuit breaker if function is on critical path, but bear in mind, that this short and controlled ‘outage’ is likely to prevent a big and uncontrollable one.

Although circuit breaker and error budgeting share similar ideas, it makes sense to configure both of them. Since error budgeting is less disruptive its threshold must be smaller.

Hystrix was a go-to circuit breaker implementation in JVM for a long time. As of now, it entered maintenance mode, advising to use resilience4j instead.

Deadlines/distributed timeouts

We’ve discussed timeouts in the first part of this article, now let’s see how we can make them ‘distributed’. First, revisit same chain of services calling each other:

Service A willing to wait at most 400ms and request requires some work to be done from all 3 downstream services. Assume that service B took 400 ms and now ready to call service C. Is that reasonable at all? No! Service A timeout’ed and doesn’t wait for the result anymore. Proceeding further will only waste resources and increase susceptibility to retry storms.

To implement it, we must add extra metadata to a request, that will help to understand, when it’s reasonable to interrupt processing. Ideally, this should be supported by all participants and being passed throughout the system.

On practice this metadata is one of the following:

• Timestamp: pass point of time at which your service will stop waiting for response. First, gateway/frontend service sets deadline to ‘current timestamp + timeout’. Next, any downstream service should check if current timestamp ≥ deadline. If the answer is yes, then it’s safe to shut it down, otherwise — start processing. Unfortunately, there is a problem with clock skew, when machines can have different clock time. If it occurs, requests will stuck or/and get rejected immediately, causing outage to happen.
• Timeout: pass the amount of time service allowed to wait for. This is a bit trickier to implement. Same as before you set deadline as soon as possible. Next, any downstream service should calculate how much time does it spend, subtract it from the inbound timeout and pass to the next participant. It’s crucial not to forget time spent waiting in the queue! So, if service A is allowed to wait 400ms and service B spent 150ms, it must append 250ms deadline timeout, when calling service C. Although it doesn’t count time spent on the wire, deadline can only be triggered later, not earlier, thus, potentially consuming slightly more resources, but not spoiling the outcome. Deadlines are implemented this way in GRPC.

Last thing to discuss is — does it ever makes sense not to interrupt call chain, when deadline is exceeded? The answer is yes, if your service have plenty of free capacity and completing request will make it hotter (cache/JIT), it’s okay to keep processing.

Rate limiters

Previously discussed patterns mostly solves problem of cascading failures — situation when dependent service collapse after its dependency collapsed, eventually leading to full shutdown. Now, let’s cover situation, when your service is overloaded. There are plenty of tech and domain-specific reasons why it might happen, just assume it happened.

Every application has its unknown capacity. This value is dynamic and depends on multiple variables — such as recent code changes, model of CPU application running on right now, busyness of host machine, etc.

What happens when load surpass capacity? Usually, this vicious cycle occurs:

1. Response time grows, GC footprint increases
2. Clients get more timeouts, even more load arrives
3. goto 1, but more severe

This is an example of what might happen. Sure, if clients have error budgeting/circuit breaker, 2nd item might not create extra load, thereby give a chance to leave this cycle. Other things might happen instead — removing instance from LB’s upstream list might create more inequality in load and shut neighbor instances and so on.

Limiters to the rescue! Their idea is to shed incoming load gracefully. This is how excessive load should be handled ideally:

1. Limiter drops extra load above capacity, thus lets application serve requests in compliance with SLA
2. Excessive load redistributes to other instances/cluster auto-scales/cluster gets scaled by a human

There are 2 types of limiters — rate and concurrency, former restricts inbound RPS, latter restricts amount of requests being processed at any moment of time.

For the sake of simplicity, I’ll make an assumption that all requests to our services are nearly equal in computational cost and have same importance. Computational inequality arise from the fact, that different users can have different amount of data associated with them, e.g. favorite TV series or previous orders. Usually, embracing pagination helps to achieve computational equality of requests.

Rate limiter is more widely used, but doesn’t provide as strong guarantees as concurrency limit does, so if you wish to choose one, stick with concurrency limit and here’s why.

When configuring rate limiter, we think we enforce following:

This service can process N requests per second at any point of time.

But what we actually declare is this:

Assuming that response time won’t change, this service can process N requests per second at any point of time.

Why this remark is important? I’ll ‘prove’ it by intuition. For those willing to have math-based proof — check Little’s law. Assuming rate limit is 1000 RPS, response time is 1000ms and SLA is 1200ms, we easily serve exactly 1000 requests in a second under given SLA.

Now, response time grew by 50ms (dependency service started doing extra work). Every second from now on service will face more and more requests being processed at the same time, because arrival rate is bigger than service rate. Having unlimited amount of workers means you will run out of resources and collapse, especially in environments, where workers map 1:1 to OS threads. How concurrency limit with 1000 workers would handle it? It will reliably serve 1000/1.05 = ~950 RPS without SLA violation and drop the rest. Also, no reconfiguration needed to catch up!

We can update rate limit every time dependency has changed, but this is immensely big burden, potentially requiring whole ecosystem to be reconfigured on every change.

Depending on how limit value is being set, it’s either static or dynamic limiter.

Static

In this case limit is configured manually. Value can be assessed by regular performance tests. Although, it won’t be 100% accurate, it can be pessimized for safety. This type of limiting requires work to be done around CI/CD pipelines and has lower resources utilization. Static limiter can be implemented by restricting size of workers thread pool (concurrency only), by adding inbound filter which counts requests, by NGINX limiting functionality or by envoy sidecar proxy.

Dynamic

Here, limit depends on metric, which is re-calculated on regular basis. Chances are high, that there is a correlation for your service between being overloaded and growth in response time. If so, metric can be a statistics function over response times, e.g. percentile, medium or average. Remember computation equality property? This property is a key to more accurate calculations.

Then, define a predicate which will answer, whether metric is healthy. For example, having p99 ≥ 500ms considered unhealthy, thus limit should be decreased. How limit is increased and decreased should be decided by an apply feedback control algorithm, like AIMD (which is used in TCP protocol). Here’s pseudocode for it:

if healthy {
    limit = limit + increase;
} else {
    limit = limit * decreaseRatio; // 0 < decreaseRatio < 1.0
}

As you can see, limit grows slowly, probing if application is doing good, and decreases steeply if faulty behavior is found.

Netflix has pioneered idea of dynamic limits and open-sourced their solution, here’s repo. It has implementations of several feedback algorithms, static limiter implementation, GRPC integration and Java servlet integration.

Huh, that’s it! I hope you learnt something new and useful today. I’d like to note that this list is not exhaustive, you would also want to achieve good observability, cause something unexpected might happen and it’s better to understand what’s going on with your application at the moment. Nevertheless, implementing those will solve vast amount of your current or potential problems.

References

• Google SRE book, especially chapters Addressing Cascading Failures and Handling Overload
• Netflix article about dynamic limits

5 patterns to make your microservice fault-tolerant was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Every single click in modern internet triggers multitudes of network calls and, as you probably know, network is unreliable. This is one of the reasons you should set request timeouts when fetching something over the wire. It’s better to incorporate some other techniques as well, but that’s the topic for another article (and even whole book).

Why bother about timeouts?

Setting them is very important, because network might fail, your instance might get slow or crash. You don’t want users to wait indefinitely just because 1 of your 1000 servers suddenly shut down. People hate waiting, it affects their happiness and, therefore, your revenue. If network call gets stuck, you should abandon it, retry and, unless your cluster is having problems, it most likely to succeed.

Let’s see how we can implement this in Java and Go.

Common Java approach

Most popular http client in java is Apache HttpClient. Here’s how configuring request timeouts will look like:

Those 3 are all different and independent. Biggest problem we’re facing here is how to represent 500 ms as 3 different timeouts — should it be 100 + 100 + 300 or 50 + 50 + 400? And, more importantly, do we even care if connect timeout will take 200 ms? Imagine that server will complete request in 50 ms, so total response time would be 250 ms, in most situations — you don’t care, it’s completely fine!

On the other side you can’t set timeouts much bigger, because it will lead to a longer requests. Also, socket timeout is just a timeout between any consecutive packets being read from the socket, not the whole response being sent back to you.

Nonetheless, it’s all we can do using Apache HttpClient. Let’s take a look at Go.

Go standard library

Go, however, has support for whole client call timeout:

Here we’re setting call timeout to 2 seconds and calling httpbin, which will imitate work for 1 second long and return a response after. Launching this will lead to a successful call.

If we set call timeout to 1 second it will fail with a message similar to given, which is exactly what we are looking for!

2019/11/25 19:06:09 Get http://httpbin.org/delay/1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

On the contrary, configuring call timeout only is not always the best we can achieve. Imagine having a long and heavy request which takes 10 seconds to complete. It would be nasty to wait 10 seconds for response and figure out that servers was trying to establish connection all this time and no actual work was done.

Solution to this? — Connect timeout. Yes, the one we were sort of blaming earlier. Combined with call timeout it gives robust protection to your network interactions:

Exceeding connect timeout will give us desirable behavior:

2019/11/25 19:12:25 Get http://httpbin.org/delay/5: dial tcp 54.172.95.6:80: i/o timeout

Go gives you a lot of flexibility with it’s standard HTTP client, and library is supported by the vendor — perfect combination, that’s why I like it very much! Now let’s get back to Java world one more time.

So, is Java doomed?

No. There are less popular alternatives, such as JDK 11 http client and OkHttp, which supports call timeout features. Sadly, those are not the top results when searching on Google, so people are less aware of them or not willing to start using them.

JDK 11

It is a modern http client delivered with standard library, which supports HTTP/1.1, HTTP/2, async calls via CompletableFuture and provides convenient api to work with. Let’s combine both timeouts with it:

This client will provide nice and easy to understand error messages(especially compared to Go) for connect and call timeouts respectively:

Exception in thread "main" java.net.http.HttpConnectTimeoutException: HTTP connect timed out

Exception in thread "main" java.net.http.HttpTimeoutException: request timed out

OkHttp

OkHttp supports call and connect timeouts too. Both can be configured at client abstraction level, which is slightly more convenient compared to JDK11 implementation:

How to choose timeout values, anyway?

Last thing I’d like to mention is how to derive those 2 numbers. Call timeout is more about SLA/SLO you have with other services and connect timeout is about expectations from underlying network. For example, if you’re sending requests to the same datacenter, then 100ms would be fine (although it should establish connection faster than 5ms), but operating on top of mobile networks (which are more error-prone) will require higher connect timeouts.

Wrap-up

In this article we discussed why timeouts are important, why ‘classic’ timeouts doesn’t meet modern requirements and which instruments to use to force them. I hope you found something useful in it.

Why I like Go HTTP client as a Java developer was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

I made a program that finds, among your starred repositories, the issues that are most likely to be a good choice for making a contribution. For a quick start, check out the How-To on the GitHub project page.

Why would one need it?

Developers are obsessed with open source development nowadays. Everyone wants to give back to the community, to get some new skills and experience while working on the most widely known software products. Imagine getting invaluable insights from experienced engineers all over the world: pretty exciting, right?

However, people struggle hard to find appropriate tasks to work on. If you are new to the project, it is most likely that the maintainers won’t let you work on some big, fancy feature, lest you miss a plenty of details and corner cases. Everyone agrees that one would better start with the easiest tasks, learn the ropes, get some trust from maintainers, and then proceed to harder ones.

So, how do you find these easy issues to work on?

Some time ago, GitHub started listing appropriate issues, e.g., https://github.com/golang/go/contribute. You can access this list from the header of the Issues page. This is how it looks like:

Although it is pretty accurate, it sometimes lacks the appropriate labeling or misses some items.

My usual “let-me-make-a-contribution” day used to look like this: I was looking through my starred repositories and jumping to this “new contribution” section, or was scrolling the issues list manually. That was a very monotonous and time-consuming task, so I used to get bored quickly by this process and would give up. That is when I decided to automate it.

Solution

First, I needed to work out the rules of the game. I did some research and came up with the following heuristics:

• An issue should not be closed.
• Assigned means assigned, don’t waste your time on it.
• An issue should be marked with an “easy” label or something similar.
• Old issues are likely to be outdated; if it remains unsolved for too long, maintainers are likely not interested in it and probably less interested to help you during the process.

Furthermore, it turns out that every repository uses its own set of labels, and it is rarely just “help wanted”. There are multitudes of them. I looked through all the labels in my starred repositories and other popular ones and came up with a final list of ~60 labels. If any of your repositories have different labels, create a pull request or contact me — I will add it as soon as possible. As I set boundaries, it was the time to create the program itself.

I built a Kotlin application, which looks through the list of starred repositories and searches for the issues with predefined labels, sorts them and has them printed to the console. The current way of distribution is a Docker image — people are more likely to have it installed than Java ;)

As the labels list grew, I noticed a drastic decrease in speed and an increase in the consumption of API usage limits. The first edition of the tool was searching for issues with all given labels, although 95% of them had never been used in that repo! That was my “AHA!” moment, so I added an additional call to fetch all the labels from the repository and intersect them with my list. I got a significant improvement of the performance, and utilized less of the API limits.

On a daily basis, the tool works fine as a guide to the world of open-source contributions, it will also help you to achieve your Hacktoberfest goals. To try it, visit the project page and follow the instructions. If you lack starred repositories — I have some advices for you as well.

If you have something to say — do it in the section below, or in the dedicated feedback issue.

Open source made easier was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.