Abstracting the basics

The series over generalizes and significantly undermines complexity of the actual implementation of the underlying systems. And is expected to just point out the more commonly seen issues in typical implementation.

It helps to understand what goes under the hood in batch systems before we jump to optimize them. The most common ones you’ll see would be

1. Coordinator and Workers

2. Tasks

3. Shuffling

4. Processing

5. Reading and Writing

The above diagram tries to put a picture to some of the components we will be discussing.

Coordinator and Worker

Almost all jobs have a coordinator ( aka manager, driver, master, etc) and worker nodes.

The responsibility of coordinator is as the name suggests:

1. Pick up what needs to be executed.

2. Create a plan that divides the huge job into stages, creating subtasks to be executed for each of these stages.

3. Manage stage completion and task progress check-pointing.

The responsibility of workers is :

1. Pick up a task from the coordinator.

2. Execute the task.

3. Return the result / or commit it to a temp storage.

Tasks

Most batch systems 100s of GBs, TBs of data if not PBs. At this scale, we have already left the realm of individual machines and entered the realm of “horizontally scaling”. A huge task of processing 100s of TBs worth of data into partitioned files. These partitions are then used to fetch the results of a query that can be divided into 100K or more smaller tasks. A typical task would only process a partition or part of partition and return the result. It would help if we look at a task as a simple metadata that looks like

task {
  task_id,
  query,
  read {
    partition: a
    from: x
    to: y 
  }
}

So when a worker receives a task it just processes a part of the request for the query provided and returns its result back to the coordinator.

Shuffling

Ok! I lied about the part where a task only reads from x to y. Depending on what the query is requesting the task might not have the complete data required to provide the results. For instance, a query to write all the values related to a key onto disk in a single file cannot be completed without reading all the petabytes of data.

So how do we solve the problem when a query that cannot be served by just processing parts of the problem? In such cases, the coordinator requests workers to rearrange the data aka shuffle the data by key. This results in data being written in chunks that facilitate creation of continuous tasks (as mentioned before).

Processing

When a worker receives a task it performs some set of operations requested by coordinator. A task only performs a subset of work so you will see workers see pick up multiple tasks at the same time and doing what is needed a part of these tasks.

It is always very important to understand what processing is happening as part of the task to debug / speed your job execution as we will see in some later posts of this series.

Reading and Writing

It is important to mention at this stage that almost all of these batch processing systems heavily rely on Distributed File Systems(DFS) ( such as HDFS, GFS, and S3) properties to store and process temporary states.

The fault tolerance and scalability of DFS significantly improves the performance at a per task level. The read and write performance is also significantly improved based on the choice of data file format you choose to store and read data from. Text files are much slower than column-oriented data formats such as parquet and Arrow.

What’s next?

In the follow up posts, I will share some of the key bottlenecks with which we were able to improve job performance and reduce infrastructure costs by as much as 50% in some cases. Additionally, new levels of complexity and details will surface as we start digging deeper into problems and their solutions.

Some time back I found myself reaching the cafe and my table being occupied. The reader should know at this point that I’m turning into one of those old man who gets massively disappointed over small changes like not getting their fav table. So, I decided to take the matters into my own hand and over engineered a solution for my problem.

Problem Statement

We want to identify the crowd in an around a particular place with minimal cost involved. No calling and confirming from cafes is not an option (we want over-engineered solution pay attention please). We will measure our solution on 3 parameters:

1. Cost.

2. Near real-time metrics.

3. Coolness.

Solution

My first thoughts were to just access to the camera’s setup but that would require creating some unethical backdoor. I have been fiddling around with these cheap ESP32 processors costs around Rs 500 or so.

These bad bois have a dual core processor, wifi, bluetooth and what not. So, I decided to use them along with a simple rechargeable battery from nearest Ikea.

How do I use them?

So typically all phones and a ton of other devices use bluetooth and Bluetooth Low Energy (BLE) for all sort of cool stuff. If you haven’t you must read about the Apple’s FindMy network. What I have done is I’ve written bunch of micropython scripts that scan the area for devices and send an update to my server at home.

The server at home just simply run a FastAPI server and captures the number of devices in and around the device. Each device has it’s own identification which tells me which cafe is seeing how many people around my table. My precious table:

So when I need to go out for a coffee I just query the API and it tells me which cafe table has minimal people around my table and I get to that one.

Corner cases

Since these devices are quite exposed I need to do some extra leg effort for solving bunch of things:

1. Battery consumption: I’m lazy I don’t want to change the battery's every so often so the device goes into deep sleep when it has pushed the data and sleeps for 5 mins. The device goes into deep sleep at night times (uses time sync from another API).

2. Home server safety: My server is DDoS protected and has TLS to avoid anyone figuring out how they can mess up with my system.

3. Internet: I’m heavily at the mercy of the cafes’ wifi I’m currently in the process of working out a solution that does not need the cafes’ wifi but that is experimental at best.

4. Losses: So far I’ve not lost or gotten my device stolen. But even if it does get stolen it’s a cheap device.

What’s next?

I’m thinking of extending the solution to some more interesting use-cases. I’d be happy to hear folks thought on what sort of over-engineered solutions you are working on.

“Proprietary” ( sounds like - pruh·prai·uh·tree) is a tough to spell word with an interesting pronunciation. Skipping the tedious explanation of how I’ll inevitably flunk the dictation despite my repeated attempts let’s get straight to the issue at hand. We find ourselves in an interesting setup for this one:

1. We have a client claiming very high latency from our service.

2. We have a backend service claiming very low latency.

needless to say the initial few calls with the client went like:

because we thought it was their network and they thought it was ours.

So, we ran a bunch of tests find some minor configuration issues on the application end and okayed the system to be used by the client. And to our surprise the client kept seeing the same issue. At this point, I’m more curious than frustrated as to what is happening, since there has been no improvement whatsoever.

Problem

Upon picking up high latency request we traced it across the logs in our system and this is what we saw from it:

So when we looked at the gateway logs (the proprietary software) we can clearly see the request reached the gateway with a 2 second delay and like all other developers we agreed it was an network issue. Because why not? I can see I did not see the request on my system before that point (nice word play huh ! ). Anyway after doing tons of optimization we saw no improvements. So we sat with a laptop next to the server plugged it in and ran the tests and we still see 2 - 3 SECOND latency. This tells us :

1. Network is not the problem.

2. Either I have a bad machine.

3. Or my logs are not reliable.

We ruled out point 2 because we tested on multiple machines and the probability of us getting that lucky to have all the bad servers is quite low (knowing my luck we cannot however completely rule that out :D). And now the meaty problem for this post. How does one debug an application who’s code you don’t own and whose logs are not reliable?

We reached out to the proprietary software support and after multiple discussion they refused to acknowledge that this is a problem in the software. We are now stuck and we still need to find the issue.

Debugging without code and logs

So we are at a point where the application owners are not able to come up with any suggestions on what might be causing this. I decided to now deep dive on the issue. While I don’t have access to the applications internal or any authoritative logs I do have access to server. What can we do with server access in a proprietary software? Well unless the application is designed keeping security in mind you can pretty much figure out everything the application is doing. In my case, I used 2 tools to identify what is happening:

1. strace
   This allowed me to see what system calls the application is making to the kernel and where is it spending time from an linux’s perspective. Strace is very useful in identifying what is happening in pretty much all applications.

2. tcpdump
   This allowed me to see (not completely correct my traffic was over TLS) what is happening on the connection stream.

How to’s

This post is useless if I don’t give back a quick way to use both strace and tcpdump. Within the next 2 mins:

strace

For linux OS you can use the following command to identify all high latency requests and what they are doing in terms of time spent and data written in these calls.

strace -f -s 100 -p PID 2>&1 | awk 'gsub(/[<>]/,"",$NF); if($NF+0 > 0.8) print $0 " " $NF}'

What does it do:

-p → uses the parent process id.
-f → tracks all the child process for PID
-s → decides what size of data to be read in the call params.
2>&1 → redirects stderror to stdout for linux (default output of strace is stderr)

awk → parses the output of strace to only show syscalls where 0.8 secs or more was spent.

On seeing the results of this command we were able to see a lot of time being spent on calls to a SQL DB.

tcpdump

For seeing what’s happening in your system on a per connection basis you can use :

tcpdump -i INTERFACE -w -s 0 -w output.pcap

What does it do:

It captures all the traffic on INTERFACE and writes it to output.pcap

You’ll need Wireshark to see or use the generated PCAP file. In my case, it clearly showed a bunch of DB calls to the MySQL server. These DB calls when they saw high latency were creating a back-pressure and causing all calls to have a higher latency (my guess is the proprietary code has some synchronized block blocking all threads but that is for another day or blog post).

I highly recommend the short read on using by Julia Evans : How I use Wireshark by now you have any ways invested 5 mins in learning why not invest another 5mins and have some more depth into the tool.

Closing thoughts

Since we didn’t own the Gateway code there was only so much we could do. The application team acknowledged the issue and decided to provide a patch for the same. I think the big takeaway was how much speed the two tools provided in deep diving on the issue even when application team denied any issues. My recommendation would be to definitely give them a shot.

The story of our system starts some years back when we were writing a service to process invoices and run some validations on them. At the time of writing the service, invoices were fairly small. To process an invoice, the workflow was fairly simple:

1. Download the invoice header.

2. Get all transactions of an invoice (line items).

3. Aggregate and match the total amounts.

For this, we would make a call to a different service that would return the transactions and their details.

The crippling slowness

Over the years, we started seeing bigger invoices coming into our system. Soon, invoices would have millions of line items. Naturally, this slowed down the speed at which invoices were being processed. So, it was ignored as a side-effect of larger invoices coming in for validation.

Soon, the problem grew bigger. It now took around 6-12-24 hours for invoices to process. Since the problem grew slowly, it did not cause an alarm. Rather, it got accepted as a norm. And now, we had teams building system architecture and solutions to account for the long-running validations.

Identification

As part of one of our resource utilization reviews, it was identified that the particular validation service was neither compute-intensive nor memory-heavy. When asked to justify this, the team said that they process huge invoices (1Gb to 10Gb) and write them to disk, which is slow. This is why they need more hosts to scale their systems to process more invoices.

A large invoice can be 1 gigabyte to 10 gigabytes in file size. This should have been the first hint for the team to go back to the drawing board and find the real issue. Modern disks do not take a day to write to in 202x. This was somehow overlooked and accepted as a reason for the slowness of the service by the team. During the review, however, this did not fly because of the 202x (2020-2030s because the future looks dark) logic.

The blunder

Yeah, so for years, we accepted the slowness of the system and changed the architecture around the slow validation process. However, during one review, we were unable to identify a single resource that was causing the host to be slow. So, we took a deep dive into the code. And lo and behold, we identified a bug in our code. I'm sharing the pseudo-code below so that readers can identify the issue in the code.

class TransactionLogger {
    // Client to call external service for getting transaction details 
    private TransactionAuthorityService tasSAO;
    
    public List<Transaction> getTransactions(List<String> transactionIds) {
        List<Transaction> transactionDetails = new DiskList();
        for(String transactionId : transactionIds) {
           transactions.append(tasSAO.getTransactionDetails(transactionId);
        }
        return transactionDetails;
    }

}

Now, at first sight, there is nothing wrong with this code. We are doing what we are expected to do: get transactions and store them to disk.

The issue is that this code is sequential. The disk is not the bottleneck; it is the damn RPC call, which waits for every call to return before making the next one.

The 10 line fix to 5 year old architectural problem

The fix for the problem is rather simple. We add parallel calls to fetch the data from dependency service.

class TransactionLogger {
    // Client to call external service for getting transaction details 
    private TransactionAuthorityService tasSAO;
    private ExecutorService executorService;
    
    public List<Transaction> getTransactions(List<String> transactionIds) {
                List<Transaction> transactionDetails = new DiskList<>();
        
        // Create transactions in parallel
        List<Future<Transaction>> transactionFutures = new ArrayList<>();
        for(String transactionId : transactionIds) {
            List<Transaction> future = executorService.submit(() ->   tasSAO.getTransactionDetails(transactionId));
            transactionFutures.add(future);
        }

        // Fetch all transactions
        for(Future<Transaction> future: transactionFutures) {
            transactionDetails.add(future.get());
        }
        return transactionDetails;
    }

}

Disclaimer: The above code is not production-ready. You need to:

1. Add batching to control the size of running futures at any time.

2. Add rate limiters to your downstream service so you don't brown them out.

3. There are better ways to write this code, but I just wanted to keep it simple for the readers.

   

The takeaway

As developers, we need to know the breaking point of our services and what resources we are constrained on before we resort to horizontal scaling for every problem.