Error handling in Go has evolved since the introduction of Go 1.13, bringing two essential functions to the standard library: errors.Is and errors.As. These tools enable developers to handle errors more robustly, allowing for better error inspection and handling in a structured, type-safe manner. This article explores how you can make the most of these two functions for cleaner and more efficient error handling in Go applications while avoiding anti-patterns that hinder readability.

The Problem with Traditional Error Checking

A common practice in error handling involves having to check for the types of errors being returned from a function. Knowing how to check for types is dependent on whether they are standard error instances or custom structs that implement the error interface.

Before Go 1.13, error handling typically relied on comparing errors directly using equality == checks. This approach works well for simple error values but becomes problematic with wrapped errors as many functions return errors that wrap the original ones.

Here’s an example of the above :

if err == io.EOF {
    // handle EOF error
}

This works fine if err is exactly io.EOF, but if err wraps io.EOF (for example, when using fmt.Errorf with %w), the direct comparison fails:

err := fmt.Errorf("read failed: %w", io.EOF)
if err == io.EOF {
    // This won't work because err wraps io.EOF
}

Likewise, another common practice in error handling is being able to extend data captured within an error by creating a custom error type which implements the error interface

For example:

type AppError struct {
    message string
    statusCode int
}

func NewAppError(message string, statusCode int) *AppError {
    return &AppError{essage, statusCode}
}

func (e AppError) Error() string {
    return e.message
}

func (e AppError) StatusCode() int {
    return e.statusCode
}

Prior to Go 1.13, checking if an error was an AppError required doing something like the following:

err := Foobar()
if appErr, ok := err.(*AppError); err != nil && ok {
    // handle AppError instance
}

It’s not rare to still find the above being done or even direct string comparisons of error content to identify specific errors. Here’s where errors.Is and errors.As come in to handle errors more effectively…

Understanding errors.Is

errors.Is allows you to check if an error is or wraps a specific error, even when it’s been wrapped by another error.

Here’s an example of errors.Is:

package main

import (
    "errors"
    "fmt"
    "io"
)

func main() {
    err := fmt.Errorf("read failed: %w", io.EOF)
    // Using errors.Is to check if err wraps io.EOF
    if errors.Is(err, io.EOF) {
        fmt.Println("EOF error detected")
    } else {
        fmt.Println("Different error")
    }
}

In this case, errors.Is successfully identifies that err wraps io.EOF. This is particularly useful when you’re dealing with libraries or functions that return wrapped errors and you need to inspect the underlying cause.

Practical Use Case for errors.Is

Consider a scenario where a function interacts with a file and may encounter multiple types of errors. Some of these errors may be more severe than others. You may want to handle `os.ErrNotExist` errors (file not found) differently from `io.EOF` (end of file).

package main

import (
    "errors"
    "fmt"
    "os"
)

func main() {
    _, err := os.Open("non-existent-file.txt")
    if err != nil {
        if errors.Is(err, os.ErrNotExist) {
            fmt.Println("File does not exist")
        } else {
            fmt.Println("An unexpected error occurred:", err)
        }
    }
}

In this example, errors.Is enables you to identify the specific error without worrying about how deeply it may be wrapped.

Using errors.As for Type-Specific Error Handling

While errors.Is checks if an error is or wraps a specific error, errors.As is used to check if an error can be assigned to a variable of a particular error type. This is valuable when you need to interact with custom error types or perform operations based on the specific properties of the underlying error.

Here’s an example using the AppError defined earlier:

func main() {
    err := fmt.Errorf("some context: %w", NewAppError("something went wrong", http.StatusInternalServerError))
    var myErr *AppErr
    if errors.As(err, &myErr) {
        fmt.Println("AppErr detected:", myErr.Msg)
    } else {
        fmt.Println("Different error")
    }
}

In this case, errors.As unwraps the error and assigns the underlying MyError to myErr, allowing you to handle it based on its type.

Practical Use Case for `errors.As`

Consider a situation where you need to handle errors from a third-party library/API. It might return errors with specific types, and you want to inspect the properties of those errors to take action. Using errors.As, you can extract and interact with these error types:

// insert a user into the table
_, err = db.Exec("INSERT INTO users (email) VALUES ($1)", "test@example.com")

// check for constraint violation errors using errors.As
if err != nil {
    var pqErr *pq.Error
    if errors.As(err, &pqErr) { // handle specific PostgreSQL error
        if pqErr.Code == "23505" { // 23505 is the PostgreSQL code for unique_violation
            fmt.Printf("Unique constraint violation on column %s\n", pqErr.Column)
        } else {
            fmt.Printf("PostgreSQL Error: Code=%s, Message=%s\n", pqErr.Code, pqErr.Message)
        }
    } else {
        fmt.Printf("Unknown error occurred: %v\n", err)
    }
} else {
    fmt.Println("Insert succeeded!")
}

In this example, errors.As allows you to extract the postgres pq error type and act based on its properties, such as its error code or message.

Conclusion

The errors.Is and errors.As functions introduced in Go 1.13 provide a more powerful way to handle errors, especially when dealing with wrapped or custom error types. By leveraging these functions, you can write more maintainable and expressive error-handling logic, making your Go applications more robust and easier to debug.

Benefits of Using errors.Is and errors.As

• Enhanced Error Inspection: You can inspect and match both specific and wrapped errors, leading to more robust error handling.
• Type-Specific Logic: With errors.As, you can differentiate and handle custom error types, improving your ability to respond appropriately to different error conditions.
• Compatibility with Existing Code: These functions work seamlessly with older Go code, making it easy to adopt them incrementally.

When to Use errors.Is vs. errors.As

• Use errors.Is when you need to check if an error is or wraps a particular error value (e.g., io.EOF or os.ErrNotExist).
• Use errors.As when you need to check if an error can be cast to a specific type and access its underlying properties.

Incorporating these practices into your codebase can improve error diagnostics and provide better control over how errors are handled across your application, especially when dealing with third-party libraries or complex, multi-layered systems.

Go’s struct tags offer a way to define metadata for struct fields allowing developers to specify how these fields should be handled during operations like serialization or validation. A common use case is the json tag, which tells the Go JSON package how to map struct fields to JSON keys. For example:

type User struct {
    ID int `json:"id"`
    Name string `json:"name"`
    Email string `json:"email"`
}

In this article, we’ll explore how to create custom struct tags to implement custom functionality. We will use the use case of sanitizing HTML content — an important task in many web applications to prevent cross-site scripting (XSS) attacks. You’ll learn how to define custom tags, implement custom tag logic, and integrate them into your Go structs seamlessly.

What are Struct Tags?

In Go, struct tags are metadata defined in backticks `` after the field declarations in a struct. These tags can be processed using the reflect package, which allows introspection of Go types at runtime. The general format for struct tags looks like this:

type StructName struct {
    FieldName FieldType `tagName:"value"`
}

You can define multiple tags on a single field, for example, json and a custom tag for sanitization sanitize:

type Comment struct {
    Body string `json:"body" sanitize:"stripall"`
}

In this example, stripall could be a custom sanitization instruction that indicates the field should be stripped of all HTML content before it’s stored or processed.

Implementing HTML Sanitization Using Struct Tags

Let’s build a custom solution to sanitize HTML content from struct fields using struct tags. Our goal is to create a function that inspects each field of a struct and applies the necessary sanitization based on the tag’s value.

Step 1: Defining the Custom Tag Values

The sanitize tag will instruct our code to strip all HTML or allow some safe HTML tags, based on its value:

• stripall: Remove all HTML tags.
• safeugc: Allow user-generated content (UGC) with limited safe HTML tags (e.g., <b>, <i>).

Here’s how the struct might look:

type Comment struct {
    Body string `json:"body" sanitize:"stripall"`
    Title string `json:"title" sanitize:"safeugc"`
}

Step 2: Writing the Sanitization Function

We’ll write a SanitizeStruct function that uses Go’s reflect package to process the struct tags and sanitize fields accordingly:

func getCleansedValue(tag string, value string) (string, error) {
 var cleanedValue string
 switch TagValue(tag) {
 case StripAll:
  cleanedValue = config.StripAll(value)
 case SafeUGC:
  cleanedValue = config.SafeUGC(value)
 default:
  return "", ErrInvalidTagValue
 }

 return cleanedValue, nil
}

// SanitizeStruct takes a struct and sanitizes it based on `sanitize` tags
func SanitizeStruct(s interface{}) error {
 val := reflect.ValueOf(s).Elem()
 typ := val.Type()

 for i := 0; i < val.NumField(); i++ {
  field := val.Field(i)
  fieldType := typ.Field(i)
  fieldKind := field.Kind()

  tag := fieldType.Tag.Get("sanitize")
  if tag == "" {
   if fieldKind == reflect.Struct {
    if err := SanitizeStruct(field.Addr().Interface()); err != nil {
     return err
    }
   }
   continue
  } else if fieldKind == reflect.Struct {
   return ErrInvalidPropertyType
  } else if fieldKind != reflect.String && !(fieldKind == reflect.Slice && field.Type().Elem().Kind() == reflect.String) {
   return ErrInvalidPropertyType
  }

  if fieldKind == reflect.Slice {
   for j := 0; j < field.Len(); j++ {
    cleanedValue, err := getCleansedValue(tag, field.Index(j).String())
    if err != nil {
     return err
    }
    field.Index(j).SetString(cleanedValue)
   }
  } else {
   cleanedValue, err := getCleansedValue(tag, field.String())
   if err != nil {
    return err
   }
   field.SetString(cleanedValue)
  }
 }

 return nil
}

Leveraging the above function would then be as easy as the following:

func main() {
 comment:= &Comment{
  Body: "<script>alert('xss')</script>",
  Title: "<b>Bold Title</b>",
 }
 
 err := SanitizeStruct(comment)
 if err != nil {
  fmt.Println("Error:", err)
 }
 
 fmt.Printf("Sanitized Comment: %+v\n", comment)
}

In this example:

• SanitizeStruct iterates over the fields of a struct and applies the relevant sanitization function based on the sanitize tag.
• StripAllHTML removes all HTML from the field’s value, and SafeUGC removes only unsafe HTML tags.

The output is as follow:

Sanitized Comment: &{Body: Title:<b>Bold Title</b>}

When not to use struct tags

Struct tags are not always the best tool for more complex logic or scenarios requiring dynamic behavior. Since struct tags are static and evaluated at runtime using reflection, they lack flexibility in cases where business logic might change frequently or needs to be conditionally applied. Moreover, tags aren’t suitable for implementing more complex validation workflows, error handling, or behavior that depends on external factors, as reflection can become slower and harder to debug when logic becomes more intricate. For more advanced or changing needs, embedding such logic elsewhere in your code, like in dedicated functions or middleware, might be a better approach.

Conclusion

Go’s struct tags offer a flexible and declarative way to add special field logic directly into your data structures. By combining custom struct tags with reflection, you can create reusable solutions that can be easily implemented across struct properties. Please note, using reflection to inspect struct tags adds overhead, so use this approach carefully in performance-sensitive areas!

For the full code used above please click here.

When building applications that interact with external APIs, it’s crucial to verify that your code makes the correct requests and handles responses appropriately. Testing such code can be challenging due to the reliance on external services. Luckily, Go’s httptest package offers the httptest.Server tool, allowing you to simulate external services within your tests seamlessly.

In this article, we’ll cover the basics of httptest.Server, how to set it up, and explore various strategies for simulating external responses to ensure your application behaves as expected.

What is httptest.Server?

The httptest.Server is a utility that enables you to create an in-memory HTTP server for testing purposes. It mimics the behavior of an actual web server, allowing you to test how your code interacts with an API without needing a live external service. This can help verify request correctness, response handling, and error scenarios efficiently and reliably.

Here’s a basic example:

package main

import (
 "fmt"
 "net/http"
 "net/http/httptest"
 "testing"
)

func TestBasicExample(t *testing.T) {
 // create a new test server with a simple handler
 server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
  fmt.Fprintln(w, `{"message": "hello world"}`)
 }))
 defer server.Close()

 // make an HTTP request to the test server
 resp, err := http.Get(server.URL)
 if err != nil {
  t.Fatalf("Failed to make request: %v", err)
 }
 defer resp.Body.Close()

 // check the response
 if resp.StatusCode != http.StatusOK {
  t.Fatalf("Expected status 200 OK, got %v", resp.Status)
 }
}

the above code can be thought of as three parts:

1. setting up the test server and responses
2. having whatever client code you write point to the test server
3. verifying responses

Simulating Different Responses

A powerful feature of httptest.Server is the ability to define custom responses, allowing you to simulate different API responses and status codes. This is invaluable for testing how your code handles various scenarios, including both success and failure cases.

Example: Custom Responses and Status Codes

func TestAPIClient_HandleErrorResponse(t *testing.T) {
 server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
  http.Error(w, `{"error": "not found"}`, http.StatusNotFound)
 }))
 defer server.Close()

 client := &APIClient{BaseURL: server.URL}
 _, err := client.GetMessage()

 if err == nil {
  t.Error("Expected an error, got none")
 }
}

Handling Configurations in Tests

Depending on how your application is configured, you’ll need to ensure that your API client points to the httptest.Server instance rather than the actual external API during testing. You can handle this in a few ways:

• Updating Configurations Temporarily: If your API base URL is defined in a configuration file or environment variable, you can temporarily update this value in your tests to point to the httptest.Server‘s base URL and then reset it once done as needed.
• Passing Base URL Directly: Alternatively, if your functions accept the URL as a parameter, you can simply pass httptest.Server’s base URL during the test.

Complete Basic Example

Combining the concepts above, let’s walk through a basic test setup that verifies different response handling:

type APIClient struct {
 BaseURL string
}

type APIResponse struct {
 Message string `json:"message"`
 Error   string `json:"error"`
}

func (client *APIClient) GetData() (*APIResponse, error) {
 resp, err := http.Get(client.BaseURL + "/data")
 if err != nil {
  return nil, fmt.Errorf("failed to make request: %w", err)
 }
 defer resp.Body.Close()

 var apiResp APIResponse
 if err := json.NewDecoder(resp.Body).Decode(&apiResp); err != nil {
  return nil, fmt.Errorf("failed to decode response: %w", err)
 }

 return &apiResp, nil
}

func TestAPIClient_GetData(t *testing.T) {
 tests := []struct {
  name           string
  serverResponse string
  serverStatus   int
  expectedMsg    string
  expectedErrMsg string
  expectedCode   int
 }{
  {
   name:           "Successful response",
   serverResponse: `{"message": "success"}`,
   serverStatus:   http.StatusOK,
   expectedMsg:    "success",
  },
  {
   name:           "Not Found response",
   serverResponse: `{"error": "not found"}`,
   serverStatus:   http.StatusNotFound,
   expectedErrMsg: "not found",
  },
  {
   name:           "Internal Server Error",
   serverResponse: `{"error": "internal error"}`,
   serverStatus:   http.StatusInternalServerError,
   expectedErrMsg: "internal error",
  },
 }

 for _, test := range tests {
  // create a new HTTP test server for each test case
  server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
   w.WriteHeader(test.serverStatus)
   w.Write([]byte(test.serverResponse))
  }))
  defer server.Close()

  client := &APIClient{BaseURL: server.URL}

  t.Run(test.name, func(t *testing.T) {
   resp, err := client.GetData()

   if resp == nil {
    t.Fatal("Expected response, got nil")
   } else if err != nil {
    t.Fatalf("Expected no error, got %v", err)
   }

   // check the response message or error message
   if test.expectedMsg != "" && resp.Message != test.expectedMsg {
    t.Errorf("Expected message '%s', got '%s'", test.expectedMsg, resp.Message)
   } else if test.expectedErrMsg != "" && resp.Error != test.expectedErrMsg {
    t.Errorf("Expected error message '%s', got '%s'", test.expectedErrMsg, resp.Error)
   }
  })
 }
}

Best Practices for httptest.Server

1. Close Test Servers: Always defer the Close method to ensure the server is shut down after the test completes.
2. Use Dedicated Configurations in Tests: Avoid hardcoding URLs in your application code. Instead, make the API URL configurable so you can easily switch to an httptest.Server during testing.
3. Handle Various Status Codes: Be sure to test how your code handles different HTTP status codes and responses, as external APIs can fail in various ways.
4. Use Response Simulation Sparingly: While it’s powerful to mock various responses, remember that over-mocking can make tests fragile and overly reliant on specific assumptions.
5. Test Realistic Scenarios with Faker: To simulate real-world scenarios, especially in cases involving user data, consider using libraries like Faker to generate random but structured data.

Conclusion

Testing external API interactions can be complex, but httptest.Server in Go provides a robust solution for simulating these scenarios. By implementing these strategies, you can write reliable tests for your application’s external service integrations, ensuring your application responds as expected to various outcomes.

I created a module called fauxmux that simulates various responses, errors, and more. Check it out for ideas on building a reusable solution for testing external API interactions in your services.

Go’s database/sql package provides automatic database connection pooling allowing developers to efficiently manage connections. Typically, as a developer, you'd request an open connection, perform your query, and close the connection to ensure that it’s returned to the pool.

One common mistake developers make is holding a database connection open for too long which can lead to bottlenecks, where new requests are forced to wait for an available connection, hindering the effectiveness of the connection pool.

In this article, we’ll explore how to avoid this issue and optimize Go applications for better throughput by identifying a common problem area and learning how to resolve it.

Basic Example

To start, Let’s take a look at a basic HTTP handler that returns employee records:

func GetEmployeesHandler(w http.ResponseWriter, r *http.Request) {
    rows, err := db.Query(`SELECT id, name, email FROM employee`)
    if err != nil {
        http.Error(w, fmt.Sprintf("error querying database: %v", err), http.StatusInternalServerError)
        return
    }
    defer rows.Close()

    var employees []Employee
    for rows.Next() {
        var e Employee
        if err := rows.Scan(&e.ID, &e.Name, &e.Email); err != nil {
            http.Error(w, fmt.Sprintf("Error scanning row: %v", err), http.StatusInternalServerError)
            return
        }
        decorateEmployee(&e)
        employees = append(employees, e)
    }

    if err = rows.Err(); err != nil {
        http.Error(w, fmt.Sprintf("error during row iteration: %v", err), http.StatusInternalServerError)
        return
    }

    w.Header().Set("Content-Type", "application/json")
    if err := json.NewEncoder(w).Encode(employees); err != nil {
        http.Error(w, "Error encoding response", http.StatusInternalServerError)
        return
    }
}

In this handler we:

1. query the database for employee records.
2. use defer rows.Close() to ensure that the connection is closed once we're done processing the result set.
3. scan each row and decorate it with data obtained from an external source.
4. append the final struct to the result array.
5. check for any errors during iteration and return the results as JSON.

At first glance, nothing seems out of the ordinary; in fact, you’d expect the above to perform relatively well under stress.…

Initial Performance Results

Using a stress-testing tool like Vegeta, we can simulate load on this endpoint. At an initial rate of 10 requests per second (RPS), the application performs relatively well during a 30 second test run:

$ echo "GET http://localhost:8080/employees" | vegeta attack -duration=30s -rate=10 | tee results.bin | vegeta report
Requests      [total, rate, throughput]         300, 10.03, 5.45
Duration      [total, attack, wait]             52.095s, 29.9s, 22.196s
Latencies     [min, mean, 50, 90, 95, 99, max]  2.318s, 11.971s, 8.512s, 26.222s, 30.001s, 30.001s, 30.001s
Bytes In      [total, mean]                     2290991, 7636.64
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           94.67%
Status Codes  [code:count]                      0:16  200:284

However, when we increase the load to 50 RPS, we see a significant drop in throughput and a sharp rise in request failures:

$ echo "GET http://localhost:8080/employees" | vegeta attack -duration=30s -rate=50 | tee results.bin | vegeta report
Requests      [total, rate, throughput]         1500, 50.03, 4.20
Duration      [total, attack, wait]             59.981s, 29.981s, 30s
Latencies     [min, mean, 50, 90, 95, 99, max]  2.208s, 27.175s, 30.001s, 30.001s, 30.001s, 30.002s, 30.002s
Bytes In      [total, mean]                     2032879, 1355.25
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           16.80%
Status Codes  [code:count]                      0:1248  200:252

( The Status Codes of 0 above signify client timeouts that occurred during the test run )

Identifying the Bottleneck

At 50 RPS, the success rate drops dramatically, and the throughput falls to just 4.2 requests per second. Why? One reason could be that 50 RPS is just an unreasonable target given our current resources. To confirm whether the code could perform better with a change, we can look into gathering some metrics. One source of metrics is the decorating process, but for the purpose of this article, we will focus on database connection pool statistics.

Go’s database/sql package offers a view into how an application’s DB pooling is performing via its DBStats function. It returns the following set of stats that we’ll be interested in:

1. InUse: The number of connections currently in use.
2. Idle: The number of idle connections.
3. WaitCount: The total number of connections waited for.

We can expose these values by adding another endpoint handler:

func GetInfoHandler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    if err := json.NewEncoder(w).Encode(db.Stats()); err != nil {
        http.Error(w, "Error encoding response", http.StatusInternalServerError)
        return
    }
}

Monitoring the /info endpoint during a rerun of the above stress test shows:

$ while true; do curl -s http://localhost:8080/info; sleep 2; done
...
{"MaxOpenConnections":15,"OpenConnections":15,"InUse":15,"Idle":0,"WaitCount":1434,"WaitDuration":1389188829869,"MaxIdleClosed":0,"MaxIdleTimeClosed":0,"MaxLifetimeClosed":0}
{"MaxOpenConnections":15,"OpenConnections":15,"InUse":15,"Idle":0,"WaitCount":1485,"WaitDuration":1582086078604,"MaxIdleClosed":0,"MaxIdleTimeClosed":0,"MaxLifetimeClosed":0}
{"MaxOpenConnections":15,"OpenConnections":15,"InUse":15,"Idle":0,"WaitCount":1485,"WaitDuration":1772844971842,"MaxIdleClosed":0,"MaxIdleTimeClosed":0,"MaxLifetimeClosed":0}
...

This reveals that the connection pool is maxed out (InUse: 15, Idle: 0), and every new request is being forced to wait (WaitCount keeps increasing). In other words, the connection pool is essentially stalled, causing the latency and timeout issues observed earlier!

Optimizing Connection Usage

Looking at our original code above, we see that the problem is either the query itself is performing poorly or the decorating function is taking long to return each time its called while we iterate through the result set. We can explore the latter by attempting to decorate the record outside of the rows.Next() loop and moving it below our database connection usage.

Here’s the updated code:

func GetEmployeesHandler(w http.ResponseWriter, r *http.Request) {
    rows, err := db.Query(`SELECT id, name, email FROM employee`)
    if err != nil {
        http.Error(w, fmt.Sprintf("error querying database: %v", err), http.StatusInternalServerError)
        return
    }

    var employees []Employee
    for rows.Next() {
        var e Employee
        if err := rows.Scan(&e.ID, &e.Name, &e.Email); err != nil {
            if closeErr := rows.Close(); closeErr != nil {
                err = fmt.Errorf("%w; failed to close rows: %v", err, closeErr)
            }
            http.Error(w, fmt.Sprintf("error scanning row: %v", err), http.StatusInternalServerError)
            return
        }

        employees = append(employees, e)
    }

    if err = rows.Err(); err != nil {
        if closeErr := rows.Close(); closeErr != nil {
            err = fmt.Errorf("%w; failed to close rows: %v", err, closeErr)
        }
        http.Error(w, fmt.Sprintf("error during row iteration: %v", err), http.StatusInternalServerError)
        return
    }

    for i := range employees {
        decorateEmployee(&employees[i])
    }

    w.Header().Set("Content-Type", "application/json")
    if err := json.NewEncoder(w).Encode(employees); err != nil {
        http.Error(w, "Error encoding response", http.StatusInternalServerError)
        return
    }
}

In this refactored handler, we:

1. Scan all the rows into memory.
2. Close the connection immediately after scanning, freeing it back to the pool (Don’t forget to handle errors related to closing connections!).
3. Decorate the employee records in memory, without holding the connection open.

Performance After Optimization

Running the same Vegeta test at 50 RPS after the optimization yields the following results:

$ echo "GET http://localhost:8080/employees" | vegeta attack -duration=30s -rate=50 | tee results.bin | vegeta report
Requests      [total, rate, throughput]         1500, 50.03, 45.78
Duration      [total, attack, wait]             32.768s, 29.98s, 2.788s
Latencies     [min, mean, 50, 90, 95, 99, max]  2.045s, 2.502s, 2.499s, 2.692s, 2.741s, 2.856s, 2.995s
Bytes In      [total, mean]                     11817000, 7878.00
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:1500

...
{"MaxOpenConnections":15,"OpenConnections":1,"InUse":0,"Idle":1,"WaitCount":0,"WaitDuration":0,"MaxIdleClosed":0,"MaxIdleTimeClosed":0,"MaxLifetimeClosed":0}
{"MaxOpenConnections":15,"OpenConnections":1,"InUse":0,"Idle":1,"WaitCount":0,"WaitDuration":0,"MaxIdleClosed":0,"MaxIdleTimeClosed":0,"MaxLifetimeClosed":0}
{"MaxOpenConnections":15,"OpenConnections":1,"InUse":0,"Idle":1,"WaitCount":0,"WaitDuration":0,"MaxIdleClosed":0,"MaxIdleTimeClosed":0,"MaxLifetimeClosed":0}
...

As we can see, not only was there huge improvement in throughput and latency with 100%, we also see that the total OpenConnections doesn’t make it past 1 and we even have Idle connections on standby thereby keeping the WaitCount consistently at zero!

Summary

Instead of holding the connections open while performing additional I/O-bound operations like decorating records, we can optimize the handling of connections by fetching all rows into memory first, then closing the connection immediately. This allows the database connection to be returned to the pool sooner, freeing up resources for other incoming requests allowing for better throughput and concurrency.

Using the Repository Pattern

Applying the repository pattern can further improve code structure by separating storage actions from business logic. The repository pattern allows you to abstract away the data access layer (database interactions) from the rest of the application. This approach offers several benefits:

1. Better Code Separation and Readability: By separating database logic from the application’s core business logic, each part becomes easier to read, understand, and maintain.
2. Easier Testing with Mocked Repositories: The repository pattern allows you to create mock implementations of the database layer, which can simulate database responses for testing. For instance, instead of querying the database, the handler function can rely on a mock repository, which returns predefined data. This makes it easier to isolate and test the handler’s performance without involving the database.
3. Improved Connection Management: With the repository pattern, you can make each database interaction more modular, ensuring that resources are managed efficiently (e.g., handling rows.Close() at the repository layer). This minimizes the risk of connection pooling bottlenecks.

Here’s an example of what applying the repository pattern to the GetEmployeesHandler might look like :

type EmployeeService interface {
    GetEmployees() ([]Employee, error)
}

...
func (es Client) GetEmployees() ([]Employee, error) {
    rows, err := es.db.Query(`SELECT id, name, email FROM employee`)
    if err != nil {
        return nil, err
    }
    defer func() {
        if closeErr := rows.Close(); closeErr != nil {
            log.Printf("error closing rows: %v", closeErr)
        }
    }()

    var employees []Employee
    for rows.Next() {
        var e Employee
        if err := rows.Scan(&e.ID, &e.Name, &e.Email); err != nil {
            return nil, err
        }
        employees = append(employees, e)
    }

    if err = rows.Err(); err != nil {
        return nil, err
    }

    return employees, nil
}
...

func (api API) GetEmployeesHandler(w http.ResponseWriter, r *http.Request) {
    employees, err := api.Employees().GetEmployees()
    if err != nil {
        http.Error(w, fmt.Sprintf("Error retrieving employees: %v", err), http.StatusInternalServerError)
        return
    }

    for i := range employees {
        decorateEmployee(&employees[i])
    }

    w.Header().Set("Content-Type", "application/json")
    if err := json.NewEncoder(w).Encode(employees); err != nil {
        http.Error(w, "Error encoding response", http.StatusInternalServerError)
        return
    }
}

Using the repository pattern in this way enables better resource management, testing, and modularity. It also means you can use defer again to close the connection because the operation itself is kept atomic!

For the full code used above please click here.

Cheers!