E2E / Integration Testing in Golang with ory/dockertest

While unit and component tests are crucial for verifying individual parts of an application work as expected, e2e/integration tests play an equally important role in validating how those components function together. They test the entire system from end to end, spanning multiple components or services. This catches issues that occur between integrated components that unit tests would not find.

The ory/dockertest package is a useful tool for testing Go applications that interact with databases and other services running in Docker containers. In this post, we'll look at how to use dockertest to write automated tests for a Go application.

Overview

When testing an application that depends on external services like databases, queues, etc., we often run into the problem of setting up and managing these dependencies during tests. The dockertest package makes this easier by spinning up Docker containers for the dependencies as needed, then cleaning them up after the tests finish.

Some key features of dockertest:

Automatically pulls Docker images if needed
Launches containers in the background
Provides helpers to wait for containers to become ready before running tests
Removes containers after tests complete

This means we can focus on writing the actual test logic rather than all the setup/teardown boilerplate.

Basic Example

Let's look at a simple example. Say we have a Go API that needs to connect to a Redis database. Here is how we could use dockertest to spin up a temporary Redis container for our tests:

package main

import (
  "testing"

  "github.com/ory/dockertest/v3"
  "github.com/ory/dockertest/v3/docker"
)

func TestMain(m *testing.M) {
  pool, err := dockertest.NewPool("")
  if err != nil {
    log.Fatalf("Could not connect to docker: %s", err)
  }

  // pulls an image, creates a container based on it and runs it
  resource, err := pool.Run("redis", "latest", []string{"POSTGRES_PASSWORD=secret", "POSTGRES_DB=myappdb"})
  if err != nil {
    log.Fatalf("Could not start resource: %s", err)
  }

  // exponential backoff-retry, because the application in the container might not be ready to accept connections yet
  if err := pool.Retry(func() error {
    var err error
    db, err = sql.Open("postgres", fmt.Sprintf("postgres://postgres:secret@localhost:%s/myappdb?sslmode=disable", resource.GetPort("5432/tcp")))
    if err != nil {
      return err
    }
    return db.Ping()
  }); err != nil {
    log.Fatalf("Could not connect to docker: %s", err)
  }

  code := m.Run()

  // You can't defer this because os.Exit doesn't care for defer
  if err := pool.Purge(resource); err != nil {
    log.Fatalf("Could not purge resource: %s", err)
  }

  os.Exit(code)
}

func TestSomething(t *testing.T) {
  // do something with db
}

The key steps are:

Create a dockertest.Pool to manage Docker containers.
Use pool.Run to start a PostgreSQL container.
Wait for the container to start using pool.Retry - the application may take some time to launch and accept connections.
Run tests as usual, connecting to the PostgreSQL container.
Cleanup the container after tests finish with pool.Purge.

This allows running the tests without having to manually install a PostgreSQL server on the test environment. The dockertest package handles all the container management under the hood.

Implementation in a Go REST API Project

Let's Implement this in a Real Golang REST API project that will expose an endpoint called /healthz which when called will connect with a redis database and return a 200 if the database is healthy and if not, then it will return a 503 service unavailable.

Project Structure

I've added the project with all the files and logic in this repository. The project structure is as follows:

├── Dockerfile         -- Dockerfile for the API
├── README.md   
├── main.go            -- Main file for the API containing the routes 
├── go.mod   
├── go.sum
└── e2e                -- Folder containing the e2e tests
    ├── init_test.go   -- File containing the TestMain function
    └── main_test.go   -- File containing the e2e tests

Dockerfile

The Dockerfile defines how to build the image for the API. It starts from the golang base image, copies the source code into the container, builds the Go binary, and defines the startup command.

FROM golang:1.21.1 as build

ENV GOPATH=/go

WORKDIR /app

COPY . .

RUN GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /app/myapi main.go

FROM alpine:3.18.3

WORKDIR /app

COPY --from=build /app/myapi /app/myapi

RUN addgroup -S appgroup && adduser -S appuser -G appgroup
RUN chmod +x /app/myapi
USER appuser

EXPOSE 8080

HEALTHCHECK --interval=60s --timeout=3s --start-period=5s --retries=3 CMD [ "wget", "-q", "http://localhost:8080/healthz", "-O", "-" ]

ENTRYPOINT ["/app/myapi"]

REST API file (main.go)

We'll create a simple API that connects to a Redis database and exposes a health endpoint. The API will be built using the gin-gonic/gin package.

It exposes the /healthz endpoint that returns a 200 response if the database is healthy, or a 503 response if the database is not healthy.

package main

import (
    "context"
    "net/http"
    "os"

    "github.com/gin-gonic/gin"
    "github.com/redis/go-redis/v9"
)

func main() {
    // Set up Gin router
    router := gin.Default()

    // Define a route for the "/healthz" endpoint
    router.GET("/healthz", func(c *gin.Context) {
        // Check the database connection health
        if err := checkDatabaseHealth(); err != nil {
            c.JSON(http.StatusServiceUnavailable, gin.H{"status": "Database is not healthy"})
            return
        }

        c.JSON(http.StatusOK, gin.H{"status": "Database is healthy"})
    })

    // Start the Gin server
    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }
    router.Run(":" + port)
}

// checks the connection to redis database
func checkDatabaseHealth() error {
    db := redis.NewClient(&redis.Options{
        Addr:     "redis-container:6379",
        Password: "",
        DB:       0,
    })

    _, err := db.Ping(context.Background()).Result()
    if err != nil {
        return err
    }

    return nil
}

Configuring the Test Suite with TestMain (e2e/init_test.go)

The TestMain function is a special function that runs before any tests are run. We can use this to set up the test suite, including starting the Redis container and connecting to it.

This file will do the following:

Create a Docker pool to manage containers.
Create a network for the containers to communicate over.
Deploy the Redis container.
Build the API container as specified in the Dockerfile.
Deploy the API container.
Run the tests that are in the e2e_test package.
Tear down the containers and network after the tests finish.
Exit the test suite.

package e2e_test

import (
    "context"
    "fmt"
    "github.com/ory/dockertest/v3"
    "github.com/ory/dockertest/v3/docker"
    "github.com/redis/go-redis/v9"
    "log"
    "net/http"
    "os"
    "testing"
)

// Declare a global variable to hold the Docker pool and resource.
var (
    network *dockertest.Network
)

func TestMain(m *testing.M) {
    // Initialize Docker pool and ensure it's closed at the end.
    pool, err := dockertest.NewPool("")
    if err != nil {
        log.Fatalf("Could not connect to Docker: %v", err)
    }

    // Create a Docker network for the tests.
    network, err = pool.CreateNetwork("test-network")
    if err != nil {
        log.Fatalf("Could not create network: %v", err)
    }

    // Deploy the Redis container.
    redisResource, err := deployRedis(pool)
    if err != nil {
        log.Fatalf("Could not start resource: %v", err)
    }

    // Deploy the API container.
    apiResource, err := deployAPIContainer(pool)
    if err != nil {
        log.Fatalf("Could not start resource: %v", err)
    }

    resources := []*dockertest.Resource{
        redisResource,
        apiResource,
    }

    // Run the tests.
    exitCode := m.Run()

    // Exit with the appropriate code.
    err = TearDown(pool, resources)
    if err != nil {
        log.Fatalf("Could not purge resource: %v", err)
    }

    os.Exit(exitCode)
}

// deployRedis builds and runs the Redis container.
func deployRedis(pool *dockertest.Pool) (*dockertest.Resource, error) {
    resource, err := pool.RunWithOptions(&dockertest.RunOptions{
        Hostname:     "redis-container",
        Repository:   "redis",
        Tag:          "latest",
        ExposedPorts: []string{"6379"},
        PortBindings: map[docker.Port][]docker.PortBinding{
            "6379/tcp": {{HostIP: "", HostPort: "6379"}},
        },
        Networks: []*dockertest.Network{
            network,
        },
    })
    if err != nil {
        return nil, fmt.Errorf("could not start resource: %v", err)
    }

    // Ensure the Redis container is ready to accept connections.
    if err := pool.Retry(func() error {
        fmt.Println("Checking Redis connection...")
        db := redis.NewClient(&redis.Options{
            Addr:     "localhost:6379",
            Password: "",
            DB:       0,
        })

        _, err := db.Ping(context.Background()).Result()
        if err != nil {
            return err
        }

        defer db.Close()

        return nil
    }); err != nil {
        return nil, fmt.Errorf("could not connect to docker: %v", err)
    }

    return resource, nil
}

// TearDown purges the resources and removes the network.
func TearDown(pool *dockertest.Pool, resources []*dockertest.Resource) error {
    for _, resource := range resources {
        if err := pool.Purge(resource); err != nil {
            return fmt.Errorf("could not purge resource: %v", err)
        }
    }

    if err := pool.RemoveNetwork(network); err != nil {
        return fmt.Errorf("could not remove network: %v", err)
    }

    return nil
}

// deployAPIContainer builds and runs the API container.
func deployAPIContainer(pool *dockertest.Pool) (*dockertest.Resource, error) {
    // build and run the API container
    resource, err := pool.BuildAndRunWithBuildOptions(&dockertest.BuildOptions{
        ContextDir: "../",
        Dockerfile: "Dockerfile",
    }, &dockertest.RunOptions{
        Name:         "api-container",
        ExposedPorts: []string{"8080"},
        PortBindings: map[docker.Port][]docker.PortBinding{
            "8080": {{HostIP: "0.0.0.0", HostPort: "8080"}},
        },
        Networks: []*dockertest.Network{
            network,
        },
    })

    if err != nil {
        return nil, fmt.Errorf("could not start resource: %v", err)
    }

    // check if the API container is ready to accept connections
    if err = pool.Retry(func() error {
        fmt.Println("Checking API connection...")
        _, err := http.Get("http://localhost:8080/healthz")
        if err != nil {
            return err
        }

        return nil
    }); err != nil {
        return nil, fmt.Errorf("could not start resource: %v", err)
    }

    return resource, nil
}

Now we have the test suite ready and configured. We can write the actual tests in the e2e_test package.

Writing the Tests

The tests will be written in the e2e_test package. This package will be run as part of the TestMain function in the init_test.go file.

When this test is run locally, it will spin up the Redis and API containers, run the tests, and then tear down the containers. When this test is run in the pipeline, it will spin up the Redis and API containers, run the tests, and then tear down the containers.

package e2e_test

import (
    "net/http"
    "testing"
)

// TestHealthRoute tests the /healthz endpoint.
func TestHealthRoute(t *testing.T) {
    // create a get request to localhost:8080/healthz
    // check that the response status code is 200

    request, err := http.NewRequest("GET", "http://localhost:8080/healthz", nil)
    if err != nil {
        t.Fatalf("Could not create request: %v", err)
    }

    response, err := http.DefaultClient.Do(request)
    if err != nil {
        t.Fatalf("Could not make request: %v", err)
    }

    if response.StatusCode != http.StatusOK {
        t.Errorf("Expected status 200, got %d", response.StatusCode)
    }
}

Running the Tests

To run the tests locally, run the following command:

go test ./...

This will run the tests in the e2e_test package, which will spin up the Redis and API containers, run the tests, and then tear down the containers. One benefit of using dockertest is that it will automatically pull the Redis and API images if they are not already present on the local machine. Also, other tests in the project will be run as usual.

The TestMain function will be used only for running tests inside the e2e_test package.

If you have another test package that needs a similar setup/teardown logic, you can create a similar TestMain function in that package.

Conclusion

In this post, we looked at how to use dockertest to write automated tests for a Go application. We saw how to use dockertest to spin up a temporary Redis container and a temporary API container for our tests, then tear it down after the tests are finished.

The ory/dockertest package provides a powerful and convenient way to spin up Docker containers for testing dependencies. By handling the setup and teardown of containers in the background, it frees us to focus on writing robust integration tests for our Go applications. The full lifecycle testing enabled by dockertest increases confidence in releasing code to production. While it does require configuring tests to use containers, this overhead pays dividends in catching issues early. The dockertest documentation contains many more examples for testing various databases, queues, and services. Overall, dockertest is an invaluable tool for anyone looking to improve their automated testing workflows with Docker and Go.

References

ory/dockertest - https://github.com/ory/dockertest

go-e2e-test-blog - https://github.com/cksidharthan/go-e2e-test-blog

E2E / Integration Testing in Golang with ory/dockertest

Comments

Golang

Create a simple fileserver in Golang

More from this blog

The day I got tired of my feeds breaking

Falling in Love with RSS Feeds Again!

My Journey into embracing Neovim as my PDE

Visualizing Your Git Repository History with Gource

Git Submodules: A Game-Changer for Streamlining Microservices Development

Overview

Basic Example

Implementation in a Go REST API Project

Project Structure

Dockerfile

REST API file (main.go)

Configuring the Test Suite with TestMain (e2e/init_test.go)

Writing the Tests

Running the Tests

Conclusion

References

Command Palette

Comments

Golang

Create a simple fileserver in Golang

More from this blog

Overview

Basic Example

Implementation in a Go REST API Project

Project Structure

Dockerfile

REST API file (main.go)

Configuring the Test Suite with TestMain (e2e/init_test.go)

Writing the Tests

Running the Tests

Conclusion

References