It all started in 2016 in my first company. I wanted to edit a shell script in a Virtual Machine where my application was deployed. The manual instructed me to log into the terminal via PuTTY, run vi <script_name>.sh, and then add a line to it. I followed the same, only to discover that I couldn't do Ctrl+C, Ctrl+V, or Ctrl+S, nor was I able to quit. Frustrated, I quit PuTTY and logged in again. I was like, "Yuck, what the hell is that editor?" and used Nano to edit the file. I thought, "Why don't they have Nano as the default editor in Linux?"

The Intriguing Encounter

In 2020, when I started working on NGINX, I was paired with a Senior Engineer. During my onboarding (via Zoom call - during COVID times), he was explaining the codebase to me. To my astonishment, instead of opening an IDE like a JetBrains product or VSCode, he opened his iTerm2, navigated to the project folder, and opened Vim. I was expecting a movement in the mouse pointer, but man, he was flying through the codebase like I've never seen before. At that moment, I thought, "I want that."

The Initial Struggles

So, I started looking at tutorial videos and documentation, but it was hard for me totally. I tried to hang on, but over the week, I lost interest (it was hard to begin as a total noob). My workflow continued to be Goland or VSCode over the year. I again gave it a try in 2021 (went into the YouTube rabbit hole one night and ended up in a Vim tutorial). I configured the editor as per the tutorial (used Vim Plug) and found it comfortable to edit files. I spent the next couple of weeks forcing myself to use it as my primary editor. At the end of the sprint, I saw that I didn't finish up my tasks. It dawned on me that maybe I'm not that kind of guy, and it took a total nerd to use Vim as fluently as an IDE, which I'm used to. So, I stopped using it.

The Renewed Attempts

In 2022, I moved to a different company, and there, another Senior Engineer in my team started using Neovim, and I thought I would try it one more time. Let's see; maybe I'm up for it now. The same pattern emerged where I was dead slow. I thought this is it. I'm not gonna try again and stopped using it once and for all (except when doing interactive rebase in Git).

The Pivotal Moment

Everything was going well until I joined a new company again (yes, hectic times, I moved to 3 different companies in 3 years - moved countries and other things out of my hand). But anyway, back to the subject. I wanted to give it one more try again because of a guy called Primeagen. YouTube suggested his video for some reason, the video was cool, and there it was waiting for me. He was using Neovim, and man, I got to see the editor again, which I thought I wouldn't touch or go near. He was basically flying at lightning speed, and it was fun watching him navigate through the codebase. This time, I wanted to use it badly (wanted to have fun while coding). I was impatient in configuring my own settings, so I installed NvChad. It looked so pretty, and I was learning the cheatsheet and was like, "Wow, this is interesting." I was learning all the keycombos and Vim motions; it was fun, and I was happy. But still, I was very slow and used GoLand and VSCode like 80% of my dev work. But still, I was getting used to the hjkl navigation and other keystrokes, but then I slowly moved to using IDEs again and forgot about it.

The Perseverance

I did continuously watch ThePrimeagen and TJDeVries, loved all their uploads. I was using Neovim here and there sometimes but not fully as my primary editor. I tried LunarVim, LazyVim, AstroNvim; all had different keycombos and project structure. I was pretty confused and thought, no way I will be able to create my own config, and even if I did, it would be shitty compared to what others are using and with the other Nvim distros. I tried using kickstart.nvim that TJ DeVries suggested, but for some reason, it didn't stick with me.

The Eureka Moment

Finally, I stumbled upon a video from Josean Martinez. I watched the video entirely, and man, at the end of the video, I got the hang of it. It was my eureka moment where I was like, "Shit, I've been dumb all along." I mean, other videos I watched over the year were good too, but this one hit my brain like nothing else. I was like, "Yeah, let's roll, baby." And there started my Neovim journey.

It's been two weeks since I watched that video, and I joined the crowd, I think, tweaking my Nvim config, adding plugins, removing them, and doing this never-ending tinkering process. Programming is fun again, and I started using Neovim as my primary editor. As TJDeVries says, I finally found my PDE (Personal Development Environment). I now get the hang of it and can't think of not using Vim motions in any editor (PS: I installed the Vimium extension in my browser and love it too).

The Takeaway

Anyway, what I was about to tell is that, to all fellow people who are trying to use Neovim or trying to do programming in a fun way, just hang in there; there is light at the end of the tunnel. I know that I barely scratched the surface and am a beginner in Neovim now, but let's see what the future holds.

Some YouTube channels that I follow for Neovim content:

• ThePrimeagean

• TJDeVries

• Josean Martinez

• typecraft

• Dreams of Code

And finally all thanks to my Senior Engineers, ThePrimeagen, TJDevries and the whole Neovim and Vim community for everything and I guess its time for me to contribute to the ecosystem. I'll share my Neovim configurations and the plugins that I use in future blogs. 😁

My Nvim Config - https://github.com/cksidharthan/nvim

Gource is an open-source command-line tool that visualizes activity in your Git repository over time through a mesmerizing animated graphical representation. It's like a real-time interactive movie that reveals the evolution of your project.

In this post, we'll cover what Gource is, why it's useful for visualizing your Git history, and how to use it with your repositories. It's especially great as a cool party trick when showing off your project's development history in presentations!

What is Gource?

Gource is a captivating and interactive visualization tool designed to breathe life into your Git repositories. Created by Andrew Caudwell, this open-source software offers a unique perspective on your project's history by translating code commits into a mesmerizing animated tree. It provides an intuitive visualization of your repo's development lifecycle over time. It's available for Windows, macOS, and Linux, making it accessible to a wide range of developers and teams.

Why Use Gource?

Here are some of the key benefits of using Gource:

• See an overview of commit frequency and patterns over time

• Identify major events like new branches, merges, deletions etc

• Get a sense of the pace of development and who is contributing

• Easily spot organizational changes like renamed files or directories

• Enjoy a cool visualization of your project's history that works great in presentations!

Gource visualizations are often eye-catching and mesmerizing, revealing insights through dynamic graphical representations. It's a fantastic party trick to quickly convey your project's growth and evolution when presenting to audiences.

Using Gource

Gource is available for Linux, macOS, and Windows. To start using it, you simply need to install it and point it to a Git repository directory.

Installation

On Linux/macOS, Gource is usually available through your package manager, e.g.:

sudo apt install gource # on Ubuntu/Debian 
brew install gource # on Mac with Homebrew

For Windows, you can download and run a pre-compiled executable.

Basic Usage

Navigate to your Git repository directory on the command line. Then run:

gource --auto-skip-seconds 0.1 -s 0.1

This will generate a real-time animation visualizing the entire commit history of the repository from inception to now.

To create a video file of the gource visualization use the below command. (NOTE: this requires ffmpeg tool to be installed in your machine.

gource --auto-skip-seconds 0.1 -s 0.1 --output-ppm-stream - | ffmpeg -y -r 30 -f image2pipe -vcodec ppm -i - -b 65536K movie.mp4

This pipes the animation into ffmpeg to encode an MP4 video. Adjust frame rate and encoding as needed.

Customizing the Visualization

Gource provides many command line options to customize the visualization. I have listed some of them below.

• Set start/end dates with --start-date and --stop-date

• Highlight specific users with --highlight-users

• Adjust simulation speed with --max-files and --time-scale

• Change render size with --viewport

See the docs for all available options.

Summary

Gource provides a mesmerizing way to visualize activity in your Git repository over time. It's easy to install and use - just point it at a repo directory to generate an animation. The graphical representation quickly conveys insights into the development process and makes for an impressive interlude in presentations.

With some customization, you can highlight important events and contributors in the visualization. Give Gource a try if you want a graphical overview of your project's Git history! Your audiences will love the eye-catching animated visualization.

References

https://github.com/acaudwell/Gource

Understanding Git Submodules

Before diving into our journey of using Git submodules to streamline microservices development, let's briefly understand what Git submodules are.

Git submodules are Git repositories nested within another Git repository. They allow you to include one Git repository (the submodule) inside another (the main repository) as a subdirectory. This means you can keep a reference to an external repository within your own repository, effectively importing its contents.

Now, let's embark on our journey and understand why and how we used Git submodules.

The Challenge: Duplicated Code and Configuration

In our development landscape, we had multiple microservices, each residing in its own Git repository. While these microservices had distinct functionalities, they all adhered to the same project structure and shared common tasks defined in a Taskfile.yml. Additionally, we used a common Vagrantfile to create standardized development environments across different operating systems.

However, maintaining this consistency became increasingly cumbersome. We found ourselves duplicating tasks and the Vagrantfile across multiple repositories, leading to a maintenance nightmare. Any changes or improvements in these shared components required manual updates in every microservice repository. It was time-consuming and prone to errors.

The Solution: Git Submodules

As we searched for a solution to streamline our microservices development, we stumbled upon Git submodules. They seemed like the perfect fit for our use case.

Step 1: Creating a Common Repository

Our first step was to create a dedicated Git repository called dev-templates. This repository would house the shared taskfiles each separated into different files like docker.yml, go.yml, helm.yml, vagrant.yml etc., with common development tasks and standardized Vagrantfile for creating development environments.

Step 2: Adding dev-templates as a Submodule

With dev-templates ready, we added it as a submodule to all our microservice repositories. This was achieved with a simple Git command:

git submodule add https://github.com/yourusername/dev-templates.git <folder-name>

the folder-name can be replaced with the folder name where you want the submodule to be cloned inside the project.

By doing this, we imported the entire dev-templates repository into each microservice repository as a subdirectory. This effectively eliminated the need to duplicate code and configuration files.

Step 3: Simplifying Maintenance

Now, when we needed to update common tasks or the Vagrantfile, we made the changes in the dev-templates repository. Whenever we wanted to pull those changes into our microservice repositories, we executed the following command:

git submodule update --recursive --remote

We added this as a task in our Taskfile.yml in the individual microservices repo. And just like that, all our microservices were up to date with the latest changes from dev-templates.

The Benefits of Using Git Submodules

Our journey to incorporating Git submodules into our microservices development process brought several key benefits:

1. Code Reusability

By centralizing common code and configurations in a dedicated repository, we significantly reduced code duplication across microservices. This not only saved us time but also made our codebase more maintainable.

2. Streamlined Maintenance

Updating shared components became a breeze. Instead of manually propagating changes to every repository, we only had to update dev-templates. The submodule update command took care of the rest.

3. Consistency

All microservices now adhered to the same project structure and development environment, ensuring consistency and reducing the chances of issues arising from differences in configurations.

Conclusion

Our journey of using Git submodules to manage shared code, tasks, and development environments across multiple microservice repositories has proven to be a game-changer. It simplified our development process, reduced maintenance overhead, and enhanced the overall consistency of our projects.

If you're facing similar challenges in your microservices development, consider giving Git submodules a try. They might be the solution you've been searching for to streamline your development workflow.

By implementing Git submodules, we were able to streamline our microservices development workflow, reduce code duplication, and simplify maintenance. If you have any questions or would like more detailed examples of how to work with Git submodules, feel free to reach out. :)

In today's digital age, online presence is a fundamental aspect of our lives. From social media platforms to discussion forums, we leave traces of our identity across the internet. But have you ever wondered how to track down someone's online profiles using just their username? Enter Sherlock, a powerful tool designed to scrape the internet and unearth user profiles associated with a given username. In this blog post, we'll delve into the world of Sherlock, exploring its features, use cases and ethical considerations. Plus, we'll guide you through how to use Sherlock responsibly.

What is Sherlock?

Sherlock is an open-source tool developed by Siddharth Dushantha. It's designed to help users discover online profiles connected to a specific username. Whether you're an investigator, a security professional, or simply curious, Sherlock can assist in finding information about individuals on the web.

Key Features of Sherlock

1. User-Friendly Command Line Interface (CLI): Sherlock boasts a straightforward CLI, making it accessible to users with varying levels of technical expertise. This ease of use sets it apart from more complex web scraping tools.

2. Extensive Platform Support: Sherlock supports a wide range of social media platforms, including Twitter, Instagram, LinkedIn, and many others. This versatility ensures that you can search for profiles across popular networks.

3. Information Aggregation: Once Sherlock has gathered information on profiles linked to a username, it neatly presents the results, making it easy to analyze and cross-reference data.

4. Active Development: Sherlock is an open-source project, meaning it's actively maintained and improved by a community of contributors. This ensures its reliability and adaptability over time.

How Does Sherlock Work?

Sherlock employs web scraping techniques to search for profiles associated with a given username. Here's a simplified overview of its operation:

1. Installation: Before using Sherlock, ensure you have Python 3 installed on your system. Then, clone the Sherlock repository and navigate to its directory in the terminal.

    git clone https://github.com/sherlock-project/sherlock.git 
    cd sherlock
    python3 -m pip install -r requirements.txt
   

2. Running Sherlock: Once installed, run Sherlock by providing a username as a command-line argument. Replace <username> with the username you want to search for.

    python3 sherlock.py john_doe
   

    Output:
   
    ❯ python3 sherlock john_doe
    [*] Checking username john_doe on:
   
    [+] 8tracks: https://8tracks.com/john_doe
    [+] 9GAG: https://www.9gag.com/u/john_doe
    [+] About.me: https://about.me/john_doe
    [+] Amino: https://aminoapps.com/u/john_doe
    [+] Apple Developer: https://developer.apple.com/forums/profile/john_doe
    [+] Apple Discussions: https://discussions.apple.com/profile/john_doe
    [+] Archive of Our Own: https://archiveofourown.org/users/john_doe
    [+] Archive.org: https://archive.org/details/@john_doe
    [+] AskFM: https://ask.fm/john_doe
    [+] Audiojungle: https://audiojungle.net/user/john_doe
    [+] BLIP.fm: https://blip.fm/john_doe
    [+] Bandcamp: https://www.bandcamp.com/john_doe
    [+] Behance: https://www.behance.net/john_doe
    [+] Bikemap: https://www.bikemap.net/en/u/john_doe/routes/created/
    [+] Bitwarden Forum: https://community.bitwarden.com/u/john_doe/summary
    [+] BodyBuilding: https://bodyspace.bodybuilding.com/john_doe
    [+] Bookcrossing: https://www.bookcrossing.com/mybookshelf/john_doe/
    [+] BuzzFeed: https://buzzfeed.com/john_doe
    ...
    ...
    ... cutting short the results to keep the blog small :)
   
    [*] Search completed with 97 results
   

3. Analyzing the Results: Sherlock will query various social media platforms and display the results in your terminal. Positive matches are displayed in green, while negative matches are in red.

4. Advanced Usage: Sherlock offers various options and flags to enhance your searches. You can exclude specific websites from the search, save results to a text file, and more. Refer to the documentation for advanced usage.

Ethical Considerations

While Sherlock is a powerful tool, it's essential to use it responsibly and ethically:

1. Privacy and Consent: Always respect privacy and obtain consent when conducting searches on individuals. Using Sherlock for malicious purposes or stalking is unethical and potentially illegal.

2. Verification: Information gathered through Sherlock should be verified independently before taking any action or drawing conclusions.

3. Legality: Be aware of the legal implications of using web scraping tools like Sherlock in your jurisdiction. Laws regarding data privacy and web scraping vary from place to place.

Conclusion

Sherlock is a valuable tool that can help users find online profiles associated with a username efficiently. Whether you're conducting investigations, managing your digital footprint, or reconnecting with lost connections, Sherlock can be a useful asset. However, it's crucial to approach its use with ethics and legality in mind, respecting privacy and obtaining consent whenever necessary. As with any tool, responsible and considerate usage should always be the guiding principle.

References

Sherlock Github Repo - https://github.com/sherlock-project/sherlock

The ory/dockertest package is a useful tool for testing Go applications that interact with databases and other services running in Docker containers. In this post, we'll look at how to use dockertest to write automated tests for a Go application.

Overview

When testing an application that depends on external services like databases, queues, etc., we often run into the problem of setting up and managing these dependencies during tests. The dockertest package makes this easier by spinning up Docker containers for the dependencies as needed, then cleaning them up after the tests finish.

Some key features of dockertest:

• Automatically pulls Docker images if needed

• Launches containers in the background

• Provides helpers to wait for containers to become ready before running tests

• Removes containers after tests complete

This means we can focus on writing the actual test logic rather than all the setup/teardown boilerplate.

Basic Example

Let's look at a simple example. Say we have a Go API that needs to connect to a Redis database. Here is how we could use dockertest to spin up a temporary Redis container for our tests:

package main

import (
  "testing"

  "github.com/ory/dockertest/v3"
  "github.com/ory/dockertest/v3/docker"
)

func TestMain(m *testing.M) {
  pool, err := dockertest.NewPool("")
  if err != nil {
    log.Fatalf("Could not connect to docker: %s", err)
  }

  // pulls an image, creates a container based on it and runs it
  resource, err := pool.Run("redis", "latest", []string{"POSTGRES_PASSWORD=secret", "POSTGRES_DB=myappdb"})
  if err != nil {
    log.Fatalf("Could not start resource: %s", err)
  }

  // exponential backoff-retry, because the application in the container might not be ready to accept connections yet
  if err := pool.Retry(func() error {
    var err error
    db, err = sql.Open("postgres", fmt.Sprintf("postgres://postgres:secret@localhost:%s/myappdb?sslmode=disable", resource.GetPort("5432/tcp")))
    if err != nil {
      return err
    }
    return db.Ping()
  }); err != nil {
    log.Fatalf("Could not connect to docker: %s", err)
  }

  code := m.Run()

  // You can't defer this because os.Exit doesn't care for defer
  if err := pool.Purge(resource); err != nil {
    log.Fatalf("Could not purge resource: %s", err)
  }

  os.Exit(code)
}

func TestSomething(t *testing.T) {
  // do something with db
}

The key steps are:

1. Create a dockertest.Pool to manage Docker containers.

2. Use pool.Run to start a PostgreSQL container.

3. Wait for the container to start using pool.Retry - the application may take some time to launch and accept connections.

4. Run tests as usual, connecting to the PostgreSQL container.

5. Cleanup the container after tests finish with pool.Purge.

This allows running the tests without having to manually install a PostgreSQL server on the test environment. The dockertest package handles all the container management under the hood.

Implementation in a Go REST API Project

Let's Implement this in a Real Golang REST API project that will expose an endpoint called /healthz which when called will connect with a redis database and return a 200 if the database is healthy and if not, then it will return a 503 service unavailable.

Project Structure

I've added the project with all the files and logic in this repository. The project structure is as follows:

├── Dockerfile         -- Dockerfile for the API
├── README.md   
├── main.go            -- Main file for the API containing the routes 
├── go.mod   
├── go.sum
└── e2e                -- Folder containing the e2e tests
    ├── init_test.go   -- File containing the TestMain function
    └── main_test.go   -- File containing the e2e tests

Dockerfile

The Dockerfile defines how to build the image for the API. It starts from the golang base image, copies the source code into the container, builds the Go binary, and defines the startup command.

FROM golang:1.21.1 as build

ENV GOPATH=/go

WORKDIR /app

COPY . .

RUN GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /app/myapi main.go

FROM alpine:3.18.3

WORKDIR /app

COPY --from=build /app/myapi /app/myapi

RUN addgroup -S appgroup && adduser -S appuser -G appgroup
RUN chmod +x /app/myapi
USER appuser

EXPOSE 8080

HEALTHCHECK --interval=60s --timeout=3s --start-period=5s --retries=3 CMD [ "wget", "-q", "http://localhost:8080/healthz", "-O", "-" ]

ENTRYPOINT ["/app/myapi"]

REST API file (main.go)

We'll create a simple API that connects to a Redis database and exposes a health endpoint. The API will be built using the gin-gonic/gin package.

It exposes the /healthz endpoint that returns a 200 response if the database is healthy, or a 503 response if the database is not healthy.

package main

import (
    "context"
    "net/http"
    "os"

    "github.com/gin-gonic/gin"
    "github.com/redis/go-redis/v9"
)

func main() {
    // Set up Gin router
    router := gin.Default()

    // Define a route for the "/healthz" endpoint
    router.GET("/healthz", func(c *gin.Context) {
        // Check the database connection health
        if err := checkDatabaseHealth(); err != nil {
            c.JSON(http.StatusServiceUnavailable, gin.H{"status": "Database is not healthy"})
            return
        }

        c.JSON(http.StatusOK, gin.H{"status": "Database is healthy"})
    })

    // Start the Gin server
    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }
    router.Run(":" + port)
}

// checks the connection to redis database
func checkDatabaseHealth() error {
    db := redis.NewClient(&redis.Options{
        Addr:     "redis-container:6379",
        Password: "",
        DB:       0,
    })

    _, err := db.Ping(context.Background()).Result()
    if err != nil {
        return err
    }

    return nil
}

Configuring the Test Suite with TestMain (e2e/init_test.go)

The TestMain function is a special function that runs before any tests are run. We can use this to set up the test suite, including starting the Redis container and connecting to it.

This file will do the following:

• Create a Docker pool to manage containers.

• Create a network for the containers to communicate over.

• Deploy the Redis container.

• Build the API container as specified in the Dockerfile.

• Deploy the API container.

• Run the tests that are in the e2e_test package.

• Tear down the containers and network after the tests finish.

• Exit the test suite.

package e2e_test

import (
    "context"
    "fmt"
    "github.com/ory/dockertest/v3"
    "github.com/ory/dockertest/v3/docker"
    "github.com/redis/go-redis/v9"
    "log"
    "net/http"
    "os"
    "testing"
)

// Declare a global variable to hold the Docker pool and resource.
var (
    network *dockertest.Network
)

func TestMain(m *testing.M) {
    // Initialize Docker pool and ensure it's closed at the end.
    pool, err := dockertest.NewPool("")
    if err != nil {
        log.Fatalf("Could not connect to Docker: %v", err)
    }

    // Create a Docker network for the tests.
    network, err = pool.CreateNetwork("test-network")
    if err != nil {
        log.Fatalf("Could not create network: %v", err)
    }

    // Deploy the Redis container.
    redisResource, err := deployRedis(pool)
    if err != nil {
        log.Fatalf("Could not start resource: %v", err)
    }

    // Deploy the API container.
    apiResource, err := deployAPIContainer(pool)
    if err != nil {
        log.Fatalf("Could not start resource: %v", err)
    }

    resources := []*dockertest.Resource{
        redisResource,
        apiResource,
    }

    // Run the tests.
    exitCode := m.Run()

    // Exit with the appropriate code.
    err = TearDown(pool, resources)
    if err != nil {
        log.Fatalf("Could not purge resource: %v", err)
    }

    os.Exit(exitCode)
}

// deployRedis builds and runs the Redis container.
func deployRedis(pool *dockertest.Pool) (*dockertest.Resource, error) {
    resource, err := pool.RunWithOptions(&dockertest.RunOptions{
        Hostname:     "redis-container",
        Repository:   "redis",
        Tag:          "latest",
        ExposedPorts: []string{"6379"},
        PortBindings: map[docker.Port][]docker.PortBinding{
            "6379/tcp": {{HostIP: "", HostPort: "6379"}},
        },
        Networks: []*dockertest.Network{
            network,
        },
    })
    if err != nil {
        return nil, fmt.Errorf("could not start resource: %v", err)
    }

    // Ensure the Redis container is ready to accept connections.
    if err := pool.Retry(func() error {
        fmt.Println("Checking Redis connection...")
        db := redis.NewClient(&redis.Options{
            Addr:     "localhost:6379",
            Password: "",
            DB:       0,
        })

        _, err := db.Ping(context.Background()).Result()
        if err != nil {
            return err
        }

        defer db.Close()

        return nil
    }); err != nil {
        return nil, fmt.Errorf("could not connect to docker: %v", err)
    }

    return resource, nil
}

// TearDown purges the resources and removes the network.
func TearDown(pool *dockertest.Pool, resources []*dockertest.Resource) error {
    for _, resource := range resources {
        if err := pool.Purge(resource); err != nil {
            return fmt.Errorf("could not purge resource: %v", err)
        }
    }

    if err := pool.RemoveNetwork(network); err != nil {
        return fmt.Errorf("could not remove network: %v", err)
    }

    return nil
}

// deployAPIContainer builds and runs the API container.
func deployAPIContainer(pool *dockertest.Pool) (*dockertest.Resource, error) {
    // build and run the API container
    resource, err := pool.BuildAndRunWithBuildOptions(&dockertest.BuildOptions{
        ContextDir: "../",
        Dockerfile: "Dockerfile",
    }, &dockertest.RunOptions{
        Name:         "api-container",
        ExposedPorts: []string{"8080"},
        PortBindings: map[docker.Port][]docker.PortBinding{
            "8080": {{HostIP: "0.0.0.0", HostPort: "8080"}},
        },
        Networks: []*dockertest.Network{
            network,
        },
    })

    if err != nil {
        return nil, fmt.Errorf("could not start resource: %v", err)
    }

    // check if the API container is ready to accept connections
    if err = pool.Retry(func() error {
        fmt.Println("Checking API connection...")
        _, err := http.Get("http://localhost:8080/healthz")
        if err != nil {
            return err
        }

        return nil
    }); err != nil {
        return nil, fmt.Errorf("could not start resource: %v", err)
    }

    return resource, nil
}

Now we have the test suite ready and configured. We can write the actual tests in the e2e_test package.

Writing the Tests

The tests will be written in the e2e_test package. This package will be run as part of the TestMain function in the init_test.go file.

When this test is run locally, it will spin up the Redis and API containers, run the tests, and then tear down the containers. When this test is run in the pipeline, it will spin up the Redis and API containers, run the tests, and then tear down the containers.

package e2e_test

import (
    "net/http"
    "testing"
)

// TestHealthRoute tests the /healthz endpoint.
func TestHealthRoute(t *testing.T) {
    // create a get request to localhost:8080/healthz
    // check that the response status code is 200

    request, err := http.NewRequest("GET", "http://localhost:8080/healthz", nil)
    if err != nil {
        t.Fatalf("Could not create request: %v", err)
    }

    response, err := http.DefaultClient.Do(request)
    if err != nil {
        t.Fatalf("Could not make request: %v", err)
    }

    if response.StatusCode != http.StatusOK {
        t.Errorf("Expected status 200, got %d", response.StatusCode)
    }
}

Running the Tests

To run the tests locally, run the following command:

go test ./...

This will run the tests in the e2e_test package, which will spin up the Redis and API containers, run the tests, and then tear down the containers. One benefit of using dockertest is that it will automatically pull the Redis and API images if they are not already present on the local machine. Also, other tests in the project will be run as usual.

The TestMain function will be used only for running tests inside the e2e_test package.

If you have another test package that needs a similar setup/teardown logic, you can create a similar TestMain function in that package.

Conclusion

In this post, we looked at how to use dockertest to write automated tests for a Go application. We saw how to use dockertest to spin up a temporary Redis container and a temporary API container for our tests, then tear it down after the tests are finished.

The ory/dockertest package provides a powerful and convenient way to spin up Docker containers for testing dependencies. By handling the setup and teardown of containers in the background, it frees us to focus on writing robust integration tests for our Go applications. The full lifecycle testing enabled by dockertest increases confidence in releasing code to production. While it does require configuring tests to use containers, this overhead pays dividends in catching issues early. The dockertest documentation contains many more examples for testing various databases, queues, and services. Overall, dockertest is an invaluable tool for anyone looking to improve their automated testing workflows with Docker and Go.

References

ory/dockertest - https://github.com/ory/dockertest

go-e2e-test-blog - https://github.com/cksidharthan/go-e2e-test-blog

Starship Prompt: A Versatile and Customizable Command Prompt

The command prompt is your gateway to the terminal, and Starship takes it to the next level. Here's a closer look at why I prefer it:

Customizability Beyond Compare: Starship's configuration is based on a simple TOML file, making it incredibly approachable for customization. You can define not only the prompt's appearance but also its behavior. Want to display the current Git branch, Python version, and virtual environment? Starship can do that.

Language Detection: Starship can intelligently detect the programming language of your current project. For instance, if you're in a Python project directory, it will display a Python logo and provide information related to Python. This dynamic adaptation ensures your prompt is always context-aware.

Multi-Shell Compatibility: Starship isn't just for Zsh enthusiasts; it's designed to work across multiple shells. Whether you switch between Zsh, Bash, Fish, or even PowerShell, you can have a consistent and feature-rich prompt experience.

To install Starship, follow the instructions on their official website.

Supercharging Zsh with Plugins

Zsh's extensibility is one of its greatest strengths. With an array of plugins at your disposal, you can fine-tune your shell to perfection. Let's explore the plugins that enhance my Zsh experience:

Kubectl Plugin: Kubernetes is a cornerstone of modern containerized applications. The kubectl plugin provides auto-completions and suggestions for kubectl commands, streamlining your interaction with Kubernetes clusters.

zsh-autosuggestions: This plugin takes inspiration from modern text editors. As you type, it predicts and suggests commands based on your history and habits, reducing keystrokes and minimizing typos.

zsh-syntax-highlighting: Command syntax highlighting is more than just eye candy; it's a real-time error prevention tool. With this plugin, your commands are color-coded to highlight potential mistakes, helping you identify mistakes in commands as you type them in the terminal.

you-should-use: It's easy to forget the optimal command in a given situation. This plugin acts as your knowledgeable assistant, suggesting better alternatives to the commands you're using. It's like having a mentor in your terminal.

alias-finder: Managing aliases can become unwieldy, especially if you have many of them. The alias-finder plugin simplifies the process by allowing you to search for and discover your aliases effortlessly.

FZF: The Ultimate Fuzzy Finder

Traditional command history search (Ctrl + R) can be a hit or miss, especially as your history grows. Enter FZF (Fuzzy Finder), a remarkable tool for searching through your history and finding files:

Efficient and Intuitive Searching: FZF introduces a fuzzy search mechanism that understands typos and incomplete input. It prioritizes and ranks results based on relevance, making it an intuitive and efficient way to locate the command you need.

Versatile File Navigation: Beyond history, FZF extends its capabilities to file navigation. You can seamlessly locate and open files with just a few keystrokes, simplifying your interactions with your file system.

Customizable Keybindings: FZF's flexibility doesn't stop at search behavior. You can create custom keybindings to tailor the tool to your specific preferences and workflow.

Zoxide: Your Path to Effortless File Navigation

File navigation is a fundamental aspect of terminal usage, and Zoxide takes it to the next level:

Intelligent Path Suggestion: Zoxide's magic lies in its ability to predict the path you want to navigate to based on your usage patterns. This means less time spent manually traversing directories and more time focused on your tasks.

Interactive Directory Jumping: With Zoxide, changing directories becomes an interactive experience. A simple 'z' command followed by a search term takes you directly to your desired directory.

Cross-Shell Compatibility: Just like Starship, Zoxide is not confined to a single shell. You can use it with Zsh, Bash, Fish, and others, ensuring a consistent experience across different shell environments.

TIP: I created an alias for zoxide with alias cd=z in the ~/.zshrc file so that my workflow is unchanged :)

Conclusion

Your terminal is more than just a command line; it's your productivity hub. By configuring Zsh with Starship and essential plugins, embracing FZF for efficient searches, and adopting Zoxide for intuitive file navigation, you can transform your terminal into a powerhouse of productivity.

Experiment with these tools, fine-tune them to your liking and watch your terminal experience become more intuitive, efficient, and enjoyable. Whether you're a seasoned developer or just starting your journey, a well-crafted terminal setup can make a world of difference in your daily workflow.

Below is my basic .zshrc file :)

ZSH_THEME="frisk" # set by `omz`

# Path to your oh-my-zsh installation.
export ZSH="$HOME/.oh-my-zsh"

fpath+=${ZSH_CUSTOM:-${ZSH:-~/.oh-my-zsh}/custom}/plugins/zsh-completions/src
source $ZSH/oh-my-zsh.sh

export GOROOT="/opt/homebrew/Cellar/go/1.21.0/libexec"
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin

# Startship Config
eval "$(starship init zsh)"
export STARSHIP_CONFIG=~/.config/starship/starship.toml

# Zoxide
eval "$(zoxide init zsh)"
alias cd="z"

plugins=(
        kubectl
        git
        alias-finder
        history
        zsh-autosuggestions
        zsh-syntax-highlighting
        you-should-use
)

# Syntax Highlighting
source /opt/homebrew/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
export ZSH_HIGHLIGHT_HIGHLIGHTERS_DIR=/opt/homebrew/share/zsh-syntax-highlighting/highlighters

# Example aliases
alias zshconfig="mate ~/.zshrc"
alias ohmyzsh="mate ~/.oh-my-zsh"
alias home="cd ~"

export BAT_THEME="Monokai Extended Bright"

[ -f ~/.fzf.zsh ] && source ~/.fzf.zsh

# Kubectl alias
alias k="kubectl"

# Kubernetes Autocompletions
autoload -Uz compinit
compinit
source <(kubectl completion zsh)

# Go
alias makedeps="go mod download && go mod tidy && go mod verify && go mod vendor"

source /Users/sid/.docker/init-zsh.sh || true # Added by Docker Desktop

Below is the starship.toml file

#### starship.toml file

# Don't print a new line at the start of the prompt
add_newline = true

# Replace the "❯" symbol in the prompt with "➜"
[character]      # The name of the module we are configuring is "character"
error_symbol = "✗"

# Disable the package module, hiding it from the prompt completely
[package]
disabled = true

[battery]
disabled = false
full_symbol = "🔋 "
charging_symbol = "⚡️ "

[[battery.display]]  # "bold red" style when capacity is between 0% and 10%
threshold = 10
style = "bold red"

[[battery.display]]  # "bold yellow" style when capacity is between 10% and 30%
threshold = 30
style = "bold yellow"

[[battery.display]]
threshold = 70
style = "bold blue"

[[battery.display]]
threshold = 100
style = "bold green"

[directory]
# truncation_length = 4

# docker
[docker_context]
symbol = "🐋 "
disabled = false

# git
[git_commit]
commit_hash_length = 6

[git_status]
style="dimmed green"
up_to_date = "[✓](green)"
staged = '[++\($count\)](green)'

# hostname
[hostname]
ssh_only = false
prefix = "⟪"
suffix = "⟫"
trim_at = ".companyname.com"
disabled = true

# kubernetes
[kubernetes]
format = 'on [⎈ ($cluster) ](#FFA500)'
style = "dimmed green"
disabled = false

Faced with these challenges, we embarked on a journey to find an alternative documentation solution that would allow us to continue using markdown for documentation, offer a painless migration process, and enhance the overall documentation experience. Our search led us to Vitepress, a static site generator that proved to be the perfect fit for our needs. In this blog post, we'll delve into our transition from Azure Git Wiki to Vitepress, highlighting the reasons behind our decision and the steps we took to make it happen.

Why We Sought an Alternative

Performance Issues: The primary motivation behind our quest for an alternative documentation platform was the performance issues we encountered with Azure Git Wiki. The website's slowness and occasional unresponsiveness became major roadblocks to efficient documentation creation and access.

Slow Search: The search functionality in Azure Git Wiki was frustratingly slow, making it challenging to find the information we needed quickly. This hindered our ability to navigate and explore the documentation effectively.

Lack of Contributor Engagement: The combination of performance issues and slow search functionality led to a lack of enthusiasm among team members for contributing to the documentation. Valuable insights and findings remained undocumented, which was detrimental to our knowledge-sharing efforts.

Exploring Alternative Documentation Solutions

In our search for an alternative, we explored various static site generation tools, including Vuepress, Hugo, and Docusaurus etc., However, Vitepress emerged as the standout choice for several reasons:

Folder Structure: Vitepress offered an intuitive and easy-to-understand folder structure for organizing our documentation. This simplicity was crucial to ensuring that our team could easily navigate and contribute to the documentation without getting lost in complex project structures.

Search Functionality: Vitepress came with built-in search functionality, addressing one of our primary pain points. This feature greatly improved the speed and efficiency of searching and navigating through our documentation.

Community Support: Vitepress had an active community, with plugins and themes readily available. This allowed us to enhance our documentation with features like Mermaid diagrams and customize the look and feel of our documentation site effortlessly.

Migration Process

Our migration from Azure Git Wiki to Vitepress was carefully planned and executed. Here are the key steps we followed:

Create a New "Docs" Folder: We started by creating a new folder named docs and moved our existing documentation into this folder while maintaining the original folder structure.

Configure Vitepress: We made the necessary configuration changes in the config.js file located inside the .vitepress folder. These changes included setting up the navigation structure, theme customization, and enabling the built-in search functionality.

Local Testing: Before deploying the documentation, we ran the site locally to ensure that everything worked as expected. This step allowed us to catch any issues or discrepancies in the migration process.

CI/CD Pipeline: To ensure continuous deployment, we configured a CI/CD pipeline in our development environment. This automated the process of publishing changes to our Vitepress documentation site whenever updates were pushed to the repository.

Benefits of the Transition

The transition to Vitepress brought about several notable benefits:

Speed and Performance: Vitepress proved to be significantly faster and more responsive than Azure Git Wiki, eliminating the frustrations we previously experienced.

Improved Search: The built-in search functionality in Vitepress drastically improved the speed and accuracy of searching within our documentation, making it a joy to use.

Enhanced Contribution: With the improved performance and user-friendly interface of Vitepress, team members were more willing to contribute actively to the documentation, leading to a more comprehensive and up-to-date knowledge base.

Conclusion

In summary, our journey from Azure Git Wiki to Vitepress was driven by the need for faster, more responsive documentation that would encourage team collaboration and knowledge sharing. Vitepress not only met but exceeded our expectations with its simplicity, performance, and search capabilities. The painless migration process ensured a smooth transition, and the positive impact on our team's productivity and engagement was immediately evident.

By making the switch to Vitepress, we've not only improved our documentation workflow but also created a more conducive environment for learning and sharing within our development team. The power of a well-structured and efficient documentation system cannot be understated, and with Vitepress, we've found a solution that fits our needs perfectly.

The Scenario

Imagine we have a long-running Go program that needs to execute some logic periodically. For example, it could be polling a server for updates or performing other scheduled tasks. In such a scenario, we want the program to:

1. Execute the logic every minute.

2. Gracefully shut down when it receives a SIGTERM or SIGINT signal.

A Ticker Goroutine

Go's time package provides a convenient Ticker object that can fire events on a channel at regular intervals. Here's how we can create a ticker that fires every minute:

ticker := time.NewTicker(time.Minute)

Additionally, we need a channel to receive OS signals:

signalChannel := make(chan os.Signal, 1)
signal.Notify(signalChannel, syscall.SIGTERM, syscall.SIGINT)

With the ticker and signal channel set up, we can proceed to launch a goroutine that uses them:

go func(ticker *time.Ticker) {
    for {
        select {
        case <-ticker.C:
            // Run required logic every minute
            fmt.Println("Ticker was fired!")
            ExampleLogic()
        case <-signalChannel:
            // remove tempfiles, close database connections etc.,
            // Shutdown goroutine
            os.Exit(0)
        }
    }
}(ticker)

In the code above, we define an anonymous function that accepts the ticker as a parameter. Inside this goroutine:

• The select block concurrently waits on the ticker and signal channel.

• When the ticker fires, it triggers the execution of the recurring logic.

• If a signal (SIGTERM or SIGINT) is received, the goroutine proceeds to handle the graceful shutdown logic.

Putting it All Together

Our main function simply needs to start the ticker goroutine:

package main

import (
    "fmt"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    ticker := time.NewTicker(time.Minute)
    TriggerGoroutine(ticker)
    time.Sleep(10 * time.Minute)
}

func TriggerGoroutine(ticker *time.Ticker) {

    signalChannel := make(chan os.Signal, 1)
    signal.Notify(signalChannel, syscall.SIGTERM, syscall.SIGINT)

    go func(ticker *time.Ticker) {
    for {
        select {
        case <-ticker.C:
            // Run required logic every minute
            fmt.Println("Ticker was fired!")
            ExampleLogic()
        case <-signalChannel:
            // remove tempfiles, close database connections etc.,
            // Shutdown goroutine
            fmt.Println("Got signal, exiting...")
            os.Exit(0)
        }
    }
    }(ticker)
}

func ExampleLogic() {
    fmt.Println("ExampleLogic was called!")
}

With this setup, the goroutine will execute every minute until the program is terminated with CTRL-C or a kill signal. When a shutdown signal comes in, the goroutine will handle the graceful shutdown logic before the program exits.

This pattern provides a clean and efficient way to build periodically executing processes that respond to common signals, ensuring the program behaves predictably and gracefully.

Feel free to reach out if you have any questions or need further clarification!

Open Policy Agent (OPA) provides a unified approach to manage authorization policies separate from application code. We can use the same OPA file and use it for applications written in multiple languages, but for the sake of this blog, we'll integrate it with a Golang application.

Introduction to OPA

OPA is an open-source policy engine that evaluates policies to make decisions about access control, configuration validation, quota management and more.

OPA decouples policy decisions from policy enforcement. Developers define policies using OPA’s declarative language Rego. These policies get enforced across infrastructure like API gateways, Kubernetes, CI/CD pipelines etc.

Creating OPA Policy

OPA policies are written in the Rego language. Rego is a declarative language designed for specifying policy rules concisely.

Some key concepts in Rego:

• Rules - These define relations between entities. For example:

    allow {
      input.subject.clearance == "secret"
      input.action == "GET"
    }
  

• Packages - Related rules can be grouped into packages:

    package authz
  
    allow {...}
  
    deny {...}
  

• Input - This contains the request details like subject, action etc.

• Query - Executing a rule returns true/false for the query.

Let's look at an example policy implementing role-based access control:

# filename - auth.rego
package authz

import future.keywords

default allow = false

allow {
  input.role == "admin"
  access_groups = ["write", "read"]
  input.access in access_groups
}

allow {
  input.role == "user"
  access_groups = ["read"]
  input.access in access_groups
}

This rego file allows the admin to have read and write access but will restrict the user from only having read access.

Testing OPA Policies

We can test the OPA file that we have created above with a test file like below

# filename - auth_test.rego
package authz

# Test allow rule for admin
test_allow_admin_write {
  allow with input as {"role": "admin", "access": "write"}
}

test_allow_admin_read {
  allow with input as {"role": "admin", "access": "read"}
}

# Test allow rule for user
test_allow_user_read {
  allow with input as {"role": "user", "access": "read"}
}

test_deny_user_write {
  not allow with input as {"role": "user", "access": "write"}
}

# Test default deny
test_default_deny {
  not allow with input as {"role": "unknown", "access": "something"}
}

To see if the tests are passing and the rego file is working perfectly, we can run the tests using the below command.

NOTE: You will need to install the OPA tool to run the commands. Click here to view the install docs

$❯ opa test auth.rego auth_test.rego --verbose

auth_test.rego:
data.authz.test_allow_admin_write: PASS (913.375µs)
data.authz.test_allow_admin_read: PASS (133.875µs)
data.authz.test_allow_user_read: PASS (194.625µs)
data.authz.test_deny_user_write: PASS (186.958µs)
data.authz.test_default_deny: PASS (120.333µs)
--------------------------------------------------------------------------------
PASS: 5/5

You can also check the test coverage of the OPA file using the below command

$❯ opa test auth.rego auth_test.rego --coverage

... start of the output
  "covered_lines": 19,
  "not_covered_lines": 0,
  "coverage": 100
....

Integrating OPA with Golang Gin Application

Creating Gin Middleware

To Integrate OPA with Golang Gin application, we will need to add it as middleware as below

func OpaMiddlware() gin.HandlerFunc {
    // open rego file
    authzFile, err := os.Open("auth.rego")
    if err != nil {
       log.Fatalf("error opening file: %v", err)
    }
    defer authzFile.Close()

    // read rego file
    module, err := io.ReadAll(authzFile)
    if err != nil {
       log.Fatalf("error reading file: %v", err)
    }

    // return middleware
    return func(c *gin.Context) {
       // prepare rego query
       query, err := rego.New(
          rego.Query("data.authz.allow"),
          rego.Module("authz.rego", string(module)),
       ).PrepareForEval(c)
       if err != nil {
          log.Printf("error preparing query: %v\n", err)
       }

       // evaluate rego query by supplying values extracted from header
       result, err := query.Eval(context.Background(), rego.EvalInput(map[string]interface{}{
          "role": c.Request.Header.Get("role"),
          "access":   c.Request.Header.Get("access"),
       }))
       if err != nil {
          log.Printf("error evaluating query: %v\n", err)
       }

       // check if the user is allowed to access the resource
       if result[0].Expressions[0].Value == true {
          c.Next()
          return
       } else {
          c.JSON(http.StatusForbidden, gin.H{
             "message": "access forbidden",
          })
          c.Abort()
          return
       }
    }
}

This middleware reads the auth.rego file and prepare the rego query. This function returns a handler function that will run the opa policy for all the incoming requests by supplying headers, before forwarding the request to appropriate endpoints.

Integrating Auth Middleware with Gin Router

We'll instruct Gin to attach the middleware to the router by calling the router.Use function like below

func main() {
    r := gin.Default()
    r.Use(OpaMiddlware()) // we are attaching the auth middleware here

    r.GET("/ping", func(c *gin.Context) {
        // do some logic with header
        c.JSON(http.StatusOK, gin.H{
            "message": "pong",
        })
    })
    r.Run()
}

Running the Go API

When we run the golang application and hit the /ping endpoint we get the output as below. (added print header statements in logs for easy understanding)

$❯ go run main.go

.......
[GIN-debug] Listening and serving HTTP on :8080
access: read
role: writer
[GIN] 2023/09/06 - 21:42:00 | 403 |    5.589708ms |       127.0.0.1 | GET      "/ping"
access: read
role: admin
[GIN] 2023/09/06 - 21:42:10 | 200 |    1.925834ms |       127.0.0.1 | GET      "/ping"
access: read
role: user
[GIN] 2023/09/06 - 21:42:15 | 200 |       775.5µs |       127.0.0.1 | GET      "/ping"
access: write
role: user
[GIN] 2023/09/06 - 21:42:20 | 403 |    3.746209ms |       127.0.0.1 | GET      "/ping"

From the console logs we see that when invalid requests are sent, we get 403 errors as per the policy and when valid requests are sent, we get back the response with the status code 200

To be exact, The middleware extracts the subject role and request method, queries OPA, and denies access if the policy evaluation fails.

Conclusion

In this post, we looked at using Open Policy Agent to externalize authorization logic from a Golang application. OPA provides a unified way to manage access policies across services using its declarative language Rego.

We saw how to:

• Authorize Rego policies for enforcing role-based access control

• Test policies thoroughly using OPA's built-in test framework

• Integrate OPA with a Golang API server using middleware

• Evaluate policies on each request to make access decisions

OPA integrates well with infrastructures like Kubernetes, allowing consistent policy enforcement across large distributed environments.

Using OPA results in more maintainable applications by separating policy code from business logic. Authorization policies can be modified independently without changing backend services. OPA provides a scalable way to manage fine-grained access control that evolves with application needs.

Hope this gives a good overview of securing Golang apps with OPA. :)

Github Link - https://github.com/cksidharthan/opa-blog

Introducing Hadolint

Hadolint is an open-source linter specifically designed to analyze Dockerfiles and identify issues, inconsistencies, and potential problems in the Dockerfiles. The primary goal of Hadolint is to enforce best practices and guidelines for writing Dockerfiles, ensuring that the resulting containers are well-structured, secure, and optimized.

The Importance of Linting Dockerfiles

Linting is a software development practice that involves using automated tools to analyze source code or configuration files for potential issues, deviations from best practices, and coding standards violations. Linting is particularly crucial in the context of Dockerfiles for several reasons:

1. Security: Docker containers are widely used in production environments, making security a top concern. A poorly written Dockerfile can inadvertently introduce security vulnerabilities, such as running processes as root or using outdated packages. Hadolint helps catch these security risks early in the development process.

2. Efficiency: Optimized Dockerfiles lead to smaller image sizes and faster build times. Hadolint suggests improvements that can help streamline the build process, reducing the overall resource consumption and making the deployment pipeline more efficient.

3. Consistency: Dockerfiles are often collaboratively written by different team members. Ensuring consistent formatting and adherence to best practices across the team is challenging. Hadolint acts as an impartial judge, helping maintain a uniform style and structure.

Features and Benefits

1. Static Analysis:

Hadolint performs static analysis on Dockerfiles without actually executing it. This allows it to identify issues and potential problems without the need for running a container. This is particularly important for catching problems early in the development cycle.

2. Customizable Rules:

Hadolint comes with a set of default rules that cover a wide range of common best practices. However, it also allows you to customize or extend these rules to align with your organization's specific requirements. This flexibility ensures that the linter remains adaptable to different project needs.

3. Integration with CI/CD Pipelines:

Hadolint can be seamlessly integrated into Continuous Integration (CI) and Continuous Deployment (CD) pipelines. By incorporating Hadolint checks into your pipeline, you can automatically detect and prevent Dockerfile issues before they reach the production environment.

4. Dockerfile Styles:

Different projects might have different styles and conventions for writing Dockerfiles. Hadolint provides support for multiple Dockerfile styles, allowing you to tailor the linting process to match the coding standards of your project.

5. Docker Image Integration:

Hadolint can be run within a Docker container itself, ensuring that the linting environment matches the runtime environment. This prevents issues arising due to differences between the linting environment and the actual build environment.

Getting Started with Hadolint

Using Hadolint is straightforward and can be done in a few simple steps:

1. Installation:

   Hadolint can be installed using various methods, including downloading pre-built binaries, using package managers like apt or brew, or running it as a Docker container.

2. Basic Usage:

   Running Hadolint on a Dockerfile is as simple as executing the following command:

    hadolint Dockerfile
   

   Hadolint will analyze the specified Dockerfile and provide feedback on any issues it finds.

3. Customizing Rules:

   You can create a configuration file (usually named .hadolint.yaml) to customize Hadolint's rules. This allows you to enable, disable, or modify rules to suit your project's needs.

4. Integration with CI/CD:

   To integrate Hadolint into your CI/CD pipeline, you can add a step that runs Hadolint on your Dockerfiles as part of the build process. This ensures that Dockerfile issues are caught before the image is pushed to a registry or deployed.

Common Hadolint Rules

Hadolint comes with a comprehensive set of rules, each targeting specific aspects of Dockerfile development. Some common rules include:

• DL3000: Use absolute WORKDIR.

• DL3003: Use WORKDIR to switch to a directory.

• DL3007: Use --no-install-recommends.

• DL4000: Avoid additional packages by specifying --no-install-recommends.

• DL3008: Pin versions in apt-get install.

• DL3015: Avoid additional packages by specifying --no-install-recommends.

Final Thoughts

Dockerfiles are the backbone of containerization, and writing them correctly is essential for building secure, efficient, and reliable containers. Hadolint serves as a powerful tool to assist developers and DevOps engineers in this endeavor. By automating the process of identifying Dockerfile issues, Hadolint contributes to a smoother development workflow and ultimately helps in delivering high-quality containerized applications.

As containerization continues to gain momentum in the software industry, tools like Hadolint play a vital role in ensuring that best practices are followed, security is maintained, and the containerization process remains efficient. By incorporating Hadolint into your development process, you can elevate the quality of your Dockerfiles and, consequently, the containers they produce.

Single Build Stage

Traditionally, a Dockerfile for a Go application would look something like this:

FROM golang:1.21.0-alpine3.17

WORKDIR /app
COPY . .

RUN go build -o goApp .

CMD ["/app/goApp"]

In this approach, the Dockerfile uses the golang:alpine image as the base image to compile the Go application. The code is copied into the image, and then the go build command is executed to build the application. Finally, the CMD instruction specifies the command to run the application.

However, this approach has a downside. It includes unnecessary bloat from the Go compiler, build tools, and dependencies in the final image, resulting in a larger image size. For a simple Go application, the image size can be over 300MB, which is not ideal for production deployments.

Multi-Stage Build

To address this issue, multi-stage Docker builds can be used. With multi-stage builds, the Dockerfile is divided into multiple stages, each with its own base image and set of instructions. The final image only includes the necessary artifacts from the build stage, resulting in a smaller and more optimized image.

Here is an example of a multi-stage Dockerfile for a Go application:

# Build stage
FROM golang:1.21.0-alpine3.17 AS build
WORKDIR /app
COPY . .
RUN go build -o main .

# Run stage  
FROM alpine:3.18.3
WORKDIR /app
COPY --from=build /app/main .
CMD ["/app/main"]

In this example, the Dockerfile consists of two stages Build Stage and Run stage

Build stage

The build stage uses the golang alpine image as the base image and performs the compilation of the Go application, If some external dependencies are required for build, it can be added in the build stage

Run stage

The resulting binary is then copied into the run stage, which uses the alpine image as the base image. The final image only includes the built binary and its dependencies, resulting in a significantly smaller image size.

Benefits of Multi-stage docker build

With multi-stage builds, the runtime image size can be reduced to as little as 15MB, compared to over 300MB in the traditional approach. This reduction in image size has several benefits, including faster image pull and deployment times, reduced storage requirements, and improved overall performance.

Further Optimization

For even further optimization, the Alpine runtime stage can be replaced with a scratch image. The scratch image is a special Docker image that is completely empty, with no operating system or libraries included. This approach further reduces the image size to around 10MB. However, it comes at the cost of losing the tools and package manager provided by the Alpine base image.

Here is an example of a Dockerfile using the scratch image:

# Build stage
FROM golang:1.21.0-alpine3.17 AS build
WORKDIR /app
COPY . .
RUN go build -o main .

# Run stage  
FROM scratch
WORKDIR /app
COPY --from=build /app/main .
CMD ["/app/main"]

Summary

In summary, multi-stage Docker builds are a powerful technique for optimizing Dockerfiles, especially for Go applications. By separating the build tools and dependencies from the runtime content, multi-stage builds significantly reduce the final image size. This reduction in image size has numerous benefits, including improved performance, faster deployment times, and reduced storage requirements.

When using multi-stage builds for Go applications, it is possible to achieve a final image size that is over 90% smaller compared to a single-stage build. Additionally, by using the scratch image as the base image, developers can further minimize the container footprint, resulting in a truly minimal runtime image.

Overall, multi-stage Docker builds are a valuable tool in the developer's toolbox for creating efficient and optimized Docker images. They are particularly beneficial for compiled languages like Go, where the elimination of unnecessary build artifacts can have a significant impact on the final image size. Give multi-stage builds a try for your next Go application and experience the benefits of smaller and more efficient Docker images.

In this post, we'll explore how GoFakeIt can effortlessly generate realistic fake data to fuel your tests.

Why Use a Mock Data Library?

Mock data is crucial for writing robust unit and integration tests:

• Allows testing logic and behavior independently from databases and services.

• Avoids polluting your dev environment with fake data.

• Data can be randomly generated for complete coverage.

• Speeds up test execution since no dependencies are required.

Manually defining sample data for every test is cumbersome. The data is often stale and lacks variety.

A dedicated mock data library like GoFakeIt solves these issues by programmatically generating fresh data on the fly.

Overview of GoFakeIt for Go

GoFakeIt is a popular mock data library ported to Go. It provides data generators for common scenarios like names, addresses, companies, users, transactions, and more.

Some key features:

• Over 100 data provider methods (and growing).

• Generates random, unique data every time.

• Localization support for different locales.

• Easy to extend with custom data providers.

• Struct and slice population helpers.

• Thorough test coverage.

• MIT license.

The GoFakeIt repo is actively maintained by Brian Voe and has over 500 contributors.

Installing GoFakeIt

To install GoFakeIt, simply run:

go get github.com/brianvoe/gofakeit/v6

Then import it:

import "github.com/brianvoe/gofakeit/v6"

Let's look at how to use it.

Generating Random Fake Data

The basic pattern is to call GoFakeIt data provider methods:

name := gofakeit.Name()
email := gofakeit.Email()
color := gofakeit.Color()

This gives you a fresh, randomized value for each call.

Some handy methods include:

• Name() - Full fake name

• Email() - Fake email address

• Phone() - Random phone number

• Address() - Fictional street address

• BS() - Random bullshit (for filler text!)

• Company() - Fake company name

• Sentence() - Random sentence

• paragraphs() - Multiple paragraphs of text

Check the godoc for the full set of data providers available.

Generating Structured Data

For more structured data, GoFakeIt can populate:

• Structs

• Slices

• Maps

• Arrays

With random values.

NOTE: `fake` struct tags have to be present in the struct for GoFakeIt to produce correct data.

type User  struct {
    ID    int    `fake:"{number:1,100}"`
    Name  string `fake:"{firstname}"`
    Email string `fake:"{email}"`
}

var user User
gofakeit.Struct(&user)

Now user will contain random fake data!

To populate a slice, provide the slice and the desired length:

var users []User
gofakeit.Slice(&users)

This makes mocking datasets a breeze.

Seeding Randomness

By default, each call generates unpredictable data.

To generate repeatable data, seed the RNG:

gofakeit.Seed(1234) // any int64 number

// Repeatable results now
name1 := gofakeit.Name() 
name2 := gofakeit.Name()

Useful for regenerating the same fake datasets.

Examples and Recipes

GoFakeIt is handy for mocking all sorts of data:

• User accounts

• Blog posts and comments

• Product listings

• Transaction histories

• File uploads

• API responses

Some examples:

package main

import (
    "fmt"
    "github.com/brianvoe/gofakeit/v6"
)

type User struct {
    ID    int    `fake:"{number:1,100}"`
    Name  string `fake:"{firstname}"`
    Email string `fake:"{email}"`
}

type Product struct {
    ID          int    `fake:"{number:1,100}"`
    Name        string `fake:"{firstname}"`
    Description string `fake:"{sentence:30}"`
    Price       int    `fake:"{number:1,100}"`
}

func main() {
    // Fake users
    var user User
    gofakeit.Struct(&user)

    fmt.Print(user)
    // Fake product listings

    var product Product
    gofakeit.Struct(&product)

    fmt.Print(product)

    // Fake upload files
    const numFiles = 5
    var files []string
    for i := 0; i < numFiles; i++ {
        files = append(files, gofakeit.ImageURL(50, 50))
    }

    fmt.Print(files)
}

The possibilities are endless!

Conclusion

GoFakeIt for Go makes generating mock data for tests almost too easy. No more hand-rolled stubs that go stale!

Some key benefits:

• Huge time savings compared to manual mocking.

• Randomized data that avoids stale samples.

• Localization for region-specific data.

• Extensible via custom providers.

• Helpers for populating structs and slices.

Proper mock data is crucial for writing robust Go tests. GoFakeIt delivers with a robust set of data providers in an easy-to-use package.

I highly recommend installing GoFakeIt in your next Go project. Browse the full docs and examples to see all it can do.

Have you used GoFakeIt in your Go code? Are there any other mock data tools you recommend? Let me know in the comments!

Defining a Cache Interface

First, let's define an interface specifying cache capabilities:

type Cache interface {
  Get(key string) interface{}
  Set(key string, value interface{})
}

This Cache interface has two methods - Get to retrieve a cached value by key, and Set to store a key-value pair.

By defining an interface, we decouple the cache usage from a specific implementation. Any cache library that implements these methods satisfies the interface.

A Simple Memory Cache

Let's implement a simple in-memory cache conforming to the interface:

type InMemoryCache struct {
  store map[string]interface{} 
}

func (c *InMemoryCache) Get(key string) interface{} {
  return c.store[key] 
}

func (c *InMemoryCache) Set(key string, value interface{}) {
  c.store[key] = value
}

The InMemoryCache uses a Go map to store entries in memory. It implements the Get and Set methods to manage entries in the map.

Using the Cache

We can now easily use the cache:

cache := InMemoryCache{make(map[string]interface{})}

cache.Set("foo", "bar")

value := cache.Get("foo") // "bar"

The interface allows us to call Set and Get without worrying about the implementation.

Swapping Cache Implementations

Now let's say we want to use Redis instead of in-memory. We can create a RedisCache implementing the same interface:

type RedisCache struct {
  client *redis.Client 
}

// Redis Get/Set implementation

And swap it in:

cache := RedisCache{redisClient} 

cache.Set("foo","bar") // Now uses Redis

The client code remains unchanged. This demonstrates the flexibility provided by interfaces.

Benefits of Interface-based Caching

Using interface-based caching gives several benefits:

• Decoupling - Client code isn't coupled to a specific cache library.

• Maintainability - The cache implementation can be changed without modifying the client code.

• Testability - Caches can be stubbed or mocked for testing.

• Reusability - Generic cache interface enables writing reusable caching logic.

Summary

Interfaces in Go helm in building flexible libraries and applications. Defining simple interfaces makes code more:

• Modular - Different implementations can be plugged in.

• Extensible - New implementations can be added without disruption.

• Maintainable - Components can be swapped for easy maintenance.

• Testable - Components can be stubbed and mocked.

By providing powerful abstraction with minimal overhead, interfaces are invaluable in Go for creating loosely coupled and scalable systems.

Enter Postbot, your revolutionary AI-powered Postman API testing assistant. Postbot is set to transform the way developers approach API testing, making the process faster, more accurate, and downright effortless. Gone are the days of manually testing each endpoint and meticulously recording results; Postbot takes the reins, unleashing the true potential of automation and artificial intelligence.

In this blog post, we'll delve into the world of Postbot, exploring its innovative features, how it streamlines API testing, and how it can empower developers to ship high-quality applications with confidence.

Automated Test case generation

When you integrate Postbot into your Postman workflow, it instantly scans and analyzes your API endpoints, understanding their structure, expected behavior, and potential inputs and outputs. This deep understanding is made possible by Postbot's sophisticated AI engine, which is trained on vast amounts of data and possesses a comprehensive knowledge of API testing best practices.

Once Postbot has gathered sufficient information about your APIs, it proceeds to automatically generate test cases based on the insights gained from its analysis.

Steps

• Open your Postman collection and click on the more actions icon and click on Generate tests.

• This is open up a tab that lists all the requests in a table. Now click on the Generate tests in the top right. This will trigger PostBot, which will in turn analyse all the requests in your collection and generate test cases for you.

• You can inspect the generated test cases, delete unwanted ones and also save the changes, by clicking on the save tests button at the top right.

Generating Test cases for Individual requests

In addition to generating test cases for the entire collection, you can also generate test cases for individual API requests. To do this, navigate to the tests tab of a request and click on Script with Postbot option. It opens a ChatGPT-like prompt window, where you can ask Postbot to write test cases, visualize responses, fix tests etc.,

Conclusion

Postbot represents a significant step forward in the world of API testing. By harnessing the power of AI, Postman has created a powerful assistant capable of streamlining API testing workflows, improving efficiency, and ensuring higher API quality. While there may be some challenges to overcome, the benefits it offers to development teams make it a compelling tool to consider integrating into your API testing arsenal. As AI technology continues to advance, we can expect even more intelligent and capable AI assistants like Postbot to reshape the future of software testing and development.

What are Receivers

In Go, a receiver is a parameter of a method that binds the method to a specific type. This is similar to what is commonly known as "this" or "self" in other programming languages. By using a receiver, you can define methods that operate on instances of the associated type. Receivers can be applied to both value and pointer types.

Value Receivers

A value receiver is a method that operates on a copy of the instance of the associated type. When you define a method with a value receiver, any modifications made to the receiver inside the method will not affect the original instance. Value receivers are represented by using the type without an asterisk (e.g., func (v MyType) MethodName()).

Here's an example of a value receiver method in Go:

type Circle struct {
    radius float64
}

func (c Circle) Area() float64 {
    return 3.14 * c.radius * c.radius
}

Pointer Receivers

A pointer receiver is a method that operates directly on the instance of the associated type. When you define a method with a pointer receiver, any modifications made to the receiver inside the method will directly affect the original instance. Pointer receivers are represented by using the type with an asterisk (e.g., func (p *MyType) MethodName()).

Here's an example of a pointer receiver method in Go

type Counter struct {
    count int
}

func (c *Counter) Increment() {
    c.count++
}

Choosing Between Value and Pointer Receivers

The choice between value and pointer receivers depends on the use case and the behavior you want to achieve. Here are some considerations:

1. Value Receivers:

   • Use value receivers when the method doesn't need to modify the instance's state and operates purely on a copy of the instance.

   • Value receivers are ideal for methods that are read-only and don't mutate the internal state of the type.

2. Pointer Receivers:

   • Use pointer receivers when the method needs to modify the instance's state directly.

   • Pointer receivers are essential when you want to modify the underlying state of a struct or any other type.

Performance Implications

There is a performance difference between value and pointer receivers. When you use a value receiver, a copy of the instance is made for the method to operate on, which can lead to more memory usage and potentially slower execution. On the other hand, pointer receivers directly operate on the instance, which is more memory-efficient and can result in faster execution, especially for large structs.

Guidelines for Receivers

• Use value receivers for read-only methods that don't modify the instance's state.

• Use pointer receivers when you need to modify the instance's state directly or when the method operates on a large struct to avoid unnecessary copying.

• For consistency, consider using either value or pointer receivers consistently across methods of a type.

Conclusion

In Go, value and pointer receivers provide a way to define methods that operate on instances of a type. Value receivers make copies of the instance to operate on, while pointer receivers directly operate on the instance. Choosing the right receiver type depends on the use case, performance considerations, and whether you need to modify the instance's state. Understanding the differences between value and pointer receivers is essential for writing efficient and correct Go programs.

In this blog post, we've explored the concept of value and pointer receivers in Go, their characteristics, use cases, and the implications of choosing one over the other. I hope this blog post has shed light on the significance of value and pointer receivers in Go, enabling you to make informed decisions while designing methods for your custom types.

In this blog post, we will explore the differences between new() and make() functions and understand when and how to use them effectively.

new() and make() Functions

Both new() and make() are built-in functions in Go, used to allocate memory. However, they are used for different data types and scenarios:

1. new() function:

   • new() is used to allocate memory for value types (e.g., integers, floats, structs) and returns a pointer to the newly allocated zeroed value.

   • It takes a single argument, which is a type, and returns a pointer to that type.

2. make() function:

   • make() is used to create and initialize slices, maps, and channels, which are reference types in Go.

   • It takes two or three arguments, depending on the type, and returns an initialized (not zeroed) value ready for use.

Understanding new() Function

The syntax of the new() function is straightforward as shown below.

func new(Type) *Type

Here, Type represents the type of the value we want to allocate memory for. Let's see an example of how to use new()

In this example, we create a new instance of the Person struct using new() and then assign values to its fields using the pointer.

package main

import "fmt"

type Person struct {
    Name string
    Age  int
}

func main() {
    // Using new() to allocate memory for a Person struct
    p := new(Person)

    fmt.Printf("%T\n", p)

    // Accessing struct fields using the pointer
    p.Name = "Alice"
    p.Age = 30

    // Displaying the values
    fmt.Println("Name:", p.Name)
    fmt.Println("Age:", p.Age)
}

This program will produce an output as shown below

> go run main.go
*main.Person
Name: Alice
Age: 30

Understanding make() Function

The syntax of the make() function varies depending on the type it is used with

For Slices

func make([]Type, len, cap) []Type

• Type: The type of elements the slice will hold.

• len: The initial length of the slice.

• cap: The capacity of the slice, which is optional and used to specify the underlying array's capacity. If not provided, it defaults to the same value as the length.

Example of creating a slice using make():

package main

import "fmt"

func main() {
    // Using make() to create a slice of integers
    numbers := make([]int, 5, 10)

    // Displaying the slice's length, capacity, and values
    fmt.Println("Length:", len(numbers))
    fmt.Println("Capacity:", cap(numbers))
    fmt.Println("Values:", numbers)

    // Using make() to create a slice of integers
    numbersWithoutOptional := make([]int, 5)

    // Displaying the slice's length, capacity, and values
    fmt.Println("Length:", len(numbersWithoutOptional))
    fmt.Println("Capacity:", cap(numbersWithoutOptional))
    fmt.Println("Values:", numbersWithoutOptional)
}

This program will produce an output as below

> go run main.go
Length: 5
Capacity: 10
Values: [0 0 0 0 0]
Length: 5
Capacity: 5
Values: [0 0 0 0 0]

For Maps

func make(map[KeyType]ValueType, initialCapacity int) map[KeyType]ValueType

• KeyType: The type of keys in the map.

• ValueType: The type of values associated with the keys.

• initialCapacity: The initial capacity of the map. This is optional but can be used to optimize performance when the number of elements is known in advance.

Example of creating a map using make()

package main

import "fmt"

func main() {
    // Using make() to create a map of string keys and int values
    scores := make(map[string]int)

    // Adding values to the map
    scores["Alice"] = 95
    scores["Bob"] = 87

    // Displaying the map
    fmt.Println("Scores:", scores)
}

> go run main.go
Scores: map[Alice:95 Bob:87]

For Channels

func make(chan Type, capacity int) chan Type

• Type: The type of values that can be sent and received through the channel.

• capacity: The buffer size of the channel. If set to 0, the channel is unbuffered.

Example of creating a channel using make()

package main

import (
    "fmt"
    "time"
)

func main() {
    // Using make() to create an unbuffered channel of integers
    ch := make(chan int)

    // Sending data into the channel using a goroutine
    go func() {
        for i := 1; i <= 5; i++ {
            ch <- i
            time.Sleep(time.Second) // Simulating some work before sending the next value
        }
        close(ch)
    }()

    // Receiving data from the channel
    for num := range ch {
        fmt.Println("Received:", num)
    }
}

> go run main.go
Received: 1
Received: 2
Received: 3
Received: 4
Received: 5

Conclusion

In this blog post, we have demystified the new() and make() functions in Go and explained their differences and use cases. To summarize:

• Use new() to allocate memory for value types and obtain a pointer to the zeroed value.

• Use make() to create and initialize slices, maps, and channels (reference types) with their respective types and initial capacities.

Understanding the distinctions between new() and make() is crucial for efficient memory allocation and data initialization in Go. Properly applying these functions will lead to cleaner and more optimized code in your Golang projects. Happy coding!

package main

import (
    "fmt"
    "net/http"
    "os"
)

func main() {
    // Replace "." with the actual path of the directory you want to expose.
    directoryPath := "."

    // Check if the directory exists
    _, err := os.Stat(directoryPath)
    if os.IsNotExist(err) {
        fmt.Printf("Directory '%s' not found.\n", directoryPath)
        return
    }

    // Create a file server handler to serve the directory's contents
    fileServer := http.FileServer(http.Dir(directoryPath))

    // Create a new HTTP server and handle requests
    http.Handle("/", fileServer)

    // Start the server on port 8080
    port := 8080
    fmt.Printf("Server started at http://localhost:%d\n", port)
    err = http.ListenAndServe(fmt.Sprintf(":%d", port), nil)
    if err != nil {
        fmt.Printf("Error starting server: %s\n", err)
    }
}

Understanding the code

1. Package and Imports: The code begins with the standard Go package declaration (package main) and the necessary imports. It utilizes the "fmt", "net/http", and "os" packages.

2. Setting the Directory Path: The variable directoryPath stores the path of the local directory that you want to expose. In this example, it is set to ".", which represents the current working directory. You can change this to the desired directory path on your system.

3. Checking Directory Existence: Before proceeding, the code checks if the specified directory exists. It does this by using os.Stat() which returns an error if the directory does not exist. The function os.IsNotExist() is then used to handle this specific error condition. If the directory doesn't exist, the code will print an error message and terminate.

4. Creating the File Server: Assuming the directory exists, the code creates an HTTP file server handler using http.FileServer(http.Dir(directoryPath)). This file server will serve the contents of the specified directory.

5. Handling Requests: The code then sets up a new HTTP server using http.Handle("/", fileServer). This means that any incoming HTTP requests to the server will be handled by the file server created in the previous step.

6. Starting the Server: The server is started on port 8080 by calling http.ListenAndServe(fmt.Sprintf(":%d", port), nil). If the server encounters any errors during startup, they will be printed out.

Running the code

To run this code on your machine, follow these steps:

1. Save the code to a file with a ".go" extension (e.g., main.go).

2. Open your terminal or command prompt and navigate to the directory containing the file.

3. Compile and run the code by entering the command: go run main.go.

Once the server is up and running, you can access the contents of the specified directory by visiting http://localhost:8080 in your web browser.

Conclusion

In this blog post, we explored a simple yet powerful Go code snippet that allows you to expose a local directory through an HTTP server. This can be immensely useful for sharing files, collaborating with team members, or even setting up a basic file-sharing solution. By understanding this code, you've taken a step towards harnessing the potential of Go for handling web-related tasks efficiently. Remember to ensure the directory you want to expose exists and to keep security considerations in mind when making files accessible over the internet.

As your development projects grow, you may find yourself repeating the same taskfiles over and over again in multiple repositories. This can be a waste of time and effort as you need to make similar changes in multiple projects, and it can also lead to errors and inconsistencies.

The team I worked on also had a similar situation, to overcome this, we created a Git repository called templates where we store our taskfiles which are common to almost all our projects (for common tasks like, build, setting environment variables, helm, docker and other commands).

Common Repository

Once we created a common repository, we started splitting up our common tasks into multiple files like docker.yml, env.yml, helm.yml, go.yml etc., After splitting up our taskfiles, our git repository looked like this

├── README.md
├── taskfiles
│   ├── docker.yml
│   ├── env.yml
│   ├── git.yml
│   ├── go.yml
│   ├── helm.yml
│   └── vagrant.yml
└── vagrant
    ├── Vagrantfile
    └── devenv_customization.sh

Since we were using Vagrant to set up our development environment (both Windows and Mac) as the same, we used Vagrant. But this is a topic for another blog :)

Anyway, we stored our multiple taskfiles in a folder named taskfiles. Once our repository was prepared, we proceeded to the individual repositories of our services and added the common repository as a Git submodule by executing the command mentioned below. (This command only needs to be run once, and then we'll commit the changes to our current repository so that we don't need to run it again.)

git submodule add https://github.com/<repo url> templates

Once we run this command, a new folder called templates will be created and all our files from the common repo will be present here.

Now, it's time to incorporate these common Taskfiles into our service repository. Create a new file called Taskfile.yml in the root directory of the repository and import the Taskfiles as described below.

version: 3

## add tasksfiles as per the project requirements
includes:
  go:
    taskfile: templates/taskfiles/go.yml
    optional: true
  docker:
    taskfile: templates/taskfiles/docker.yml
    optional: true
  helm:
    taskfile: templates/taskfiles/helm.yml
    optional: true

tasks:
  submodule:
    desc: "Update submodules"
    cmds:
      - git submodule update --init --recursive --remote --merge

  default:
    desc: "Default task"
    cmds:
      - task --list-all

In this example, we have imported the necessary taskfiles for this project using the "includes" keyword. Additionally, we added a task called "submodule," which will be utilized when someone clones this repository and wants to retrieve files from the common repo into the templates directory.

Now that everything is set up, it's time to give it a test run. Execute the task command in the terminal, and you will see that all the tasks from the imported taskfiles are displayed in the terminal.

❯ task
task: [default] task --list-all
task: Available tasks for this project:
* default:                        Default task
* submodule:                      Update submodules
* docker:build-image:             Build Docker Image
* go:build                        Build the golang application
* helm:up                         Deployes the helm chart
* helm:down                       Deletes the helm deployment

Voila! Everything works as expected. Now that we have imported all the tasks into our repo's Taskfile, we can run commands like task go:build, task helm:up, etc.

This method worked great for our team to clean up the clutter of common tasks in the Taskfiles of multiple repositories. By storing them in a single place and using them only where needed, we've streamlined our workflow.

In this blog post, we discuss advanced workflows for maintaining common Taskfiles across multiple projects using a Git repository called templates. By splitting common tasks into separate files and incorporating them as Git submodules, we streamline the development process, reduce repetitive work, and minimize errors and inconsistencies. I hope you find this blog helpful to implement common taskfiles in your project too :) See you in my next blog. Until then, Happy Coding!

Sample github repo - https://github.com/cksidharthan/taskfile-blog-example

We were searching for an alternate build tool that is much more flexible and maintainable. Enter Taskfiles, a versatile tool that simplifies the process of automating tasks and streamlining your development workflow. In this blog post, we will dive deep into Taskfiles, understanding what they are, how they work, and how they can enhance your development experience.

What is a Taskfile?

Taskfiles are a powerful tool designed to automate various tasks in your development workflow. They offer an elegant and straightforward way to define, organize, and execute common tasks such as building, testing, linting, deploying, and more. Unlike complex Makefiles or shell scripts, Taskfiles provide a user-friendly interface and eliminate the need for repetitive command-line typing.

Installation and Setup

Setting up Taskfiles is a breeze. It requires minimal configuration, making it an excellent choice for developers of all skill levels. To get started, follow these simple steps:

Step 1 - Install Task

Task is the command-line utility that reads and executes Taskfiles. To install Task, you can use Go's package manager, simply run:

go install github.com/go-task/task/v3/cmd/task@latest

Step 2 - Create Taskfile.yml

In your project directory, create a file named "Taskfile.yml". This file will contain the tasks you want to automate.

Syntax and Usage

Taskfiles use YAML as their configuration language, which offers a human-readable and straightforward structure. A typical Taskfile consists of two main components: global settings and individual tasks.

Let's explore a basic Taskfile to understand the syntax:

# Taskfile.yml
version: 3

tasks:
  default:
    desc: List all tasks
    cmds:
      - task --list-all

  build:
    desc: Build Go application
    cmds:
      - go build -o myapp .
    silent: true

  test:
    desc: Test Go Code
    cmds:
      - go test ./...

  lint:
    desc: Lint Go application
    cmds:
      - golint ./...

In this example, we define three tasks: default, build, test, and lint. Each task contains a cmds key that lists the commands to be executed when the task is run. The optional silent key can be set to true to suppress command output.

When you type task and press Enter, and you will get the list of tasks available as below

❯ task
task: [default] task --list-all
task: Available tasks for this project:
* build:         Build Go application
* default:       List all tasks
* lint:          Lint Go application
* test:          Test Go Code

To run various tasks that you have specified in the taskfile, you can type task build, task lint or task test as per your requirement.

You can also specify environment variables that has to be set in the current environment when running the tasks. Below is an example of how to do it.

env:
  GREETING: Hey, there!

There are more powerful workflows that can be done via Taskfiles which I will cover in future blog posts.

PS: I don't have anything against makefiles or any other build tools, I just saw that using Taskfiles greatly increased the maintainability of our codebase better for our team and we are continuing to use it. We were blown away by the capabilities of the Taskfiles. I will try to cover more advanced workflows that our team has created using the taskfiles in coming blogs. Until then, Happy Coding! :)