The Go Embed Package: A Hidden Gem That's More Useful Than You Think

Over the past couple of years, we've had a number of big features make their way into the Go programming language. Some of these include generics added back in version 1.18, improved backwards compatibility starting with 1.21, and of course the all-new advanced routing features added in 1.22.

For every big feature found in the release notes, there are a number of smaller ones added to the language as well—some of which tend to get overlooked. One of these overlooked features was actually added to the language back in February 2021 in version 1.16: the embed package.

The embed package allows you to embed files inside of your Go binary at compile time. Despite it being in the language for nearly 4 years, I hadn't actually used it in a production setting. However, I think that was a mistake. Recently, I've started to see its value, and it's actually a lot more useful than I initially thought.

Why I Initially Overlooked the Embed Package

The reason I hadn't used this package before was because initially I didn't really know what problems it could be used to solve. However, whilst recently working on a new production-ready middleware package that I intend to use with multiple projects, I actually ran into a problem that I ended up using this feature to solve. Through doing so, I've come to appreciate the package more than I ever thought I would—so much so that I now use it in a number of different ways.

How the Embed Package Works

Before we take a look at some of the ways I use this package, let's quickly take a moment to see how it actually works. The basic idea is that you use the embed package to bundle files inside of your application binary at compile time, allowing you to then access the contents as if they were hardcoded.

Basic Example

To see this in action, here's a project that contains a file called hello.txt that lives inside of the same directory as my main function. In this case, I want to load the contents of this file inside of my application and print it to the console.

Rather than taking the typical approach of loading the file in at runtime using something such as the os.ReadFile function, instead we can use the embed feature to load it in at compile time.

First, we need to import the embed package:

import (
    _ "embed"
    "fmt"
)

You'll notice that I'm importing it using the blank identifier as the explicit package name. This is used to prevent the compiler from throwing an error as there won't be any explicit references to this package in this code.

Next, in order to embed a file, we first need to define a variable to store the contents of it:

//go:embed hello.txt
var data string

Important notes:

This variable is defined inside of the package scope rather than inside of the local scope of the main function. This is intentional as the embed package can't work with locally scoped variables.
The variable type can be one of three: a byte array, a string, or the fs type of the embed package (which we'll explore later).
The //go:embed hello.txt directive specifies the name of the file to load from the relative path where the code or package lives.

Now all that remains is to print out this content:

func main() {
    fmt.Println(data)
}

When I build and execute this binary, it prints out "hello world"—the contents of my hello.txt file. Even if I delete the hello.txt file from the file system and run the code again, it still prints the same string because the file has been embedded in the binary at compile time.

Production Use Cases

Now that we know how the embed package works, let's take a look at some of the ways I actually use it in production.

1. Embedding Lua Scripts for Redis

The first way I came to rediscover the embed package was when building a production-ready middleware package. One of the middleware components is a rate limiter that uses a Redis client to store the number of requests a client makes. To implement this algorithm, I used a Lua script.

Redis and its forks allow you to use Lua scripts to perform application logic that interacts with multiple Redis commands atomically, preventing race conditions.

Initially, I was loading this script at runtime using the ReadFile function of the OS package. However, because I wanted to distribute this code as a third-party package, having a Lua script that needs to be loaded at runtime caused several issues:

Security concerns: The script could be modified before loading (either maliciously or accidentally).

Distribution complexity: The middleware package needed the script to be available on the file system of any deployed application.

By using the embed package, I could solve both problems:

//go:embed rate_limiter.lua
var rateLimiterScript string

This solution gave me the best of both worlds: effective distribution while keeping the script as a separate file with proper syntax highlighting and linting.

2. Database Migrations

Perhaps my favorite use case is embedding database migrations. Typically, I perform database migrations during application startup using the fantastic golang-migrate package, which loads SQL files from the file system at runtime.

While this approach works, it comes with caveats—most notably increased complexity when deploying applications. You need to ensure migration files are available in the file system wherever your app is deployed, often requiring additional Docker configuration and environment variables.

This is where the third type supported by the embed directive comes in: the embed.FS type. This type allows you to embed multiple files, creating an embedded file system.

Here's how to embed database migrations:

//go:embed migrations/*.sql
var migrations embed.FS

The embed.FS type has several useful methods:

ReadDir: Obtains a list of entries in a named directory
ReadFile: Returns the contents of a file as a byte array
Open: Returns a file type from the fs package

The Open method is particularly powerful because it makes the embed.FS type conform to the fs.FS interface, which is accepted by many packages, including golang-migrate:

source, err := iofs.New(migrations, "migrations")
if err != nil {
    log.Fatal(err)
}

migrator, err := migrate.NewWithSourceInstance("iofs", source, databaseURL)
if err != nil {
    log.Fatal(err)
}

err = migrator.Up()
if err != nil && err != migrate.ErrNoChange {
    log.Fatal(err)
}

This allows me to solve deployment complexity issues completely. I've already implemented this change in my guestbook web app, and you can review the actual commit on GitHub.

3. HTML Templates

Similar to database migrations, I typically load HTML templates during application startup from the file system. Using embedded files solves the same distribution and deployment issues.

First, load the templates using the embed directive:

//go:embed templates/*.html
var templates embed.FS

Then, instead of using the ParseGlob method of the template struct, use the ParseFS method:

tmpl, err := template.ParseFS(templates, "templates/*.html")
if err != nil {
    log.Fatal(err)
}

This makes it easy to distribute the binary anywhere with HTML templates included.

4. Static Files

The fourth way I use the embed package is for serving static files. Instead of reading from the file system at runtime using the http.FileServer function coupled with http.Dir, I now use the embedded file system with the http.FileServerFS function:

//go:embed static/*
var staticFiles embed.FS

http.Handle("/static/", http.StripPrefix("/static/", http.FileServerFS(staticFiles, "static")))

Important considerations for static files:

All embedded files are loaded into memory, providing increased performance but using more system memory
Only recommend this for certain types of static files like stylesheets, JavaScript, and smaller images
For larger files like photos or videos, consider keeping them on the file system
If files aren't available at build time, use the traditional http.FileServer function instead

Additional Use Cases to Consider

I'm still considering other use cases, such as loading configuration files that I typically use when working with Viper. The embed package continues to prove its value in solving various pain points I've encountered in Go development.

Key Benefits of Using Embed

Simplified deployment: No need to manage separate files during deployment
Improved security: Files can't be modified after compilation
Better distribution: Third-party packages can include necessary files
Performance: Files are loaded from memory rather than disk
Reliability: Eliminates runtime file system dependencies

Conclusion

I'm really glad to have rediscovered the embed package and how it can solve a number of pain points that I've typically encountered in Go. From Lua scripts and database migrations to HTML templates and static files, the embed package provides elegant solutions to common distribution and deployment challenges.

The embed package has been available since Go 1.16, but it's one of those features that truly shines when you encounter the right use case. If you haven't explored it yet, I highly recommend giving it a try in your next Go project.

Useful Links

This article was sponsored by boot.dev. Use code DREAMSOFCODE to get 25% off your first payment for boot.dev - that's 25% off your first month or your first year, depending on the subscription you choose.