I spent the last few weeks building out custom Terraform providers for a half-dozen internal platforms, and it was a great learning experience.

There was one particular oddity that I ran into with custom providers when they were run in certain docker containers.

Background

For those unfamiliar, you can take any CRUD-based API endpoint and turn it into a custom Terraform provider that can be used to manage resources as-code. Custom providers can be quite powerful in reducing the barrier to entry for platforms and services that want to be interacted with as infrastructure-as-code.

While the learning curve for writing providers isn’t steep, it is plagued with oddities and rabbit holes. One of the most frustrating was my accidental discovery of CGO.

“cgo is an amazing technology which allows Go programs to interoperate with C libraries.” - Dave Cheney

Problem

I had these custom providers happily compiling and customers actively using them in their CI/CD pipelines without issue until one customer started reporting an error when they tried to use one of the custom providers in their pipeline.

They were getting the following error:

* provider.myprovider: fork/exec /workspace/terraform/.terraform/plugins/linux_amd64/terraform-provider-myprovider_v0.6.4: no such file or directory

It was quite the odd error, as the binary was present with the appropriate permissions and when manually invoked executed with the expected terraform warning not to run it directly.

Solution

It turns out that the Terraform Go module that is used when creating providers uses some C libraries internally. Typically, this wouldn’t be a problem (and wasn’t up until that point) because the C libraries that are referenced are present on all operating systems.

But, the customer having issues was building a custom CI/CD container from SCRATCH, which meant it was the most barebone container possible. It also meant that those standard C libraries were not present. The “no such file or directory” error was a reference to not being able to find the standard C libraries.

The upside is that it’s a relatively quick fix: disable CGO, so the necessary C libraries are compiled into the same binary instead of being dynamically referenced.

$ export CGO_ENABLED=0

Re-compiling with that environment variable set instantly resolved the issue, though at the cost of marginally larger binary sizes.