Skip to content

Read & write config in S3

When a tool's configuration file lives in an AWS S3 bucket, the config-aws-s3 sibling module wraps an *s3.Client as a config.FS.

go get gitlab.com/phpboyscout/go/config-aws-s3

You build and configure the S3 client — region, credentials, endpoint all stay yours — and hand it in with the bucket name:

import (
    awsconfig "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"

    "gitlab.com/phpboyscout/go/config"
    configawss3 "gitlab.com/phpboyscout/go/config-aws-s3"
)

cfg, _ := awsconfig.LoadDefaultConfig(ctx)   // your region, your credentials
client := s3.NewFromConfig(cfg)

store, err := config.NewStore(ctx,
    config.WithFiles(configawss3.Wrap(client, "my-bucket"), "config.yaml"),
    config.WithEnv("APP"),
)

The config.FS name is the object key — Wrap(client, "my-bucket") reads and writes the object config.yaml. To scope a bucket shared with unrelated objects, add a key prefix:

configawss3.Wrap(client, "my-bucket", configawss3.WithKeyPrefix("apps/mytool/"))

When you do not need this

If you read one object once at startup, fetching it with the AWS SDK and handing the bytes to WithReaders needs no adapter at all.

Reach for config-aws-s3 when you want the bucket to behave like a filesystem: several files, writes routed back to the object, or hot-reload picking up a change someone else made.

The write is a copy-then-delete, and that is fine

S3 has no atomic rename, so the commit is CopyObject + DeleteObject. The target is still replaced atomically — PutObject/CopyObject are atomic per object, so a reader never sees a half-written object. The one imperfection is a crash between the copy and the delete, which leaves a staged object behind under a recognisable .config-stage key — a findable orphan you can lifecycle-expire by rule, never a corrupted target or a lost write. Conflict detection is unaffected: the write path fingerprints content by SHA-256 at load and refuses a write if the object changed underneath it.

Hot-reload is polled, at a calm cadence

An object store has no local path, so it is watched by polling. Because each poll is a billed GetObject, config-aws-s3 declares a 60-second default through config.PollIntervalHinter — far calmer than the 2-second local default, and overridable with WithPollInterval.

Getting a client

Building the client yourself is the default. Two further rungs exist, and the second lives in a subpackage:

// You resolved the config; the adapter builds the client.
fsys, err := configawss3.FSFromConfig(cfg, "my-bucket")

// Nothing at all — note the separate import.
import s3ambient "gitlab.com/phpboyscout/go/config-aws-s3/ambient"

fsys, err := s3ambient.Default(ctx, "my-bucket")

The subpackage is not decoration, and it costs less here than for its siblings: seven further modules rather than ten, because S3's larger service graph already carries internal/v4a, accept-encoding and presigned-url. Each adapter measures its own rather than quoting another's.

There is no default region, and the bucket is required. AWS documents no region default, so an empty one is ErrNoRegion rather than a guess.

What it costs
Modules added 20 — 11 for the AWS S3 SDK, 9 for the config graph

The AWS SDK for Go v2 S3 packages, plus config and what it already brings — asserted by an allowlist test pinned to exactly the S3 SDK modules, so a consumer reading from S3 never compiles another cloud's SDK. The tests run against LocalStack under testcontainers, so the suite needs no AWS account.