Skip to content

Read & write config in S3

When a tool's configuration file lives in an AWS S3 bucket, the config-aws-s3 sibling module wraps an *s3.Client as a config.FS.

go get gitlab.com/phpboyscout/go/config-aws-s3

You build and configure the S3 client — region, credentials, endpoint all stay yours — and hand it in with the bucket name:

import (
    awsconfig "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"

    "gitlab.com/phpboyscout/go/config"
    configawss3 "gitlab.com/phpboyscout/go/config-aws-s3"
)

cfg, _ := awsconfig.LoadDefaultConfig(ctx)   // your region, your credentials
client := s3.NewFromConfig(cfg)

store, err := config.NewStore(ctx,
    config.WithFiles(configawss3.Wrap(client, "my-bucket"), "config.yaml"),
    config.WithEnv("APP"),
)

The config.FS name is the object key — Wrap(client, "my-bucket") reads and writes the object config.yaml. To scope a bucket shared with unrelated objects, add a key prefix:

configawss3.Wrap(client, "my-bucket", configawss3.WithKeyPrefix("apps/mytool/"))

The write is a copy-then-delete, and that is fine

S3 has no atomic rename, so the commit is CopyObject + DeleteObject. The target is still replaced atomically — PutObject/CopyObject are atomic per object, so a reader never sees a half-written object. The one imperfection is a crash between the copy and the delete, which leaves a staged object behind under a recognisable .config-stage key — a findable orphan you can lifecycle-expire by rule, never a corrupted target or a lost write. Conflict detection is unaffected: the write path fingerprints content by SHA-256 at load and refuses a write if the object changed underneath it.

Hot-reload is polled, at a calm cadence

An object store has no local path, so it is watched by polling. Because each poll is a billed GetObject, config-aws-s3 declares a 60-second default through config.PollIntervalHinter — far calmer than the 2-second local default, and overridable with WithPollInterval.

What it costs

The AWS SDK for Go v2 S3 packages, plus config and what it already brings — asserted by an allowlist test pinned to exactly the S3 SDK modules, so a consumer reading from S3 never compiles another cloud's SDK. The tests run against LocalStack under testcontainers, so the suite needs no AWS account.