Read and write etcd¶

An etcd v3 prefix becomes a read-write config layer through config-etcd, a sibling module, so a consumer who needs etcd takes it and its etcd client, and one who does not pays nothing. You build the client and hand it in with a prefix that scopes the read and is stripped from the keys.

go get gitlab.com/phpboyscout/go/config-etcd

import (
    "gitlab.com/phpboyscout/go/config"
    configetcd "gitlab.com/phpboyscout/go/config-etcd"
    clientv3 "go.etcd.io/etcd/client/v3"
)

client, err := clientv3.New(clientv3.Config{Endpoints: []string{"localhost:2379"}})
if err != nil {
    return err
}
defer client.Close()

store, err := config.NewStore(ctx,
    config.WithFiles(config.OS(), "/etc/app.yaml"),            // YAML defaults
    config.WithBackend(configetcd.FromClient(client, "app/")), // etcd outranks them
)
if err != nil {
    return err
}

port := store.View().GetInt("server.port") // 8080, from app/server/port

An etcd layer takes part in precedence, per-key merge, provenance and hot-reload exactly as a file does.

Connect to etcd¶

Building the client yourself is the default, because it is where every endpoint, credential, TLS and namespace decision lives. Two further rungs exist:

// You assembled the config; the adapter builds the client.
b, err := configetcd.FromConfig(clientv3.Config{Endpoints: []string{"localhost:2379"}}, "app/")

// Nothing at all: the environment etcdctl reads.
b, err := configetcd.Default("app/")

store, err := config.NewStore(ctx, config.WithBackend(b), config.WithCloser(b))
defer store.Close()

FromConfig and Default return *OwnedBackend, which must be closed. The adapter built the etcd client, so it owns the client's connections and goroutines. Hand it to the store with WithCloser, as above, and closing the store releases it. FromConfig has done this since config-etcd v0.3.0; before that, nothing could close it. A backend from FromClient is not owned, and the client you built stays yours to close.

Neither rung contacts etcd, so an unreachable cluster is a Load error. FromConfig returns ErrNoEndpoints when the config names none, because it does not fall back to localhost. The reasoning, including why the etcdctl convention was adopted when clientv3 has none, is in who owns the connection.

Default reads what etcdctl reads¶

clientv3 has no DefaultConfig of its own, but etcd's command-line tool documents one: every global flag can be set as ETCDCTL_ plus its name. So a machine that can run etcdctl get against a cluster already has what Default needs, and a tool built on config reaches the same cluster from the same shell.

Default reads seven variables, once, at construction, and no others:

It departs from etcdctl in three places, each on purpose:

• No endpoint is an error, not localhost:2379. ErrNoEndpoints names the variable. A configuration store that guesses its source serves wrong values instead of failing.
• A user with no password is an error, not a prompt. A library has no terminal to ask on.
• ETCDCTL_INSECURE_SKIP_TLS_VERIFY is never read. If a test cluster needs it, build the config yourself and use FromConfig.

A variable it cannot use, such as a dial timeout of 3 with no unit or a certificate without its key, fails at Default with ErrMalformedEnv, naming the variable. With ETCDCTL_DIAL_TIMEOUT unset there is no dial timeout at all: nothing dials until Load, and Load's context bounds it.

Map etcd keys to config keys¶

Keys under the prefix, split on /, become the nested tree:

app/server/host = "localhost"        store.View().GetString("server.host") // "localhost"
app/server/port = "8080"        →    store.View().GetInt("server.port")    // 8080

A dot inside an etcd key also separates segments, because a config path splits on dots: app/log.level reads as log.level. A write to an existing key goes back to that key, dot and all, and a new key is written with slashes. Two keys that read as one path, such as app/log.level beside app/log/level, fail the load with config.ErrBackendUnsafe rather than one hiding the other.

Values are strings, or decoded documents¶

etcd stores bytes, so by default every value is a scalar string and the View's typed accessors coerce it. GetInt("server.port") parses "8080", the same as an environment variable.

Where a key holds a whole JSON or YAML document, pass a value codec. It can be any config.Codec, the interface the sibling format adapters already implement:

import configjson "gitlab.com/phpboyscout/go/config-json"

configetcd.FromClient(client, "app/", configetcd.WithValueCodec(configjson.Codec{}))

A value that decodes to a mapping becomes a subtree at its key's path; anything the codec rejects stays a scalar string. A prefix mixing flat keys and document blobs therefore reads correctly, and enabling a codec never turns a scalar key into a load error.

Leased keys can vanish under you¶

etcd keys can carry a lease and disappear when it expires. This adapter reads them like any other key: the value contributes while it exists, and once the lease lapses the key is simply absent at the next load.

Write back to etcd¶

Unlike the parameter stores, etcd has genuine compare-and-swap, so writes are supported from the first release rather than deferred.

_, err := store.Apply(ctx, config.Set("server.port", 9090))

The guard is the revision each key held when the store last loaded, not when the write was staged. That is the protection worth understanding. If another client changed the key between your program reading the configuration and deciding to write, the write is refused with ErrConflict rather than silently overwriting them.

if errors.Is(err, config.ErrConflict) {
    // Somebody else moved a key. Reload and re-plan; do not retry blindly.
}

A batch is one transaction¶

Every key in one Apply goes into a single etcd transaction, so it applies completely or not at all. If any key's guard fails, none of the writes land.

That is stronger than most of this family can offer (AtomicMultiKey is true), and it means the Store's partial-write rollback path is never needed here.

There is no batch-size limit in the adapter¶

etcd bounds transactions in two ways, and both are server settings your client cannot know: operation count (--max-txn-ops, 128 by default) and request size (--max-request-bytes).

This adapter imposes no cap of its own, because a guessed one would be too low on a tuned cluster and too high on a default one. An oversized batch fails with etcd's own error, naming which limit it hit:

configetcd: committing 200 operation(s): etcdserver: too many operations in txn request

Match it with errors.Is against rpctypes.ErrTooManyOps or ErrRequestTooLarge. It is not a conflict, and treating it as one would send a caller into a retry loop that can never succeed.

Watch for changes¶

etcd's watch is a genuine change feed, so a change reaches your observers by push (NativeWatch: true). The store still passes a foreign change through its settle window and a value comparison before telling your observers.

The watch starts from the revision just after the one captured at load, not from the head, so a change landing between reading the prefix and the watch attaching is replayed rather than missed. Without that the store could hold stale values with no pending notification and no way to discover it.

The cost is that a revision old enough to have been compacted away cannot be replayed from. When that happens the adapter treats it as "reload and re-watch from the current revision", which is the honest recovery, because the alternative would leave the layer silently unwatched.

Test without etcd¶

The adapter reaches etcd through the three-method configetcd.KV interface (Get, Txn, Watch), so a unit test hands New a fake and needs no cluster:

type fakeKV []configetcd.Pair

func (f fakeKV) Get(context.Context, string) ([]configetcd.Pair, int64, error) {
    return f, 1, nil
}

func (fakeKV) Txn(context.Context, []configetcd.Cmp, []configetcd.Op) (bool, error) {
    return true, nil
}

func (fakeKV) Watch(context.Context, string, int64, func()) (func(), error) {
    return func() {}, nil // nothing ever changes
}

backend := configetcd.New(fakeKV{
    {Key: "app/server/port", Value: []byte("8080"), ModRevision: 1},
}, "app/")

Wrap(client) is the real implementation, and what FromClient uses. The module's integration suite, under test/integration, runs against a real etcd under testcontainers and is skipped unless INT_TEST_INTEGRATION=1 is set; it needs a Docker daemon.

Errors you'll see¶

Limits¶

• etcd v3 only.
• Not a secrets store. The layer is never marked sensitive; see below.

It is not a secrets store¶

The layer is not marked sensitive, and there is deliberately no option to mark it so.

Kubernetes keeps Secret objects in etcd, so people demonstrably store secrets there. But an etcd prefix has no audit trail of who read a value, no leasing or rotation, and no separation between the credential that reads configuration and the one that reads secrets. Those are exactly what Vault, AWS Secrets Manager and the other secrets backends provide. An option that made etcd look like a secrets store would make the weaker choice the easier one.

If you have secrets in etcd and cannot move them, you are not stuck:

• wrap the backend in config.Filtered to keep those keys out of the layer, or
• contribute those keys through a backend that does declare itself sensitive.

When you don't need this¶

If nothing needs to change between deploys, a file is simpler and cannot be unreachable at start-up.

Reach for etcd when values change at runtime, when you need a write across several keys to be atomic, or when you already run etcd and would rather not add a second system.

What it costs¶

The etcd client weighs the same as Consul's and less than half of the Kubernetes client-go. That measurement decided this adapter was worth building while a ConfigMap adapter was not, since a mounted ConfigMap is already a directory of files. An allowlist test in the module pins the footprint, so a transitive upgrade cannot widen it quietly.

Related¶

• How dynamic backends work: what every remote backend shares, and where they legitimately differ
• Read and write Consul: the closest sibling, and the reference this follows
• A directory of files: the Kubernetes ConfigMap case, which needs no cluster client
• Filter a backend: bounding which keys a backend contributes