Note: This issue draft was generated with AI assistance and reviewed against observed query responses and the relevant Trickster source paths.

Summary

When a Prometheus query whose top-level result is a scalar (for example, scalar(count(...))) is sent through an ALB-backed Trickster backend, Trickster can return NaN even though one or more ALB members return a valid scalar value.

The issue is particularly visible for an instant query. A range query can also contain a large number of duplicate or otherwise non-equivalent scalar fragments instead of one scalar value per evaluation timestamp.

Observed behavior

For an ALB-backed Mimir frontend query, the upstream members returned a mix of valid scalar values and NaN values.

• Trickster instant query: NaN.
• Trickster range query: 3,920 returned values for the same window; 547 were finite and the rest were NaN.

This makes the ALB result unsuitable for callers that expect Prometheus scalar semantics: a scalar aggregation should yield one result for an instant query and one value per requested step for a range query.

Why NaN is returned

The ALB backend fans the request out to multiple origins. Each origin evaluates the scalar expression independently. Origins with no matching input series may legitimately return a scalar NaN, while other origins return finite values.

Trickster recognizes resultType: "scalar" and converts it to an internal vector-like series representation for processing. However, the ALB merge/selection path appears to treat those independent per-origin scalar results as ordinary compatible time-series fragments. That permits a NaN member response to be selected for an instant query, and permits many member fragments to be retained in a range response. The final response therefore does not represent a single global evaluation of the scalar expression.

Expected behavior

Trickster should not select an invalid scalar (NaN) when at least one healthy ALB member returned a finite scalar, and it should not expose per-origin scalar fragments as though they were a single Prometheus scalar result.

At minimum, the response behavior should be deterministic and documented for scalar results through ALB backends.

Suggested fix

Treat resultType: "scalar" as a distinct ALB merge case rather than normal series data:

• Preserve scalar result type and provenance through the fanout/merge pipeline.
• For instant queries, exclude NaN candidates when finite candidates exist, then apply an explicit and documented selection policy (for example, the ALB mechanism's chosen origin).
• For range queries, return at most one scalar per requested evaluation timestamp; do not concatenate independent origin scalar samples into one matrix.
• If global aggregation across origins is intended, re-evaluate or aggregate with semantics appropriate for the PromQL expression. Do not infer global count semantics merely by merging independently evaluated scalar outputs.
• Add ALB integration tests covering mixed finite/NaN scalar responses for both /query and /query_range.

Relevant implementation area

Prometheus scalar responses are recognized in pkg/backends/prometheus/model/timeseries.go, where scalar results are transformed into a vector-like internal representation. The ALB merge path needs to retain the scalar distinction when selecting or combining member responses.