Some questions may be computations other than LLM calls.
An example was given on #4 -- a question whose output is a tunable similarity function.
When the question has differentiable parameters (possibly shared with other questions), the calibrator should be able to backpropagate into the question to make its output more useful.
We may want the ability to improve the calibrator even more by forking the question into two copies with slightly different (perturbed) parameters that can be improved separately. This gives an ensemble of different experts (questions) that may be attended to in different circumstances, and which specialize to those circumstances accordingly.
There are also non-differentiable questions that are still tunable because they specify a space of possible ways to ask the question. E.g., a regex-like language that provides alternative phrasings for a prompt (or opportunities to call an LLM to get alternative phrasings). However, maybe we should just think of that as a factory for generating alternative versions of the question that are variants on existing successful versions (cf. genetic algorithms and DSPy).
Some questions may be computations other than LLM calls.
An example was given on #4 -- a question whose output is a tunable similarity function.
When the question has differentiable parameters (possibly shared with other questions), the calibrator should be able to backpropagate into the question to make its output more useful.
We may want the ability to improve the calibrator even more by forking the question into two copies with slightly different (perturbed) parameters that can be improved separately. This gives an ensemble of different experts (questions) that may be attended to in different circumstances, and which specialize to those circumstances accordingly.
There are also non-differentiable questions that are still tunable because they specify a space of possible ways to ask the question. E.g., a regex-like language that provides alternative phrasings for a prompt (or opportunities to call an LLM to get alternative phrasings). However, maybe we should just think of that as a factory for generating alternative versions of the question that are variants on existing successful versions (cf. genetic algorithms and DSPy).