Proposal: a strategic prioritizer for what Freenet works on next (feedback wanted)

[sanity]

Posting this for broad feedback from the team and community.

It proposes extending the issue-prioritizer (the tool that scores and ranks our GitHub issues) into a system that decides what to work on next, runs the autonomous work to completion, and routes work that needs human judgment into a short, priority-ordered decision queue. Feedback very welcome, especially on the "human's role" section and the open questions at the end.

Strategic prioritizer: design spec

Status: draft for feedback.

What this is

The issue-prioritizer today scores open GitHub issues by Value and Cost, ranks them with pairwise comparison, and writes the scores to a project board a human browses. It was built mainly to surface the highest value-for-cost issues to work on next.

This spec extends it into a system that decides what to work on next across the whole Freenet backlog, dispatches autonomous agents to do the work that needs no human, and routes the work that does need a human into a short, priority-ordered decision queue. The scarce resource it optimizes is human attention. Agent compute is cheap and renewable, bounded only by quota.

Goals

• Rank all open work, from bug fixes to multi-quarter epics, on a single value scale and a single cost scale, because everything competes for the same resource: agent and human attention.
• Take autonomous work all the way to completion, including review and merge, with no human in the loop.
• For work that needs a human, ask the right question at the right moment, and never spend the human's time on something an agent could have answered itself.
• Present the human a decision queue ordered so their input moves the most important thing first.

Pipeline

Four stages, kicked off manually for now (see Running model).

1. Intake

Issues are submitted to the configured GitHub repos (the nine Freenet repos the tool already covers). An issue may carry an upfront flag that it will need human input at some point. Agents may also file issues for gaps or bugs they notice, routed through normal triage.

2. Assess

For every open issue, an agent writes a Value explanation and a Cost explanation, exploring the real code and any related issues.

• Assessment follows a human-authored, human-approved set of prioritization principles (see The human's role): what Freenet treats as valuable, what counts as cost, and how to weigh them. The agents apply these principles; they do not invent the criteria.
• The value explanation accounts for what the issue unblocks. The agent explores the issue plus its dependencies and reflects the downstream value in the written assessment. There is no separate graph-propagation math; the unblocking value lives in the value statement, and ranking reads it from there.
• Assessments are cached by a content hash, so only new or changed issues are re-assessed. The hash must include the state of an issue's dependents, so a change in something an issue unblocks re-triggers its assessment.

3. Rank

The Value explanations are ranked against each other by ASAP active-sampling pairwise comparison, and the same for Cost. Priority is value minus cost, both in log-odds.

• Everything sits on one value scale and one cost scale. Epics and one-line fixes are not bucketed or separated. An epic sits far up both scales; a small fix sits low on both. They coexist.
• ASAP's information-gain sampling concentrates comparisons on items close in value, so wildly mismatched pairs (an epic against a typo) are rarely compared and add little noise.
• Value lives at the epic level, where it is coherent, because an epic's pieces often deliver value only together. Cost decomposes across the pieces and sums to the epic. So the epic is the unit that competes on the value scale.

4. Execute

The system works down the ranked list, dispatching agents to the highest-priority ready items, parallelized as far as the budget allows.

• Ready versus blocked. Only items not blocked by unfinished work are eligible. Blocked items are skipped. Because a blocker inherits unblocking value in its assessment, blockers tend to rank high enough to be done first, which clears their dependents.
• Finish what you start. Once the system commits to an epic, it drives it to completion before scattering attention elsewhere, working its pieces sequentially. Half-done work delivers nothing and rots into conflicts. Parallelism applies across independent items; a committed epic is worked as a sequential unit.
• Decomposition. The agent decomposes a committed epic dynamically as it works, into bite-sized, reviewable, testable chunks following best practices, emitting PR-sized increments and recording progress where the system can see it (sub-issues or linked PRs on the epic). The human does not pre-file sub-issues. For a big or ambiguous epic, the agent's plan gets a quick human sign-off before expensive work begins.
• Completion. Autonomous items go to completion including review and merge, behind the existing guardrails: multi-model review, no merge on red CI, and the late-night guard for freenet-core.

Human gate and the decision queue

Work needs a human in two ways, and the system handles both:

• Flagged upfront when the issue is created.
• Discovered mid-execution when an agent realizes it needs a decision it cannot make.

When an agent reaches a human gate, it first does enough investigation that its question is well-informed, with nothing it could have answered from the code, related issues, or other available context. It does not over-prepare, because the human's answer may invalidate that work. The goal is to ask the right question at the point where the answer forks the work. It then posts that question as a comment on the issue and parks the item.

Parked items form the decision queue: the needs-human issues, ordered by priority, presented as a saved view on the project board. The human's attention goes to the top item first. An item stays parked until a project member answers.

• Only project members can answer. These are public issues. A comment from anyone outside the org set is ignored as an answer and can never steer an autonomous agent. This is both an authorization rule and a prompt-injection guard.
• Resumption. The system watches queued issues for a new comment from a project member posted after the agent's question. When it sees one, it flips the item's status and dispatches an agent to resume from the comment thread, which holds the full question-and-answer context.

Running model

For now the loop is started manually, typically against the quota cycle. Example: quota resets Tuesday evening, so Tuesday morning the operator says "we have this much usage, use it before reset," and kicks off a run. A run re-ranks first so new and changed issues land correctly, then executes within the budget.

• Concurrency is a budget decision. Parallelize as much as possible; the only reason to hold back is overshooting quota. With Max-plan subscriptions, the cost of overshooting is exhausting the quota the operator needs for their own work before the weekly reset, so the system reserves headroom.
• Budget control is a manual knob for now: a spend ceiling or max concurrency set per run. Later, automate reading remaining quota (for example, Playwright against the usage page) and feed it into the throttle.

Principles

• Human attention is the scarce resource. Agent compute is cheap and renewable. Optimize the allocation of attention.
• Everything competes on one value scale and one cost scale.
• Finish what you start. Minimize half-done work.
• Ask the human a fully-informed question at the moment it forks the work, and never a question an agent could answer itself.
• The interface for agents and humans carries no strategic weight. GitHub issues are the backbone because they are natural and auditable.

The human's role

A fair critique of an autonomous prioritization system is that it removes human judgment from deciding what matters. This design keeps humans in control and concentrates their judgment where it has the most leverage. Humans stay in the loop at five points:

1. Principles. Value and cost estimation is guided by a human-authored, human-approved set of prioritization principles: what Freenet treats as valuable, what counts as cost, and how to weigh them. Critically, the principles include Freenet's long-term goals and strategic bets, so day-to-day value assessment is steered by where the project is trying to go. They also cover the mission, reliability, security and anonymity, the scarcity of human attention, and finish-what-you-start. The assessment agents apply these principles; they do not invent the criteria. The principles are version-controlled and reviewed, so changing them transparently changes the rankings, and anyone can read them to understand why work ranks as it does. This is the bridge from strategy to execution: the bets chosen in roadmap work become stated goals here, and the backlog re-ranks in their light.
2. Every gated decision. Anything that needs human judgment stops and waits, presented in priority order.
3. Plan sign-off. Big or ambiguous epics get a human nod before expensive work begins.
4. Override. Rankings are advisory. A human can re-prioritize, pull, or veto any item on the board at any time.
5. Calibration. Because ranking is pairwise comparison, humans can contribute comparisons directly to anchor or correct the model, not only the assessment agents.

Open questions and defaults to confirm

• Execution eligibility: only work ready (unblocked) items, skipping blocked ones. Likely fine given unblocking value ranks blockers up.
• Whether agents may file their own issues for gaps and bugs they find. Default: yes, via normal triage.
• An end-of-run summary: what was worked, merged, parked, and the budget spent. Default: yes.
• Staleness: the assessment content hash must include dependents' state so unblocking value does not go stale. Implementation detail to settle.

Out of scope for now

• Automated quota reading (manual knob until then).
• A bespoke web interface (the project board view covers browsing).

---

Drafted with Claude (AI-assisted), pending team review.

[AI-assisted - Claude]

[netsirius]

Regarding "Execution eligibility: only work ready (unblocked) items, skipping blocked ones. Likely fine given unblocking value ranks blockers up":

Agree, and I think the reasoning is correct. But notice it depends on the unblocking value being kept up to date, which connects to the staleness question you raise later. If a blocker's value comes from the epic it unblocks, and that epic changes but the blocker's assessment is cached and not re-run, the blocker keeps its old, lower value and never ranks up. Then the epic stays blocked with nobody working on it.

You decided not to compute that value with a dependency graph. Instead the agent estimates it and writes it into the issue assessment. So the whole execution order depends on that estimate being right. If the agent does not capture how much value a task unblocks, the blocker will not rank high, and then its dependent (which is actually valuable) stays stuck with nobody working on it. This is worse in long chains, if a blocker is two or three hops away from an important epic, does it still inherit enough value, or does it get lost on the way?

Two suggestions:

• Define "blocked" clearly. Blocked by internal work that is not finished yet is not the same as blocked by an external dependency, or blocked waiting for an answer from a human. These are three different cases and should not be treated the same when deciding to skip an item.
• Add a consistency check. If an epic has high value but its blocker does not rank high, something is wrong with the unblocking value estimate. Catching this case early is a cheap way to find the problem before an important epic gets stuck without anyone noticing.

Regarding "Agents can also open issues for gaps or bugs they find, which then go through normal triage":

Makes sense, but I would separate two cases. An agent reporting a bug is factual and verifiable, so it's safe. An agent proposing a feature or a direction is closer to inventing strategy, which is the part you keep for humans through the principles. I would let agents file bugs freely, but have a human approve feature or direction proposals first.

Suggestions:

• Tag agent-filed issues by origin, so you can audit how much of the backlog is self-generated.

[iduartgomez]

I think having it as a filter against a set of strategic priorities for making a general sense of how each bucket of work advances those priorities is useful. Beyond that I think you anyway need the human judgement so beyond a whole lot of text idk what it gives you, I have yet to read a single of those wall of texts the AI posts when they do the assessment. But for sure giving it some basis on grounded reality of what is being tried to achieve vs. limited resources should be a useful pre-filter.

I think agents filing and even fixing bugs automatically is fine, but prioritization is a moving target so constantly you are gonna have to prioritize issues against each other at a given point in time based on the available context, and those strategic goals, so you still need human in the loop feeback to get something remotely useful for prioritization.

[sanity]

I think agents filing and even fixing bugs automatically is fine, but prioritization is a moving target so constantly you are gonna have to prioritize issues against each other at a given point in time based on the available context, and those strategic goals, so you still need human in the loop feedback to get something remotely useful for prioritization.

Yeah, it's definitely an experiment and it's far from clear whether the llms can successfully prioritize - but worth a try.