# AI Fleet — Local LLM Plan v2 (corrected)
*Built from Gemini deep research + WHOIS + WebFetch verification, 2026-05-15*
*Supersedes: ai-fleet-install-plan-2026-05-15.md (v1)*
*Audience: Jonathan (human)*

---

## What changed since v1

The original plan leaned heavily on Gemini's deep research report. Subsequent verification revealed Gemini hallucinated several specific claims while getting the broad strokes right. This plan is grounded in **verified facts only** (in-text citations included). Specific corrections from v1:

| v1 said | Truth | Action |
|---|---|---|
| `geminicli.com` is malicious — block at firewall | Google's real Gemini CLI product site (MarkMonitor registrar, googledomains NS) | Removed firewall block. Google CLI is a competitor IDE; not a pivot, but worth ~30 min of reading. |
| `ollama pull gemma4:26b` | Gemma 4 isn't on Ollama yet (library still Gemma 3, last update 5 months ago) | Phase removed. Defer until Ollama adds tag or pivot to llama.cpp + HF GGUF |
| Mistral Small 4 = 119B MoE | The real "Mistral Small" is **3.1, 24B dense**. No 119B variant exists. | Test Mistral Small 3.1 24B instead — real model, fits 24GB Q4_K_M cleanly |
| Fix Qwen3 via Modelfile template `{{.Function \| json}}` | Issue #14601 confirms: NOT fixable at Modelfile level. Requires Ollama Go-source change. Workaround is exactly what we already do (Hermes mode). | Phase removed. Our existing Hermes workaround IS the answer. |

**Verified-still-valid from Gemini's report:**
- Ollama issue #14601 (Qwen3 Go-struct + tool-call-stripping bugs) — real, dated March 3, 2026
- DeepSeek-R1 prose-loop diagnosis — matches our bake-off
- WSL2 auto-shutdown issue — matches our experience
- Qwen3-Coder Aug 2025 chat-template fixes exist (unsloth) — worth investigating

---

## Strategic posture

**Keep `qwen2.5-coder:32b` as Tier 1A baseline.** It works end-to-end. Don't replace what's not broken.

**Three additive moves**, in order of ROI:

1. **Add Mistral Small 3.1 (24B) to Tier 1A as alternative** — real model, designed for tool calling, fits 24GB cleanly. Cheap to test.
2. **Apply non-controversial harness changes** to `local-agent.py` — bigger context window, prefix-cache normalization. Skip the temperature change (unverified).
3. **Investigate Gemini CLI architecture** for design ideas only — not a pivot away from our harness.

**Defer:**
- Gemma 4 until Ollama supports it
- Mistral Small "4" — doesn't exist
- Tier 1B EVO-T1 work until Intel GPU PPA driver block resolves (separate track)

**Permanently dropped:**
- deepseek-r1 as agentic model (reasoning model, wrong tool for the job)
- Any Modelfile-based fix for Qwen3 family (impossible per upstream)
- Geminicli.com firewall block (it's Google)

---

## Phase 0 — Pre-flight (15 min)

### 0.1 Baseline regression check
```bash
cd /home/claude/ai-fleet
./tests/runners/run-routing.sh  # must pass 50/50

python3 scripts/local-agent.py \
  --task "Add a comment to scripts/dispatch.sh explaining tier 0" \
  --repo /home/claude/ai-fleet \
  --model qwen2.5-coder:32b \
  --backend http://10.0.0.199:11436 \
  --dry-run --max-turns 10
```
**Acceptance:** Status=success on qwen2.5-coder baseline. This is the regression floor for everything that follows.

### 0.2 Verify Ollama version on JONATHAN-PC
```bash
ssh administrator@10.0.0.199 'powershell -c "(Invoke-WebRequest http://localhost:11436/api/version -UseBasicParsing).Content"'
```
Note current version. Some bug fixes for tool calling shipped post-March 2026 — if version is older than issue #14601's report date (2026-03-03), consider upgrading.

### 0.3 Disk space + GPU temp check
```bash
ssh administrator@10.0.0.199 'powershell -c "Get-PSDrive C | Select-Object Used,Free; nvidia-smi --query-gpu=temperature.gpu,utilization.gpu,memory.used --format=csv"'
```
**Acceptance:** ≥ 25GB free for Mistral pull; GPU idle (temp <50°C, util <5%, VRAM <4GB used per existing safety guards).

---

## Phase 1 — Broader Tier 1A candidate sweep (~3 hr)

**Update 2026-05-15:** After Reddit r/LocalLLM + ollama.com library re-check, three more candidates surfaced that Gemini's report missed (and one that I wrongly dismissed). Test in this priority order:

| # | Model | Tag | VRAM | Why test |
|---|---|---|---|---|
| 1 | **gpt-oss:20b** | `gpt-oss:20b` | ~16GB | OpenAI open-source, **native function calling**, designed for agentic. Lowest risk. |
| 2 | **Gemma 4 26B MoE** | `gemma4:26b` | 18GB | 4B active params (fast), 256K context, recent release. Verify Jinja template. |
| 3 | **Mistral Small 3.1 24B** | `mistral-small3.1:latest` | ~14GB | Designed for tool calling, real model (the v1 plan's "Mistral Small 4" was hallucinated). |
| 4 | **Qwen 3.6 35B A3B MoE** | `qwen3.6:35b-a3b` | ~22GB | 73.4% SWE-bench Verified claimed. **CAUTION:** qwen3 family went 0-for-5 on our stack. MoE may dodge the slowness. Test last. |

### 1.0 Verify exact Ollama tags before pull
```bash
# Browse each library page to confirm current tag names + sizes:
# https://ollama.com/library/gpt-oss
# https://ollama.com/library/gemma4
# https://ollama.com/library/mistral-small3.1
# https://ollama.com/library/qwen3.6
```

### 1.1 For each candidate, pull on JONATHAN-PC Windows Ollama (port 11436)
```bash
ssh administrator@10.0.0.199 'powershell -c "ollama pull <TAG>"'
```

### 1.2 Bake-off task per model (same benchmark we used May 14)
```bash
python3 scripts/local-agent.py \
  --task "Add --help flag to scripts/dispatch.sh (include tier descriptions for tier 0, 1A, 1B, 2)" \
  --repo /home/claude/ai-fleet \
  --model <TAG> \
  --backend http://10.0.0.199:11436 \
  --max-turns 20
```

### 1.3 Acceptance criteria per candidate
- ✅ **Pass:** Task completes ≤ 12 turns, valid commit, valid PR. Becomes a Tier 1A option alongside qwen2.5-coder.
- ⚠️ **Partial:** Completes but takes >12 turns or has quirky tool-call format issues. Document quirks; fallback only.
- ❌ **Fail:** Stalls, loops, errors. Document why. Disqualified.

### 1.4 Model-specific notes

**gpt-oss:20b** — known continuedev/continue thread reports tool-calling friction. May need harness adjustments. Try Hermes-mode first (`NO_TOOLS_MODELS` substring match for "gpt-oss").

**gemma4:26b** — Gemini's research warned about "infinite tool-call loop" if the `--jinja` flag isn't set or if tool results aren't injected as structured `tool_response` blocks. Verify Ollama template with `ollama show gemma4:26b --template` first.

**mistral-small3.1** — should work cleanly with native function calling. No special harness needed.

**qwen3.6:35b-a3b** — MoE architecture means inference is fundamentally different from qwen3:32b dense (which ran 30x slower). MIGHT dodge our prior failures. But may inherit Ollama bug #14601 if Ollama still has the same Go-struct serialization issue. Test with Hermes mode (tools-in-prompt) first.

### 1.5 If any candidate passes — multi-task validation
Don't anoint a winner on one task. Run 3 additional varied tasks:
```bash
# Task A: Single-file refactor
"Rename the function get_tier_label to format_tier in scripts/dispatch.sh, update all references in the same file"

# Task B: New file creation
"Create a new file tests/runners/test-newmodel.sh that runs run-smoke.sh 3 times and reports pass/fail count"

# Task C: Bug fix style
"Look at hooks/local-route-classifier.sh and add a comment explaining what rule 4 does"
```
Track turn count + completion rate per task per model. Real evaluation data.

### 1.6 Disk space reality check
4 models @ 14-22 GB each = ~70 GB of pulls on JONATHAN-PC. Confirm Phase 0.3's disk-free check covers this before bulk-pulling. Can delete a candidate after testing to make room for the next if tight.

---

## Phase 2 — Harness changes to `local-agent.py` (1 hr)

**Only changes verified safe; dropped speculative ones from v1.**

### 2.1 Prefix-cache normalization (safe, verified harmless)
Strip empty `<think></think>` blocks from message history before each LLM call. Per issue #14601's adjacent discussion, these empty blocks defeat the prefix cache on Ollama for reasoning models, causing full-context reprocessing every turn.

In `local-agent.py`, add helper near `parse_content_tool_calls`:
```python
def normalize_messages_for_cache(messages: list) -> list:
    """Strip empty <think></think> pairs to preserve Ollama prefix cache."""
    cleaned = []
    for m in messages:
        content = m.get("content") or ""
        content = re.sub(r"<think>\s*</think>", "", content)
        cleaned.append({**m, "content": content})
    return cleaned
```
Call it inside `call_llm()` right before `resp = requests.post(...)`.

### 2.2 Context window bump
Change `--num-ctx` default from `8192` → `32768` (in `argparse.add_argument` block). Ollama silently crops longer inputs; 32K matches what real repo work needs.

Make this a CLI override too — for the Mistral 3.1 test, you may want `--num-ctx 16384` (lower memory) until you confirm 32K doesn't bust VRAM with 24B+KV.

### 2.3 **Don't** change temperature
v1 of the plan said raise temperature to 1.0 for thinking models per Gemini's report. **Skip this.** It's unverified, and high temp on a coding agent risks introducing bugs. Stay at 0.1 unless empirical evidence says otherwise.

### 2.4 Acceptance
```bash
# Regression: qwen2.5-coder baseline must still pass
python3 scripts/local-agent.py --model qwen2.5-coder:32b --task "Add a comment to scripts/dispatch.sh explaining tier 0" --repo /home/claude/ai-fleet --backend http://10.0.0.199:11436 --dry-run --max-turns 10

# If Phase 1 passed Mistral, re-test it with the harness changes applied
python3 scripts/local-agent.py --model mistral-small3.1:24b-instruct-2503-q4_K_M --task "<same as Phase 1.2>" ...
```

---

## Phase 3 — Investigate Gemini CLI for design ideas (30 min, read-only)

Google's `@google/gemini-cli` is a real, polished agentic CLI. Goal: spend 30 min skimming their public docs/code for design patterns we might steal — NOT to adopt the tool.

### Look at
- https://github.com/google-gemini/gemini-cli — README, /docs folder
- Their tool-calling format (do they use Hermes-style? Native function calls? Custom?)
- Their multi-turn state handling
- Whether they support local Ollama-compatible backends (probably no — they're a Gemini-first product)

### Output
A short note in `plans/` summarizing:
- What they do better than our harness
- What we should ignore
- Any code patterns worth copying (NOT licensing — just design)

If turns out they DO support local backends as a first-class option, escalate that finding — could change strategy.

---

## Phase 4 — ~~Investigate Qwen3-Coder unsloth chat-template fix~~ DROPPED

**Removed after empirical receipts review.** The May 14 bake-off log data is decisive:

- qwen2.5-coder:32b: 2-for-2 success, ~8.5 min average
- qwen3:32b: 0-for-2, hit max turns at **89 minutes** once
- qwen3-coder:30b: HTTP 500 from Ollama on first call (1 turn, 0s)
- deepseek-r1:32b: 0-for-2, hit max turns at **75 minutes** once

Three qwen3-family attempts across two model variants, three distinct failure modes. The unsloth chat-template might "fix" the chat formatting but doesn't address the ~30x per-turn slowdown of the qwen3 family on our stack. Not worth 30 min of investigation when we already have a clear winner.

If qwen3-coder ever becomes attractive again, the path forward is probably **off-Ollama** (vLLM or llama.cpp direct with proper Jinja), not patching the Ollama template.

---

## Phase 5 — Tier 1B (EVO-T1) — separate track (BLOCKED)

Per `plans/SESSION-RESUME.md`, EVO-T1 is blocked on Intel GPU PPA (`kobuk-team/intel-graphics`) connectivity. When unblocked, the work is:

### 5.1 Resolve PPA blocker
Option A (wait), B (force IPv4), or C (switch to intel.com direct .deb) per SESSION-RESUME.md.

### 5.2 Install IPEX-LLM via pip
```bash
ssh claude@10.0.0.56 'pip install --pre --upgrade ipex-llm[cpp]'
```
**Verify current version yourself** — Gemini's "2.3.0b20251029" pin is unverified. Look at https://github.com/intel-analytics/ipex-llm for current stable as of EVO-T1 install day.

### 5.3 Model: Mistral Nemo 12B (preferred) OR Qwen 2.5 Coder 14B
Mistral Nemo 12B is quantization-aware-trained, fits 8GB Arc 140T cleanly, 128K context. Better fit than 14B-class for the iGPU.

```bash
ssh claude@10.0.0.56 'ollama pull mistral-nemo:12b-instruct-q4_K_M'
```

### 5.4 Acceptance
- ≥ 12 tps sustained
- Bake-off `--help` task completes ≤ 12 turns

**Defer all of Phase 5 until PPA resolves.** Separate work stream.

---

## Phase 6 — Acceptance + commit

### 6.1 Definition of done (post-Phases 1-4)
- ✅ qwen2.5-coder:32b baseline still passes (no regression from harness changes)
- ✅ Mistral Small 3.1 24B status documented (pass/partial/fail)
- ✅ Harness changes (Phase 2.1, 2.2) merged to main
- ✅ Gemini CLI investigation note written
- ✅ Qwen3-Coder unsloth template tested OR documented as "not viable"

### 6.2 Commit and update plans
```bash
cd /home/claude/ai-fleet
git checkout -b harness-improvements-2026-05
git add scripts/local-agent.py plans/
git commit -m "feat(local-agent): prefix-cache norm + num_ctx 32K default

- Strip empty <think></think> blocks to preserve Ollama prefix cache
  (per Ollama issue #14601 discussion)
- Bump default num_ctx 8192 → 32768 for repo-level work
- Mistral Small 3.1 24B added as Tier 1A option (alongside qwen2.5-coder:32b)
"
gh pr create --base main --draft --title "AI Fleet harness improvements (post-bake-off v2)"
```

### 6.3 Update SESSION-RESUME.md and master plan
- Note that Gemma 4 is deferred until Ollama support lands
- Note Mistral Small 3.1 added as Tier 1A alternative
- Note deepseek-r1 permanently disqualified for agentic work

---

## What we are NOT doing (and why)

| Item | Why skipped |
|---|---|
| ~~Pull `gemma4:26b`~~ | **CORRECTED 2026-05-15**: IS on Ollama (`ollama.com/library/gemma4`). Now in Phase 1. |
| Pull "Mistral Small 4" | Doesn't exist. The actual model is Mistral Small 3.1, 24B (Phase 1). |
| Edit Qwen3 Modelfile to fix tool calls | Ollama issue #14601 explicitly says this is impossible at template level. Need upstream Go fix. We already work around it via Hermes mode. |
| Raise temperature to 1.0 for thinking models | Unverified Gemini claim. Risk of introducing bugs. Test only if we have a specific failure mode to address. |
| Block geminicli.com at firewall | It's Google's real product. |
| Migrate `/delegate` off OAuth to API key | "Mass 403 bans" claim is unverified. Don't change auth without evidence. |
| Replace `local-agent.py` with Gemini CLI | Gemini CLI is a competitor IDE, not a backend. Doesn't help AI Fleet's goal of routing local. |

---

## Time budget

| Phase | Time | Blocker |
|---|---|---|
| 0 — Pre-flight | 15 min | None |
| 1 — Mistral Small 3.1 test | 1-2 hr | None |
| 2 — Harness changes | 1 hr | None |
| 3 — Gemini CLI design read | 30 min | None |
| ~~4 — Qwen3-Coder unsloth template~~ | DROPPED | Empirical receipts disqualify the qwen3 family on our stack |
| 5 — EVO-T1 IPEX-LLM | (separate track) | Intel PPA driver block |
| 6 — Accept + commit | 30 min | All prior |

**Total active work: ~3 hours, plus EVO-T1 (separate).** (was 4 hr — Phase 4 dropped)

---

## Empirical bake-off receipts (May 14, 2026)

The May 14 bake-off conclusively settled Tier 1A model selection. Score:

| Model | Success Rate | Avg Duration |
|---|---|---|
| **qwen2.5-coder:32b** | **2/2 ✅** | 8.5 min |
| qwen3:32b | 0/2 ❌ | 26-89 min until failure |
| qwen3-coder:30b | 0/1 ❌ | HTTP 500 on first call |
| deepseek-r1:32b | 0/2 ❌ | 75 min until max turns |

**Tier 1A primary: qwen2.5-coder:32b. Locked.** Mistral Small 3.1 24B (Phase 1) is the only forward-looking experiment to find an alternative.

---

## Open questions worth more research (not blocking)

1. Will Ollama add gemma4 to its library, or do we need to pull via HuggingFace? Watch `ollama.com/library` for changes.
2. What's the actual current stable IPEX-LLM version as of EVO-T1 unblock day? Check Intel's GitHub at that time.
3. Is there a meaningful difference between Mistral Small 3.1 and Mistral Nemo 12B for tool calling? Probably yes (different generations, different sizes) — worth a side-by-side once both are pullable.
4. Are there new Ollama versions post-March 2026 that improved the Qwen3 tool-calling bugs? Check changelogs before installing newer Ollama on JONATHAN-PC.