# AI Fleet — Local LLM Upgrade Install Plan
*Built from Gemini Deep Research output, 2026-05-15*
*Audience: Jonathan (human) + Gemini CLI follow-up*

---

## Executive Summary

Based on Gemini's May 2026 LLM evaluation, the AI Fleet bake-off failures (qwen3:32b, deepseek-r1:32b) are caused by **server-side Ollama bugs**, not the models themselves. The fix is a combination of:

1. **Modelfile patches** for Qwen3 family (Ollama issue #14601)
2. **Harness changes** to `local-agent.py` (prefix cache normalization, context window, temperature)
3. **New model evaluation** of Gemma 4 26B MoE and Mistral Small 4 (with caveats below)
4. **Tier 1B (EVO-T1)** unblocked via IPEX-LLM `[cpp]` pip path once drivers resolve

**Keep `qwen2.5-coder:32b` as Tier 1A baseline.** Don't replace it until alternatives prove out.

---

## ⚠️ Claims to Verify Before Acting

Gemini's report has a few claims I couldn't independently verify. Treat them as hypotheses, not facts:

| Claim | Why suspect | How to verify |
|---|---|---|
| Mistral Small 4 is "119B MoE, fits comfortably in 24GB VRAM at 4-bit" | Math doesn't work: 119B × 4 bits = ~60GB weights alone. MoE doesn't free VRAM by default | Check `huggingface.co/mistralai` and `ollama.com/library/mistral-small` for actual size + memory footprint |
| Tau2-bench 86.4% (Gemma 4 31B) vs 6.6% (Gemma 3) | Specific benchmarks unverified | Cross-check Gemma 4 model card on ai.google.dev |
| `ipex-llm[cpp]==2.3.0b20251029` + Ollama v0.9.3 | Very specific version pin | Check Intel's IPEX-LLM GitHub for current stable |
| Mass 403 ToS bans on OAuth Gemini CLI in Feb/Apr 2026 | Policy claim, needs confirmation | Check google AI dev forum + your own Gemini Pro account status before changing auth |
| `geminicli.com` malicious | ✅ Already confirmed from prior analysis + multiple security citations | No action — block at firewall |

**Action: Spot-check the Mistral Small 4 sizing before pulling it (it's a ~60GB download otherwise).**

---

## Phase 0 — Pre-flight (do these first, ~10 min)

All commands run from EVO-T1 (`claude-server-evo`, 10.0.0.56) unless noted.

### 0.1 Verify Ollama version on JONATHAN-PC
```bash
ssh administrator@10.0.0.199 'powershell -c "(Invoke-WebRequest http://localhost:11436/api/version -UseBasicParsing).Content"'
```
**Acceptance:** Ollama version ≥ 0.5.0 (needed for the Modelfile template fix). Upgrade if older.

### 0.2 Check disk space for new models
```bash
ssh administrator@10.0.0.199 'powershell -c "Get-PSDrive C | Select-Object Used,Free"'
```
**Acceptance:** ≥ 100 GB free. Mistral Small 4 (if claim is right) is ~60 GB; Gemma 4 26B MoE is ~16 GB.

### 0.3 Confirm current baseline still works
```bash
cd /home/claude/ai-fleet
./tests/runners/run-routing.sh  # must pass 50/50
python3 scripts/local-agent.py \
  --task "Add a comment to scripts/dispatch.sh explaining tier 0" \
  --repo /home/claude/ai-fleet \
  --model qwen2.5-coder:32b \
  --backend http://10.0.0.199:11436 \
  --dry-run --max-turns 10
```
**Acceptance:** Status=success, valid commit. This is the regression baseline.

### 0.4 Pull research citation evidence (for sanity)
Before pulling a 60GB model, verify the size:
```bash
# Check Ollama library page
curl -s https://ollama.com/library/mistral-small | grep -iE "size|param"
# Or check HF
curl -s https://huggingface.co/api/models/mistralai/Mistral-Small-4 | jq .
```

---

## Phase 1 — Fix Qwen3 family (high-value, low-risk)

Per Gemini's report (cross-referenced with Ollama issue #14601), the Qwen3 Go-struct bug has a known Modelfile fix.

### 1.1 SSH to JONATHAN-PC and inspect the qwen3 Modelfile
```bash
ssh administrator@10.0.0.199 'powershell -c "ollama show qwen3:32b --modelfile"' > /tmp/qwen3-modelfile.txt
grep -n "Function" /tmp/qwen3-modelfile.txt
```

### 1.2 Apply the template fix
The fix is replacing `{{.Function }}` with `{{.Function | json }}` in the template. Then create a custom model:
```bash
# Save patched Modelfile to JONATHAN-PC
# Then on JONATHAN-PC:
ollama create qwen3-fixed:32b -f C:\Tools\qwen3-fixed.Modelfile
```

### 1.3 Re-run bake-off with patched model
```bash
python3 scripts/local-agent.py \
  --task "Add --help flag to scripts/dispatch.sh (include tier descriptions)" \
  --repo /home/claude/ai-fleet \
  --model qwen3-fixed:32b \
  --backend http://10.0.0.199:11436 \
  --max-turns 20
```
**Acceptance:** Task completes in ≤ 12 turns. If it stalls, revert to qwen2.5-coder.

---

## Phase 2 — Test Gemma 4 26B MoE (the safer new model)

Gemma 4 26B MoE math checks out (~15.6 GB Q4_K_M, fits 24 GB with headroom). This is the lower-risk model to add.

### 2.1 Pull Gemma 4 26B MoE on JONATHAN-PC
```bash
ssh administrator@10.0.0.199 'powershell -c "ollama pull gemma4:26b"'
# Verify the tag exists first by browsing ollama.com/library/gemma4
# If the tag differs (e.g. gemma4:26b-instruct-q4_K_M), adjust
```

### 2.2 Critical: enable Jinja template parsing
Per Gemini's report, Gemma 4 needs `--jinja` flag set or it enters infinite tool-call loops. Ollama uses Jinja by default in recent versions, but verify:
```bash
ssh administrator@10.0.0.199 'powershell -c "ollama show gemma4:26b --template"'
```
Confirm the template uses `tool_response` blocks (not plain text injection).

### 2.3 Bake-off test
```bash
python3 scripts/local-agent.py \
  --task "Add --help flag to scripts/dispatch.sh (include tier descriptions)" \
  --repo /home/claude/ai-fleet \
  --model gemma4:26b \
  --backend http://10.0.0.199:11436 \
  --max-turns 20
```

### 2.4 If infinite-loop bug appears
Symptom: model calls same tool repeatedly because it can't "see" prior tool results.
Fix: ensure `local-agent.py` injects tool results in the format Gemma's template expects (structured `tool_response`, not plain content text).

---

## Phase 3 — Test Mistral Small 4 (CONDITIONAL — verify sizing first)

**DO NOT pull until you've confirmed it actually fits in 24 GB.** Gemini's report says 119B MoE Q4 fits in 24GB which is mathematically impossible without expert offloading. Either:
- The model is smaller than claimed (some Mistral variants are 22B/24B "Small"), or
- The fit requires CPU offloading with major speed cost

### 3.1 Verify model size
```bash
curl -s https://huggingface.co/api/models/mistralai/Mistral-Small-4-Instruct-2503 | jq '.safetensors.total'
# Or check the unsloth GGUF page Gemini cited: huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF
```

### 3.2 If actually ≤ 25 GB at Q4_K_M
Proceed to pull and bake-off:
```bash
ssh administrator@10.0.0.199 'powershell -c "ollama pull mistral-small:4-q4_K_M"'
python3 scripts/local-agent.py \
  --task "Add --help flag to scripts/dispatch.sh (include tier descriptions)" \
  --repo /home/claude/ai-fleet \
  --model mistral-small:4-q4_K_M \
  --backend http://10.0.0.199:11436 \
  --max-turns 20
```

### 3.3 If actually 60GB+
Skip Mistral Small 4 for Tier 1A. Either:
- Use a smaller Mistral variant (Mistral Nemo 12B — fits anywhere)
- Wait until EXL2/AWQ deployment for 4090 (TabbyAPI plan)
- Test on EVO-T1 with offload only if it has serving infra

---

## Phase 4 — Harness changes to `local-agent.py`

Pure code changes, no model pulls. Test against qwen2.5-coder:32b baseline.

### 4.1 Prefix-cache normalization (strip empty think blocks)
In the message-trimming logic around line 431-432 of `local-agent.py`, before sending messages to Ollama:
```python
def _strip_empty_think_blocks(messages):
    cleaned = []
    for m in messages:
        content = m.get("content") or ""
        # Strip empty <think></think> blocks that break prefix cache
        content = re.sub(r"<think>\s*</think>", "", content)
        cleaned.append({**m, "content": content})
    return cleaned
```
Call this before every `call_llm()`. Expected impact: 30-50% latency reduction on thinking models per Gemini's report.

### 4.2 Context window enforcement
Line 592 — change default `--num-ctx` from `8192` to `32768`:
```python
parser.add_argument("--num-ctx", type=int, default=32768, ...)
```
**Why:** 8K is too small for repository-level work. Ollama silently crops longer inputs.

### 4.3 Temperature per model class
In `call_llm()` around line 346, currently `temperature: 0.1`. Add model-class branching:
```python
THINKING_MODELS = ("qwen3", "deepseek-r1", "gemma4")  # raise temp for these
temperature = 1.0 if any(m in model for m in THINKING_MODELS) else 0.1
top_k = 40 if any(m in model for m in THINKING_MODELS) else None
```
**Why:** Low temp causes thinking-model loops. Gemini's report claims raising to 1.0 stabilizes reasoning.

### 4.4 Acceptance test for harness changes
```bash
# Regress against baseline first
python3 scripts/local-agent.py --model qwen2.5-coder:32b ... --dry-run
# Must still pass — these changes shouldn't break the working model

# Then re-test the previously-failing models
python3 scripts/local-agent.py --model qwen3:32b ... --max-turns 20
python3 scripts/local-agent.py --model deepseek-r1:32b ... --max-turns 20
```

---

## Phase 5 — Tier 1B (EVO-T1) — wait until drivers resolved

Currently blocked on Intel GPU PPA (kobuk-team) per `plans/SESSION-RESUME.md`. Once that's fixed:

### 5.1 Install IPEX-LLM via pip path (NOT the portable zip)
Per Gemini's recommendation (cite #6, #7 — Reddit IPEX-LLM solutions):
```bash
ssh claude@10.0.0.56 'pip install --pre --upgrade ipex-llm[cpp]'
# Verify the version Gemini cited (2.3.0b20251029) — but use whatever's current stable
```

### 5.2 Run Ollama via IPEX-LLM init script
```bash
ssh claude@10.0.0.56 '
  source /opt/intel/oneapi/setvars.sh
  init-ollama  # creates symlinks to IPEX-LLM accelerated binary
  OLLAMA_HOST=0.0.0.0:11434 ollama serve &
'
```

### 5.3 Pull 12-14B model
Three candidates, in order of preference:
1. **Mistral Nemo 12B** — Quantization-aware trained, 128K context, fits 8GB Arc
2. **Qwen 3 Coder 14B** — Best SWE-bench at this size, may need offload for >8K context
3. **Gemma 4 E4B** — Fastest option, multimodal support

```bash
ssh claude@10.0.0.56 'ollama pull mistral-nemo:12b-instruct-q4_K_M'
```

### 5.4 Acceptance
- Inference at ≥ 12 tps for sustained generation
- Tool calling works end-to-end on same `--help` benchmark task

---

## Phase 6 — Security cleanup (do alongside Phase 4)

### 6.1 Confirm geminicli.com isn't reachable
```bash
# Block at UniFi firewall as belt-and-suspenders
# Add DNS sinkhole or firewall rule for geminicli.com → 0.0.0.0
```

### 6.2 Audit Gemini CLI installation
```bash
which gemini
npm ls -g @google/gemini-cli
# If any non-@google namespace gemini package is installed, uninstall
```

### 6.3 Verify Gemini CLI version ≥ 0.39.1 (CVSS 10.0 patch)
```bash
gemini --version  # if installed
```

### 6.4 Authentication review (low priority, verify claim first)
Gemini's report claims OAuth-based CLI users got mass-banned in Feb/Apr 2026. Before changing anything:
1. Check current `/delegate` flow auth method
2. Verify with current Google AI dev forum status
3. If true, migrate to API-key auth via Google AI Studio
4. If false, no change needed

---

## Phase 7 — Acceptance & Rollback

### 7.1 Definition of done
- Baseline `qwen2.5-coder:32b` still completes the bake-off task (no regression)
- At least one previously-failing model (qwen3 or deepseek-r1) now completes the task with the Modelfile fix + harness changes
- Gemma 4 26B MoE either completes the task OR is documented as "needs Jinja work"
- Mistral Small 4 either tested or explicitly ruled out (sizing claim verified)
- All harness changes regression-tested against baseline
- Tier 1B unblocked (EVO-T1 drivers + 12B model serving)

### 7.2 Rollback if something breaks
```bash
cd /home/claude/ai-fleet
git status  # see what changed
git stash   # stash harness changes
# qwen2.5-coder:32b on baseline harness is always the fallback
```

### 7.3 Commit changes only after acceptance
```bash
git add scripts/local-agent.py
git commit -m "feat(local-agent): prefix cache norm, num_ctx 32K, temp per model class"
```

---

## Open Questions for Gemini Follow-up Research

If you want to ping Gemini again, these are the still-unresolved questions:

1. **Mistral Small 4 actual VRAM footprint at Q4_K_M** — is it really 119B? Is there a way to fit it in 24GB without major offload penalty?
2. **Exact Ollama Modelfile template diff** for qwen3 fix — Gemini said `{{.Function}}` → `{{.Function | json}}` but didn't show the full template
3. **Current stable IPEX-LLM version** as of May 15, 2026 — the report cited a beta build, want current stable
4. **OAuth ban status** — is this actually affecting `/delegate` or only `gemini-cli` direct OAuth users?
5. **TabbyAPI EXL2 path for the 4090** — when EXL2 is deployed (post-Phase 5), does Gemma 4 or Qwen 3 Coder have EXL2 quants available?

---

## Order of Operations (TL;DR for execution)

1. **Phase 0** — Pre-flight, 10 min
2. **Phase 1** — Fix Qwen3 Modelfile, 30 min
3. **Phase 4** — Harness changes (parallel with Phase 1), 1 hr
4. **Phase 2** — Test Gemma 4 26B MoE, 1 hr (includes ~16GB pull)
5. **Phase 6** — Security cleanup, 15 min
6. **Phase 3** — Mistral Small 4 (CONDITIONAL on sizing verification)
7. **Phase 5** — EVO-T1 Tier 1B (BLOCKED until drivers resolve, do separately)
8. **Phase 7** — Acceptance + commit

Total active work: ~3-4 hours, plus EVO-T1 work when unblocked.