# Gemini Deep Research Prompt — AI Fleet Local LLM Evaluation
*Generated 2026-05-15 for AI Fleet bake-off analysis*

---

I am building an agentic coding system called "AI Fleet" that routes Claude Code 
prompts to local LLMs to offset cloud API token costs. I need deep research on 
which open-source LLMs are best suited for autonomous tool-calling coding agents 
as of May 2026, given my specific hardware and the failures I've already encountered.

---

## HARDWARE SETUP (exact)

### TIER 1A — JONATHAN-PC (10.0.0.199, RTX 4090, Windows 11)
- **GPU:** NVIDIA RTX 4090, 24 GB VRAM
- **CPU:** Intel (Windows 11 desktop, gaming rig)
- **RAM:** ~64 GB system RAM
- **OS:** Windows 11
- **Constraint:** `OLLAMA_MAX_LOADED_MODELS=1` — only one model in VRAM at a time
- **Constraint:** 4 safety guards before auto-routing (no game process, GPU <5% util, VRAM <4 GB used, GPU temp <50°C). Gaming auto-evicts all models via NVENC session poller (JPC-Ollama-Poller, 60s interval).

**WSL2 SETUP (Ubuntu-Ada distro, separate from main serving):**
- Port 11434: WSL2 Ollama serving `qwen2.5-coder:32b` — used ONLY for Super-Ada training pipeline. NOT for the agentic harness.
- **CRITICAL WSL2 PROBLEM:** WSL2 distro auto-shuts-down ~15-30s after the last external `wsl.exe` interop session ends. This kills any in-VRAM model mid-inference. We tried `vmIdleTimeout=-1`, `sleep infinity` services, masking `wsl-pro-service` — none prevented it reliably for external API serving.
- `netsh portproxy` handles external routing for the WSL port.

**Windows-native Ollama (port 11436) — THIS IS WHAT THE AGENTIC HARNESS USES:**
- Runs as SYSTEM scheduled task `OllamaWindows-11436` (auto-restart, AtStartup)
- `OLLAMA_HOST=0.0.0.0:11436`, `OLLAMA_MAX_LOADED_MODELS=1`, `OLLAMA_KEEP_ALIVE=1h`
- Available models (all ~Q4_K_M GGUF): `qwen3:32b`, `deepseek-r1:32b`, `qwen2.5-coder:32b`
- Cold-load time: ~14s per model swap
- API: OpenAI-compatible `/v1/chat/completions` endpoint via Ollama

### TIER 1B — EVO-T1 (Intel NUC, 10.0.0.56, not yet fully deployed)
- **CPU:** Intel Core Ultra 9 285H (P+E cores)
- **RAM:** 64 GB DDR5
- **iGPU:** Intel Arc 140T (8 GB shared VRAM)
- **NPU:** Intel NPU 3 (for OpenVINO Whisper transcription)
- **Plan:** Run Qwen 2.5 Coder 14B Q4_K_M via IPEX-LLM (~12-18 tps, always-on). Blocked on Intel GPU PPA driver install; not yet serving models.

### TIER 0 — Jetson Orin Nano (10.0.0.104)
- **RAM:** 8 GB unified (CPU+GPU shared)
- **Runtime:** Ollama with CUDA (JetPack)
- **Models:** `llama3.2:3b-instruct-q4_K_M` (~25 tps), `bge-small-en-v1.5` embeddings
- **Role:** Simple/small tasks only. AI-dedicated — no creep allowed.

### TIER 2 — Cloud (escalation)
- Claude API (Sonnet 4.6 / Opus 4.7) — architecture, KPMG work, multi-file refactors

---

## THE AGENTIC HARNESS (local-agent.py)

The harness calls the Ollama `/v1/chat/completions` endpoint with:
- OpenAI-compatible tool definitions (JSON schema, `"tools"` + `"tool_choice":"auto"`)
- 5 tools: `read_file`, `write_file`, `run_bash`, `list_files`, `done()`
- System prompt with Hermes-format tool call instructions (XML `<tool_call>` blocks)
- `num_ctx: 8192` tokens, `max_tokens: 4096`, `temperature: 0.1`
- 20-turn max agentic loop
- Hermes-mode fallback: if structured `tool_calls` field is empty, parse `<tool_call>{...}</tool_call>` XML blocks from content text
- 3-strike corrective prompt on no-tool turns
- Context trimming: keeps system + first user + last 10 turns

**Benchmark task:** "Add --help flag to scripts/dispatch.sh (include tier descriptions)"
- Model must: explore repo, read the file, edit it, verify with `bash -n`, `git commit`, call `done()` — typically 6-10 tool calls

---

## BAKE-OFF RESULTS (what we've already tried, May 2026)

### qwen2.5-coder:32b Q4_K_M (Windows-native Ollama :11436)
- **Result:** ✅ SUCCESS
- **Turns:** ~8 turns, completed the task end-to-end
- **Speed:** ~50-65 tps on 4090
- **PR:** Created valid draft PR with correct changes
- **Notes:** Uses Hermes-mode tool calling (structured `tool_calls` not populated by Ollama, but `<tool_call>` XML in content works reliably)
- **Status:** CURRENT WINNER / BASELINE

### qwen3:32b (Windows-native Ollama :11436)
- **Result:** ❌ FAILED — 37-minute timeout, only 11 turns completed
- **Failures:**
  - Round 1: Empty content loops — Ollama strips `<think>...</think>` tokens natively, delivering empty string as assistant content. No tool calls.
  - Round 2 (after fix — `options.think:false`): Stalled repeatedly. Recovered briefly at turn 9 after 3-strike corrective prompt, then stalled again at turn 11. Never completed task.
  - Speed: ~30x slower than qwen2.5-coder:32b on same hardware
- **Fix attempted:** Added `think:false` to Ollama options dict; dropped tools param (NO_TOOLS_MODELS list). Neither fully resolved the problem.
- **Notes:** Even when it made tool calls, execution was erratic and unreliable for a multi-step agentic task requiring sustained tool-call chains.

### deepseek-r1:32b (Windows-native Ollama :11436)
- **Result:** ❌ FAILED — immediate API rejection, then prose loop
- **Failures:**
  - Round 1: HTTP 400 Bad Request when `"tools"` param included in request. Ollama's deepseek-r1 implementation hardcodes rejection of function calling / tools parameter.
  - Round 2 (after fix — dropped tools param, Hermes-only): Emits prose ~87% of the time instead of `<tool_call>` blocks. After any LLM timeout, enters unrecoverable prose loop (explains reasoning endlessly in `<think>` tags but never emits a tool call).
  - Max turns hit without task completion across all rounds.
- **Notes:** deepseek-r1 is a reasoning model, not optimized for tool use. The model architecture (heavy chain-of-thought) conflicts with the structured output requirements of an agentic harness.

### qwen3-coder:30b (Windows-native Ollama :11436) — most recent test
- **Result:** In progress / partial data
- **Notes:** Required the same `think:false` treatment as qwen3 (substring match). Ollama Go-struct serialization bug corrupts tool schema for qwen3-coder variants → also in NO_TOOLS_MODELS (must use Hermes-only mode). Model was pulled fresh (~19 GB), test launched but not fully evaluated yet.

---

## KNOWN OLLAMA BUGS / CONSTRAINTS (as of May 2026)

1. **Tool schema serialization bug:** Ollama's Go struct for certain model families (confirmed: qwen3, qwen3-coder) corrupts the tool schema when passed via the `"tools"` parameter. Workaround: drop tools param entirely, rely on Hermes-format `<tool_call>` XML in content + system prompt instruction.

2. **Think token stripping:** For qwen3 family, Ollama intercepts `<think>...</think>` tokens and strips them before returning content. When thinking is enabled, content may be empty string. Fix: `options.think:false`.

3. **deepseek-r1 tools rejection:** Ollama's deepseek-r1 Modelfile has hardcoded rejection of the `"tools"` API parameter (returns HTTP 400 immediately).

4. **GGUF-only on Ollama:** All models running as GGUF quantized files. EXL2/AWQ not supported natively by Ollama. TabbyAPI would support EXL2 but is not yet deployed.

---

## RESEARCH QUESTIONS (answer all)

### 1. GEMMA 4 ASSESSMENT
Gemma 4 was open-sourced (Apache 2.0) in April 2026. The 27B dense and 26B MoE variants fit in 24GB VRAM at Q4_K_M.

a) How does Gemma 4 27B perform on agentic tool-calling coding tasks specifically? Not general benchmarks — actual function/tool calling reliability in multi-turn agentic loops.

b) Does Ollama support Gemma 4 natively with proper tool calling as of May 2026? Any known Ollama bugs specific to Gemma 4?

c) What GGUF quantization fits in 24GB VRAM with ~4GB headroom for KV cache?

d) Is Gemma 4 better or worse than qwen2.5-coder:32b for agentic coding tasks specifically (not general reasoning)?

e) What is the Hermes / ChatML / Gemma prompt format for Gemma 4 in Ollama, and does it support function calling natively or require Hermes-style workaround?

### 2. ALTERNATIVE MODELS TO EVALUATE
What other models (besides Gemma 4 and the ones we've already tested) should we try in May 2026 for agentic coding tool-calling on a 24GB VRAM 4090?

Consider: Mistral family (Mistral Nemo, Codestral, Devstral), Llama 3.3 70B (quantized), Command R+, Yi-Coder, OpenCoder, any new entrants April–May 2026.

For each candidate, provide:
- VRAM footprint at recommended GGUF quantization
- Known tool-calling reliability score / evidence
- Ollama support status and any known bugs
- Comparison to qwen2.5-coder:32b on coding agent tasks

### 3. qwen3-CODER DEEP DIVE
qwen3-coder:30b is being tested now.

a) Is the Ollama Go-struct tool schema bug confirmed for qwen3-coder, or is there a newer Ollama version that fixes it?

b) What prompt format does qwen3-coder use for tool calling — Hermes, native function calling, or something else?

c) How does qwen3-coder:30b compare to qwen2.5-coder:32b on coding benchmarks (HumanEval, SWE-bench, MBPP, tool-use specific evals)?

d) Is qwen3-coder architecturally a thinking model (like qwen3) or a pure coder (like qwen2.5-coder)? Should `think:false` always be set?

### 4. EVO-T1 MODEL SELECTION (Intel Arc 140T, 8GB VRAM, 64GB RAM)
The EVO-T1 will run a 14B model via IPEX-LLM (Intel Extension for PyTorch on Arc GPU). The plan is Qwen 2.5 Coder 14B Q4_K_M.

a) Is Qwen 2.5 Coder 14B the best choice for Arc 140T, or is there a better 14B-class model for agentic coding tool calling in May 2026?

b) Does Gemma 4 have a 12B or smaller variant that would fit in 8GB VRAM and has reliable tool calling?

c) IPEX-LLM vs llama.cpp (with SYCL backend) for Intel Arc — which has better tool-calling support and more active maintenance as of May 2026?

### 5. PROMPT FORMAT / HERMES COMPATIBILITY MATRIX
Our harness relies on Hermes-format tool calling as primary fallback (XML `<tool_call>{...}</tool_call>` blocks, system prompt injected schema).

For each model family — Gemma 4, Mistral/Codestral, Llama 3.3, Qwen 2.5 Coder, Qwen 3 Coder — what is the native tool-calling prompt template, and does it support Hermes-format as a workaround when native structured tool_calls are not populated by Ollama?

### 6. RECOMMENDATION
Given all the above:

a) Should we swap qwen2.5-coder:32b with something else on the 4090, or keep it as the Tier 1A model?

b) Is Gemma 4 worth pulling and testing?

c) What's the single most impactful experiment to run next given our setup?

d) Any architectural changes to local-agent.py we should consider (different context window, different temperature, different turn limit)?

---

## CONSTRAINTS / THINGS NOT TO SUGGEST

- No commercial API models (OpenAI, Anthropic, Gemini Pro) as replacements — the whole point is offsetting cloud costs with local hardware
- No models requiring >23GB VRAM at recommended quantization (leaves no KV headroom)
- No TabbyAPI/EXL2 suggestions for Tier 1A until that's deployed — Ollama only now
- Do NOT suggest downloading any binaries from `geminicli.com` — that domain is not affiliated with Google and appears in a hallucinated PDF we already flagged
- KPMG/Arena work (FastAPI, React, Terraform, AWS) always stays on Claude API — local routing is for personal/infra tasks only