GRPO Research Prompt for Gemini
================================

I need a detailed, implementation-ready research report on using GRPO (Gradient Reward Policy Optimization) to align a Qwen 2.5 3B Instruct model fine-tuned with LoRA for personality/behavioral alignment. Here's my exact setup:

Training hardware: NVIDIA RTX 4090 (24GB VRAM) running in WSL2 (Ubuntu-Ada) on Windows 11. NOT training on the edge device.

Deployment target: NVIDIA Jetson Orin Nano Super (8GB unified memory), running Ollama with GGUF export.

Framework: Unsloth + TRL (already using SFTTrainer + QLoRA, r=64, alpha=128, all linear layers, adamw_8bit). Python 3.12, PyTorch 2.10+cu128.

Goal: The model is a personal AI assistant named "Ada" with a specific personality — casual, concise, playful, self-aware of her limitations (3B model on a Jetson), knows when to defer to a more capable model. The problem we're solving is NOT reasoning quality — it's behavioral alignment: she over-helps on greetings (responds to "Morning" with unsolicited recipe essays and code), hallucinates confidently, and doesn't match the desired conversational tone.

Current training pipeline: SFT on ~550 curated JSONL conversation examples (ShareGPT/ChatML format with system/user/assistant turns). This gets us 80% there but the remaining 20% (appropriate response length, knowing when NOT to talk, casual tone consistency) seems like it needs preference optimization.

What I need researched:

1. TRL's GRPOTrainer — exact API, how it differs from SFTTrainer, what data format it expects, how to chain it after SFT (SFT first for base style, then GRPO for refinement)

2. Reward function design for personality alignment — NOT helpfulness/harmlessness scoring like standard RLHF. I need reward functions that score:
   - Response length proportionality (short greeting → short response, complex question → longer response)
   - Absence of sycophancy and filler phrases ("It's important to note," "Let's explore," "Great question!")
   - Casual tone consistency (not switching between formal and informal mid-response)
   - Appropriate uncertainty (admitting "I don't know" vs. confidently hallucinating)
   - NOT volunteering unsolicited information (the "oatmeal problem" — responding to "Morning" with a breakfast lecture)

3. Can these reward functions be rule-based / heuristic? We can't fit a separate neural reward model alongside the policy model. Can we use simple Python functions (regex for filler phrases, token count ratios, sentiment analysis) as the reward signal? What are the tradeoffs vs. a neural reward model?

4. Memory budget on RTX 4090 — GRPO needs to generate multiple completions per prompt. With QLoRA (4-bit base + bf16 adapters), how many completions per prompt can we generate on 24GB? What's the max group_size parameter? Do we need to reduce LoRA rank from 64 during GRPO?

5. Training data format for GRPO — Does it need prompt-only data (no reference responses)? Or prompt + reference? How to convert our existing curated JSONL into GRPO format?

6. Qwen 2.5 3B specific issues — Any known problems with GRPO on Qwen architecture? Token generation speed in GRPO's sampling phase? KV cache considerations?

7. Concrete code example — A working Python script using Unsloth + TRL GRPOTrainer with a custom rule-based reward function, targeting Qwen 2.5 3B with QLoRA. Not pseudocode — actual runnable code with all imports and config.

8. Is GRPO actually worth it for our case? Honest assessment — with only ~350-550 training examples and a personality (not reasoning) alignment goal, does GRPO add meaningful value over just doing more/better SFT? Or is this diminishing returns for our scale?