llm stock trader, fine-tuning & rl alignment
#deepseek 7b#unsloth#lora#trl#grpo#pytorch#hugging face
15 fine-tuning experiments (sft to grpo) on deepseek 7b with lora r=16. score improved from 0.300 to 0.537 (+79%). diagnosed 3 training failures: hold collapse, reward hacking, kl catastrophe.
/ training journey
- 15 fine-tuning experiments (sft -> grpo) on deepseek 7b with lora r=16 (q/k/v/o), lr=5e-6
- score: 0.300 (base) -> 0.417 (sft) -> 0.537 (grpo), +79% total improvement
- sft trained on 12k reverse-distilled dataset, outperformed larger noisy sets
- grpo trained against a 1.22m-param causal transformer world model as synthetic environment
/ training failures diagnosed
- hold collapse: agent learned inaction is never penalized, 85% hold actions. fixed with data rebalancing
- reward hacking: 84% of reward came from formatting, not trading quality. fixed with asymmetric reward decomposition
- kl catastrophe: kl divergence hit 4.2, destroyed base model knowledge, score dropped to 0.301. fixed with kl coefficient tuning
/ dataset engineering
- 12k reverse-distilled sft dataset from gpt-4 rollouts through the trading environment
- outperformed larger noisy datasets — quality over quantity for domain-specific fine-tuning
- asymmetric reward decomposition separates format compliance from trading quality to prevent spec gaming
/ how it works
01generate training data by running gpt-4 through the stock trading environment
02filter and curate 12k high-quality demonstrations for sft
03train sft checkpoint with lora r=16 on deepseek 7b (q/k/v/o targets)
04train grpo on top of sft checkpoint using neural world model as environment
05evaluate on held-out market episodes, diagnose failures, iterate on reward design
/ features
sft -> grpo pipeline
two-stage training: supervised fine-tuning on distilled demonstrations, then grpo reinforcement learning against a neural world model. each stage addresses different failure modes.
specification gaming diagnosis
documented 3 distinct training failures (hold collapse, reward hacking, kl catastrophe) with root cause analysis and fixes. each failure taught a lesson about reward design and training dynamics.
reverse distillation
12k training examples generated by running gpt-4 through the trading environment and filtering for high-scoring trajectories. smaller, cleaner dataset outperformed larger noisy alternatives.
asymmetric reward decomposition
separates format compliance reward from trading quality reward. prevents the model from gaming the reward by producing well-formatted but poor trading decisions.