Writings

When I first read the Coconut paper1 in 2024, I got interested immediately. Their idea is simple: for the intermediate thinking process, you no longer send one token at a time; instead, you feed the model’s own output into the next token’s position, acting as an embedding. Therefore, the model can think in the latent vector space. As a neuroscientist, I’d definitely agree that we think in the latent space, aka the brain, instead of just in words, so how could I not like it? However, the authors acknowledge in the paper that their successful model, though thinking with vectors, was trained with a curriculum that is human-generated, and they tried the ideal case without human supervision, i.e., just questions and answers, but they failed. Since then, this question has kept coming back to me: if a model only sees the final answer, can it develop its own reasoning, in the latent space? I know GRPO and many of its RL descendants do, but I want more of an SFT-like solution. ...

I was recently curious about the alignment mechanism in those Vision-Language Models (VLMs). We know some VLMs are not inherently multi-modal by design: they glue a pretrained vision encoder to a text-only LLM, usually through a linear projection layer.1 Then joint training is introduced, probably through some multi-modal task. The practice, which also matches our intuition, works pretty well. And I’ve heard that such Frankenstein models can infer unseen characters from a pretrained vision encoder.2 However, I was still puzzled by why such joint training does not disrupt the learned representations and instead aligns different modalities pretty well. This alignment mechanism is related to much broader questions, such as why fine-tuning (in many cases) does not break the learned general knowledge, even with full fine-tuning rather than LoRA. But here let’s focus on the simple two-model alignment problem. Let’s say one is a vision model $f_v​$, and the other is a language model $f_l​$: ...