Two Ways to Invent an Image, and an Agent That Relearns Gravity

Part A and Part B followed the same basic pattern: build a baseline, test changes, and evaluate the final model. Part A focused on image generation. Part B moved to reinforcement learning and continuous control.

The research paper came after those two parts. It was a separate study, not an extension of the Pendulum gravity experiment.

Part A used CIFAR-10 and compared a conditional VAE with a conditional GAN. The CVAE used a 128-dimensional latent vector, while the GAN used 128-dimensional noise. I evaluated them side by side with the same frozen classifier, 1,000 generated images, the same seeded validation reference sample, NMI, FID, classifier agreement, and visual grids.

Using one reference set and one extractor mattered because the FID numbers shifted when the same checkpoints were sampled in different notebooks. The comparison used only the two values recomputed side by side.

The models also had to be read on their own terms. The CVAE used reconstruction and KL losses. The GAN had generator and discriminator losses that moved against each other, so comparing their raw loss values directly would have meant nothing.

The final GAN scored 0.1982 NMI versus 0.1177 for the CVAE. Its FID was 122.694 against 179.661, a 31.71% reduction, and it led classifier agreement in seven of ten classes.

The CVAE was smoother to train but produced blurrier samples. The GAN produced sharper, more varied images, though some of that variation came with pixelation and distorted objects.

Part A favoured the GAN for class separation and distributional similarity. The cost was less stable training and more visible artifacts.

Part B used Pendulum-v0, not LunarLander. I built a DQN baseline for default gravity, free-fall, anti-gravity, and supergravity, then tested Double DQN, dueling DQN, double-dueling DQN, and finer action bins.

Only supergravity showed a clear improvement. Double DQN moved the mean return from -268.80 to -260.47 and reduced seed-to-seed standard deviation from 6.02 to 0.82, roughly seven times tighter. The other three settings barely moved; their selected variants mostly reduced variance.

The paper asked whether a diffusion policy could imitate continuous expert behaviour on LunarLanderContinuous-v3 better than standard behaviour cloning and a discretised DQN. The diffusion and MLP students learned from the same SAC demonstrations and predicted the same eight-step action chunks. All four controllers were evaluated on the same 30 seeds.

It did not. MLP behaviour cloning matched the SAC expert's 96.7% landing-success rate. The diffusion student reached 90.0%, and the discretised DQN reached 86.7%. Their mean returns were close: 231.0 for the MLP, 228.5 for diffusion, 233.4 for DQN, and 233.2 for SAC.

Diffusion was also slower and less smooth than the MLP: 3.766 ms per action versus 0.083 ms, and 0.5676 action smoothness versus 0.1171. The SAC demonstrations were close to deterministic, so there was little multimodal behaviour for diffusion to model.

The DQN, SAC, and MLP were each trained with one seed. The diffusion student was retrained with three seeds, but every model still used the same fixed hyperparameters and the main comparison covered only LunarLanderContinuous-v3. The paper could not show how the result changes across environments or action-space dimensions.

What I took from the paper was narrower: continuous or multidimensional actions alone do not make diffusion useful. The demonstration data needs to contain several valid ways to act from the same state. Here, plain behaviour cloning was the better fit.