Fit a LoRA paramerized by a VAE labels: experimental i.e. so we can sample over LoRAs in the neighborhood