labels: experimental, tooling
- subtract checkpoint from base model to get delta weights
- SVD the delta weights
- that's it, we're done.
even if this is lossy, it might still be a good way to "warm start" LoRA training with a strong but noisy (i.e. low rank) prior
Looks like LoCON does this already? https:/KohakuBlueleaf/LyCORIS