Accuracy drop after fuse_layer_norms

[Niko-zyf]

I encountered an unexpected precision loss while using Quarot. I conducted comparison experiments on LLaMA-2-7b:

Performing w4a16 RTN quantization on the model resulted in a PPL (Perplexity) of 7.354664.
Performing only fuse_layer_norms without rotation operations resulted in a PPL of 7.641615. Based on this, the precision of these two results should be the same.
I also replaced RMSN with LlamaRMSNorm in fuse_layer_norms and set self.weight to 1, which resulted in a PPL of 9.577629.
Logically, these three results should be equal, shouldn't they?
`class RMSN(torch.nn.Module):
def init(self, hidden_size, eps=1e-6):
"""
LlamaRMSNorm is equivalent to T5LayerNorm
"""
super().init()
self.weight = torch.nn.Parameter(torch.ones(hidden_size)).to(torch.float16)
self.variance_epsilon = eps

def forward(self, hidden_states):
    input_dtype = hidden_states.dtype
    hidden_states = hidden_states.to(torch.float32)
    variance = hidden_states.pow(2).mean(-1, keepdim=True)
    hidden_states =self.weight * hidden_states * torch.rsqrt(variance + self.variance_epsilon)
    return hidden_states.to(input_dtype)

`

[tellyoung]

The reason for removing the non-equivalence is that the weight in rms cannot be multiplied by the end of XQ. Otherwise, Q.T cannot be canceled.