Do Value Vectors in Deep Layers Need Context from the Residual Stream?
The success of the transformer architecture is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces…