Cure the headache of Transformers via Collinear Constrained Attention
Paper • 2309.08646 • Published • 14
Positional embeddings
-
-
YaRN: Efficient Context Window Extension of Large Language Models
Paper • 2309.00071 • Published • 82 -
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
Paper • 2309.10400 • Published • 26 -
Dynamically Relative Position Encoding-Based Transformer for Automatic Code Edit
Paper • 2205.13522 • Published • 1 -
The Impact of Positional Encoding on Length Generalization in Transformers
Paper • 2305.19466 • Published • 2 -
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings
Paper • 2305.13571 • Published • 2 -
Position Prediction as an Effective Pretraining Strategy
Paper • 2207.07611 • Published • 1 -
Transformer Language Models without Positional Encodings Still Learn Positional Information
Paper • 2203.16634 • Published • 5 -
CONFLATOR: Incorporating Switching Point based Rotatory Positional Encodings for Code-Mixed Language Modeling
Paper • 2309.05270 • Published • 1 -
Your Transformer May Not be as Powerful as You Expect
Paper • 2205.13401 • Published • 1 -
Language Modeling with Deep Transformers
Paper • 1905.04226 • Published • 1 -
Dynamic Position Encoding for Transformers
Paper • 2204.08142 • Published • 1 -
Uncovering hidden geometry in Transformers via disentangling position and context
Paper • 2310.04861 • Published -
Transformers Can Do Arithmetic with the Right Embeddings
Paper • 2405.17399 • Published • 54