NOTES / READING LIST AND GLOSSARY

Foundational papers

Eleven papers behind the topics on this site, each with a one-line takeaway in my own words and where it shows up in these notes.

UPDATED 18 NOV 2026#papers#reading

Diagram: timeline of foundational papers Diagram: my own timeline, by first arXiv submission.

Summaries below are mine, based on the abstracts and my reading; open the papers for the real detail. Order is chronological.

  1. Attention Is All You Need (2017). Introduces the Transformer, which drops recurrence for attention and trains faster in parallel.1 Background for LLM fundamentals.
  2. Retrieval-Augmented Generation (2020). Combines a generator with a retriever over external documents, so knowledge can be updated and inspected.2 See RAG overview.
  3. LoRA (2021). Adapts a large model by training small low-rank matrices instead of all weights.3 See LoRA and QLoRA.
  4. Chain-of-thought prompting (2022). Showing worked reasoning steps in the prompt improves performance on reasoning tasks for large models.4
  5. InstructGPT (2022). Fine-tunes with human feedback; labellers preferred a much smaller tuned model to a far larger base model.5 See preference tuning.
  6. ReAct (2022). Interleaves reasoning and actions so a model can use tools and update its plan.6 See ReAct and planning.
  7. Toolformer (2023). A model learns, in a self-supervised way, when to call simple APIs.7 See tool calling.
  8. Direct Preference Optimization (2023). Optimises preferences without a separate reward-model and RL loop.8
  9. Lost in the Middle (2023). Models use information at the start or end of a long input better than the middle.9 See long-context trade-offs.
  10. PagedAttention (2023). Manages the KV cache in pages to waste less memory when serving.10 See KV cache.

Findings in older papers may not hold for newer models; treat them as ideas to test with your own evals.

Sources

  1. [1]Attention Is All You Need — Vaswani et al. · 2017PARAPHRASED
  2. [2]Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al. · 2020PARAPHRASED
  3. [3]LoRA: Low-Rank Adaptation of Large Language Models — Hu et al. · 2021PARAPHRASED
  4. [4]Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Wei et al. · 2022PARAPHRASED
  5. [5]Training language models to follow instructions with human feedback — Ouyang et al. · 2022PARAPHRASED
  6. [6]ReAct: Synergizing Reasoning and Acting in Language Models — Yao et al. · 2022PARAPHRASED
  7. [7]Toolformer: Language Models Can Teach Themselves to Use Tools — Schick et al. · 2023PARAPHRASED
  8. [8]Direct Preference Optimization, your language model is secretly a reward model — Rafailov et al. · 2023PARAPHRASED
  9. [9]Lost in the Middle: How Language Models Use Long Contexts — Liu et al. · 2023PARAPHRASED
  10. [10]Efficient memory management for large language model serving with PagedAttention — Kwon et al. · 2023PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.