reading

• The Way of Code +

  On vibe coding by Anthropic with the words based on Rick Rubin.

• Unveiling Language-Specific Features in Large Language Models +

  A paper by Deng et al. (2025) introduce a new metric to assess the monolinguality of features obtained from SAEs. They further use these features to enhance steering vectors, which allows for full control over the generation language.

• YaRN: Efficient Context Window Extension of LLMs +

  A paper by Peng et al. (2023) proposes an effective way of mathematically stretching existing position encodings to handle longer sequences (instead of training the whole model). It requires only a small fraction of original training data (10x) and training steps (2.5x) than previous methods.

• Language Models Solve Math with a Bag of Heuristics +

  A paper by Nikankin et al. (2024) discovers a sparse set of neurons that implement simple heuristics, which identify an input pattern and outputs the appropriate answer.

• Good vs Great Animations +

  A "guide" by Emil Kowalski on how to make your websites stand out with great animations.

• 70% Size, 100% Accuracy: Lossless LLM Compression +

  A paper by Zhang et al. (2025) introduces DFloat11 that compresses any BFloat16 model to 70% of its size while keeping 100% performance on any task.

• DCAD-2000: Data Cleaning as Anomaly Detection +

  A paper by Shen et al. (2025) compiles a new dataset for 2282 languages. They get various statistical features (landuage identification score, word repetition, perplexity score, and special character ratio) from each document to assess quality and then use anomoly detection (language-agnostic) methods to identify and remove outliers that deviate from typical document quality metrics.

• Intro to RLHF and Post-Training +

  A book by Nathan Lambert on the reinforcement learning from human feedback and post-training in the context of language models.

• Latents +

  A blog post by Sander Dieleman on the latent representations learned through representation learning and reconstruction. "Three main aspects to consider when designing latent spaces are capacity (how many bits of information are encoded in the latents), curation (which bits from the input signals are retained) and shape (how this information is presented)".

• The Second Half +

  Shunyu Yao argues that we now have the proper RL priors (language pre-training) and RL environment (language reasoning), and as a consequence we can use basic RL algorithms for building strong AI. He argues that it is "the second half" for AI currently, where we should be focusing on developing novel evaluations that will reflect real world utility.

• When Is Multilinguality a Curse? +

  Chang et al. (2025) show that adding multilingual data, as dataset sizes increase, starts to hurt performance both for low-resource and high-resource languages and suggest that more targeted models can be more beneficial. Related languages can help low-resource languages only if syntactically similar.

• Does BERT Rediscover the Classical NLP Pipeline? +

  Niu et al. (2022) introduce a novel probe method, GridLoc, and show that layer-depth is not the best way to explain the inner workings of BERT.

• BERT Rediscovers the Classical NLP Pipeline +

  Tenney et al. (2019) show that BERT represents the steps of the classical NLP pipeline; layers responsible for each step appear to be in the expected order (POS tagging, parsing, NER, semantic roles, coreference).

• The BELEBELE Benchmark +

  Bandarkar et al. (2024) build a multilingual multiple-choice machine reading comprehension dataset for 122 language variants based on FLORES-200.

• SliceGPT: Compress Large Language Models by Deleting Rows and Columns +

  Ashkboos et al. (2024) introduce a novel post-training sparcification method that applies orthogonal transformations to transformer layers (which leave the model unchanged) and then "slice off" lest important rows and columns.

• SuperBPE: Space Travel for Language Models +

  Liu et al. (2025) introduce SuperBPE that learns both subwords and superwords, which uses up to 33% less tokens than a regular BPE tokenizer and provides a better performance on 30 downstream tasks. SuperBPE captures comon multi-word expressions that function as a single unit.

• Compression Laws for Large Language Models +

  Sengupta et al. (2025) show that "the test cross-entropy loss increases quadratically with the compression ratio, whereas performance on downstream tasks declines only linearly." They apply calibration-free and calibration-based structured prunning to Qwen and Llama ranging from 0.5B to 14B.

• Transformer Models without Positional Encodings Still Learn Positional Information +

  Haviv et al. (2022) show that decoder-only LMs without posititional encoding still seem to learn the positional information (possbily from causal attention). This does not hold for decoder-only models.

• Transformer Feed-Forward Layers Are Key-Value Memories +

  Geva et al. (2021) demonstrate that MLP layers resemble key-value memories, where keys detect specific input patterns and values provide a distribution over possible next words that commonly follow the detected pattern.

• Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders +

  Galichin et al. (2025) use SAEs to find the features responsible for "reasoning" in the distilled Deepseek-R1-based Llama model. They show that implifying these features makes the thinking process longer and improves performance on reasoning related tasks.

• From System 1 to System 2: A Survey of Reasoning Large Language Models +

  Li et al. (2025) talk about reasoning and non-reasoning models as System 1 (fast decision making) and System 2 (logical reasoning with more accurate judgements) immitators.

• Language-specific Neurons Do Not Facilitate Cross-Lingual Transfer +

  Mondal et al. (2025) identify language-specific neurons and perform test-time interventions on those neurons (fine-tuning only language-specific neurons and other variations). They demonstrate that this approach is not effective in cross-lingual knowledge transfer on XNLI and XQuAD. Finally, they hypothesize that language-specific neurons lack independence due to their polysemantic nature.

• How to Test Generation Capabilities of LLMs? +

  lm_evaluation_harness package by EleutherAI. 60 standard academic benchmarks and all HuggingFace models supported.

• How do Large Language Models Handle Multilingualism? +

  A paper by Zhao et al. (2024) hypothesizes that mutilingual models go through three stages when prompted with non-English queries - (1) LLMs understand a prompt by converting linguistic features into a unified representation, (2) they reason in English (with self-attention) and incorporate multilingual knowledge to get factual information (with feed-forward networks), and (3) generate responses in the language of the initial prompt. They further propose Parallel Language-specific Neuron Detection that, unlike other methods, works with unlabeled data and then fine-tune language-specific neurons on the target language.

• Do Multilingual LLMs Think in English? +

  A paper by Schut et al. (2025) suggests that LLMs make decision concerning semantically-loaded words in the intermediate layers in English whereas other types of words are routed through non-English language representations.

• The Right Philosophy for Our Times +

  An essay (by Adam Dhalla) on transcendentalism, a philosophical movement placing an emphasis on self-reliance and individuality. Nurturing these elements can "build mental resilience at the individual level of our society and increase the strength of our network".