The independent artificial intelligence magazine
The future of AI may belong not only to the largest laboratories, but also to researchers who learn how to achieve more intelligence with less compute.
Read the full story →
Self-attention, positional encoding and multi-head attention explained without making the original paper feel lifeless.
Read entry »
Mixed precision, gradient accumulation, checkpointing and realistic experiments for a 12GB consumer GPU.
Read entry »
Dataset quality, duplication, contamination and why larger datasets do not automatically produce better intelligence.
Coming soon »
Why allowing a model to spend more compute during inference may change the meaning of scaling.
Coming soon »Experiment No. 001: Can better data and training decisions improve a small language model more than simply adding parameters?
| 28M Parameters | 6 Layers | 12GB VRAM |