This page is the messy part of the website: hypotheses, failed runs, suspicious loss curves, configuration changes and notes written before I fully understand what happened.

Experiment 001

The first experiment compares two small autoregressive language models with the same architecture and token budget. The difference will be the dataset preparation and training recipe.

SettingBaselineImproved run
Parameters28M28M
Context256256
Layers66
Heads66
DataRaw-cleanedFiltered + deduplicated

Configuration

model: vocab_size: auto context_length: 256 embedding_dim: 384 num_layers: 6 num_heads: 6 dropout: 0.1 training: precision: fp16 optimizer: adamw gradient_clip: 1.0 batch_size: 16 accumulation_steps: 4

Run Log

Run 001-A

Baseline configuration. First objective: verify that data loading, masking, loss calculation and checkpoint saving work correctly. No optimization tricks until the full pipeline is stable.

Run 001-B

Enable mixed precision and measure peak memory use. Compare throughput against the baseline before changing any other variable.

Run 001-C

Introduce cleaned and deduplicated data. Keep the training token count fixed so the comparison focuses on data quality rather than training duration.

Important: Do not change architecture, data and optimizer settings simultaneously. That would make the result impossible to interpret.

Results Placeholder

Add validation loss, sample generations, throughput, peak VRAM and training time here after each run. Screenshots can be stored in assets/experiments/.