About SARA

SARA is a research project exploring how far a compact, fully custom conversational language model can go when trained on a single consumer GPU.

The goal is not to scale existing architectures — it is to understand what is possible when every component is designed from scratch: attention, feed-forward blocks, normalization, positional encoding, loss function, and optimizer. No component was borrowed from an existing model family.

SARA is an encoder-decoder transformer with a parallel local-context stream injected into each decoder block. Keys and values are compressed into a shared latent via a learned bottleneck, normalized, and decompressed per-head. Queries and keys are projected with a SwiGLU gate and normalized with QK-Norm before the dot product. Positional information is injected via Rotary Position Embeddings (RoPE).

The feed-forward layer uses ShutterGLU — a soft Mixture-of-Experts block with 4 experts, SwiGLU activations, and a learnable routing temperature. Each decoder block also runs a secondary branch that mixes the current decoder state with the fixed input embedding via a low-rank bilinear projection and a dynamic local convolution with a query-conditioned output gate.

Every training run is managed by SOFA — a custom meta-learning rate scheduler that treats the learning rate itself as an optimization problem. It runs a lightweight search algorithm alongside training and automatically transitions between exploration, local search, and fine-tuning phases based on real loss signals, gradient norms, and accuracy.

Token embeddings are factorized (ALBERT-style) and tied to the output logit layer. A secondary frozen synbedding, generated via DCT-based spectral encoding over each token's character-level signal, provides the model with a language-agnostic semantic prior from the very first training step — no pre-training on external corpora required.