HardGPT

HardGPT

A character-level GPT running entirely on a TI MSPM0G3507 Cortex-M0+ — no host PC, no network, no accelerator. Press Generate and the chip composes fake-Shakespeare one character at a time on a 16x2 LCD, beeping at the start and end. Three buttons, three LEDs, an active buzzer, and a dollar's worth of flash.

The model is a 3-layer transformer (4 heads, 48 embedding dims, block size 48, 65-character vocab) trained on the tiny-shakespeare corpus. Weights are int8 symmetric per-output-channel quantized so the whole network fits in flash, and the KV cache is also int8 to keep SRAM use sane during generation.

Repo layout

HardGPT/
├── main.c              firmware entry point — UI, transformer inference loop
├── lcd.h               header-only HD44780-via-PCF8574 driver
├── weights.h           int8 weights + vocab, generated by training/export
├── hardgpt.syscfg      sysconfig project: clocks, GPIO, I2C
├── targetConfigs/      CCS debug configs for the launchpad
├── pcb/                kicad project for the board
├── img/                photos used in this readme
└── training/           offline training + weight export pipeline

How it works

The whole firmware lives in a single main.c because the model is small enough that splitting it would just add ceremony. Walking through it top to bottom:

Activation and KV-cache buffers

A handful of static SRAM arrays hold the activations of one token at a time:

• x, x_residual, x_norm — the running embedding vector (length N_EMBD)
• qkv — the combined query/key/value projection (length 3 * N_EMBD)
• ffn_hidden — the 4×-wide MLP hidden state
• logits — the output distribution over the 65-character vocabulary
• x_q — int8 scratch buffer used whenever we need to quantize an activation

The KV cache is int8 kv_cache[N_LAYER][BLOCK_SIZE][2 * N_EMBD] with float kv_scales[...] alongside, so each cached K and V vector keeps its own quantization scale. Storing it as int8 cuts the cache footprint by ~4× versus float.

Math primitives

Everything that touches the hot path uses fast, approximate floats:

• quantize_vec — symmetric int8 quantization of a vector with one shared scale, used to feed activations into the int8 matmul.
• matvec_i8i8_pc — the workhorse. Multiplies an int8 weight matrix (with per-row scales) against an int8 activation vector (with one scale) and accumulates into int32, then converts to float at the end. This single routine handles QKV projection, attention output projection, and both MLP matmuls.
• fast_invsqrt — the famous Quake 0x5f3759df trick + one Newton-Raphson step. Used by both LayerNorm and the attention scaling factor.
• fast_expf — Schraudolph's bit-hack exp. Wildly inaccurate in absolute terms but plenty good once softmax normalizes.
• fast_tanhf — a Padé approximation, ~1 % error in [-3, 3], used inside GELU.
• layernorm, gelu, softmax — built on top of the above.

The transformer block

Layer is just a struct of pointers into weights.h so the block code can work with L->attn_qkv_w etc. instead of macro-mangled names. init_layers() wires layer 0/1/2 once at boot.

block_forward(layer_i, pos) does one transformer block:

1. Save the residual, layernorm into x_norm.
2. Quantize x_norm, run the QKV matmul, split into q, k, v.
3. Quantize the new k and v and store them at kv_cache[layer_i][pos].
4. For each head: dot the head's q against every cached k_t (dequantizing on the fly), scale by 1/sqrt(head_dim), softmax, then accumulate the weighted sum of cached v_t.
5. Quantize the attention output, run the output projection, add the residual.
6. LayerNorm again, run the 4× MLP with GELU, project back, add the residual.

forward(token_id, pos) adds the token + position embeddings, calls block_forward for each layer, runs the final LayerNorm, and reuses the embedding table as the LM head (weight tying) to produce logits.

slide_kv_cache() shifts the cache left by one when generation runs past BLOCK_SIZE, so the model always sees the most recent BLOCK_SIZE - 1 tokens.

Sampling

Two sampling modes:

• sample_argmax — greedy.
• sample_temp(temperature) — divide logits by temperature, softmax, draw from the distribution using a small linear-congruential RNG.

The probability of the chosen token is stashed in last_token_prob so the "confidence" LED can light up when the model is sure.

LCD typewriter

LcdView holds a 2-row shadow buffer plus the current (row, col). As characters arrive, lcd_view_putchar writes them straight to the LCD and mirrors them into the shadow buffer. When the cursor walks off row 1, we copy row 1 up to row 0, blank row 1, and continue — the result is a smooth typewriter that scrolls upward indefinitely.

UI / main loop

Boot sequence:

1. SYSCFG_DL_init() brings up clocks/GPIO/I2C from sysconfig.
2. init_layers() wires the layer pointer table.
3. lcd_init() runs the HD44780 wake-up dance and switches it into 4-bit mode.
4. A short buzzer beep confirms power-on.

Then the forever loop:

• Idle screen says HardGPT v1.0 / Press Generate!. The buttons let you cycle through seeds (Random, KING, ROMEO, LORD, HENRY) or peek at the temperature.
• Generate: beep, light up the "running" LED, clear the KV cache, print the seed name as a cosmetic prefix, then loop:
  • run forward(token, pos)
  • sample the next token
  • print it through the LCD typewriter
  • toggle LED 2 (one tick per token), light LED 3 if last_token_prob > 0.5
  • increment pos; slide the KV cache if we hit BLOCK_SIZE
  • bail out early if Generate is pressed again (interrupts the run)
• After 300 tokens (or interrupt), beep twice, wait, and return to idle.

Hardware

Pin assignments live in hardgpt.syscfg; the SDK generates ti_msp_dl_config.{c,h} into Debug/ from it.

Training

The model is trained offline using a trimmed-down fork of Andrej Karpathy's nanoGPT. Pretty much all of the training code (model.py, train.py, sample.py, configurator.py, the data prep scripts, the overall project structure) is his — go give that repo a star. The training pipeline lives in training/ and is a UV-managed Python project.

training/
├── pyproject.toml
├── uv.lock
├── nanoGPT/                  Karpathy's nanoGPT (trimmed to what we use)
│   ├── model.py              GPT model definition
│   ├── train.py              training loop
│   ├── sample.py             inference / sampling reference
│   ├── configurator.py
│   ├── config/train_micro.py our tiny config (3 layers, 48 dims, ...)
│   ├── data/shakespeare_char/ tiny-shakespeare, char-level tokenizer
│   ├── out-micro/ckpt.pt     trained checkpoint
│   ├── LICENSE               nanoGPT MIT license
│   └── README.md             upstream nanoGPT README
├── export/
│   ├── export_weights.py     converts ckpt.pt → weights.h
│   └── weights.h             generated header (also copied into the firmware)
└── inference_c/              desktop C reference impl, used to sanity-check
    ├── main.c                  the on-chip math before flashing
    └── weights.h

End-to-end flow:

training/nanoGPT/data/shakespeare_char/prepare.py
        │
        ▼     (tokenize)
   train.bin / val.bin
        │
        ▼     (uv run python nanoGPT/train.py config/train_micro.py)
training/nanoGPT/out-micro/ckpt.pt
        │
        ▼     (uv run python export/export_weights.py)
training/export/weights.h ── copied to ──▶ HardGPT/weights.h
                                                   │
                                                   ▼  (CCS build + flash)
                                            on-chip generation

config/train_micro.py sets the architecture used by the firmware (n_layer=3, n_head=4, n_embd=48, block_size=48, dropout 0.2) and trains for 30k iterations on shakespeare_char with the AdamW recipe nanoGPT ships with.

export_weights.py does symmetric int8 per-output-channel quantization of every weight matrix, keeps LayerNorm gains as float (they're tiny), and emits a single weights.h containing:

• the architecture #defines (VOCAB_SIZE, N_LAYER, N_HEAD, N_EMBD, HEAD_DIM, BLOCK_SIZE)
• the character vocab[] lookup table
• token / position embeddings + per-row scales
• per-layer attention QKV, attention proj, MLP fc, MLP proj weights and scales
• per-layer LayerNorm gains as float
• the final LayerNorm gain

That header is exactly what main.c #includes. The on-chip int8 matmul in matvec_i8i8_pc is the dequantization counterpart to what export_weights.py does at quantization time.

Reproducing

cd training
uv sync                                       # install torch, numpy, etc.
uv run python nanoGPT/data/shakespeare_char/prepare.py
uv run python nanoGPT/train.py config/train_micro.py
uv run python export/export_weights.py
cp export/weights.h ../weights.h              # firmware picks it up on next build

Then build the firmware in CCS (Project → Clean → Build) and flash to the LP-MSPM0G3507.

Credits

• Andrej Karpathy — the entire training stack (nanoGPT) is his work, used here under its MIT license. See training/nanoGPT/LICENSE.
• Tiny-Shakespeare corpus from char-rnn, also via nanoGPT.
• HardGPT firmware (main.c, lcd.h, weights.h packaging, sysconfig, board bring-up) is original to this project.