Manshu LLM: A Character-Level Transformer Language Model Built from Scratch#
#LLM; #Transformer; #PyTorch; #FastAPI
Every LLM component (embedding, GQA, RoPE, sliding window, SwiGLU, RMSNorm, LM head) implemented from scratch, with both local and HTTP inference services.
Repository: (not public yet; local project manshu_llm)
The flowchart below summarizes the main pipeline, “data preparation → training → inference/serving”, along with the data flow of the model’s forward pass.
graph TD
A[Tiny Shakespeare] --> B[download_data]
B --> C[tokenize_data]
C --> D[train.npy val.npy vocab]
D --> E[train.py]
E --> F[checkpoints]
F --> G[infer.py]
F --> H[serve.py]
H --> I["/generate"]
H --> J["/health"]
K[Input IDs] --> L[Token Embedding]
L --> M[Transformer Block x N]
M --> N[RMSNorm LM Head]
N --> O[Logits]
Project Overview#
Manshu LLM (v0.1.0) is one of my flagship projects, with two core emphases: implementing every LLM component from scratch (embedding, GQA, RoPE, sliding-window attention, SwiGLU, RMSNorm, LM head, etc.) and providing a complete inference service (local scripts plus a FastAPI HTTP API with greedy/sampling decoding and health checks). The project covers the full pipeline of “data preparation → training → checkpoint management → inference service”, demonstrating a deep understanding of Transformers and LLM engineering. It uses character-level modeling, so no BPE or WordPiece is needed and the vocabulary is derived from the data. The default dataset is Tiny Shakespeare, which suits small-scale experiments and teaching.
Tech stack: a decoder-only Transformer in PyTorch featuring GQA (grouped-query attention), RoPE (rotary position embeddings), sliding-window attention, a SwiGLU FFN, and Pre-RMSNorm, aligned with mainstream architectures such as Llama. Training uses AdamW, cosine/linear learning-rate scheduling, and gradient accumulation, all configured through YAML (configs/model.yaml, train.yaml, data.yaml). Inference supports greedy decoding and temperature / top-k / top-p sampling. The service is built with FastAPI + Uvicorn, exposing POST /generate (text generation) and GET /health (health check), with API docs at /docs and /redoc. Docker deployment is supported (Dockerfile, docker-compose.yml, and build.sh under docker/). Project layout: llm/ (model, data, training, inference, utils), scripts/ (download_data, tokenize_data, train, infer, check_model), configs/, data/ (raw, processed, vocab), checkpoints/, and serve.py at the root.
Highlights#
- Every LLM component built from scratch: no reliance on Hugging Face or other model libraries. Token embedding, GQA, RoPE, sliding-window attention, SwiGLU, RMSNorm, and the LM head are all hand-written, which builds a solid grasp of the math and implementation details while staying aligned with architectures like Llama.
- Complete inference service: local inference (
infer.py, greedy/temperature/top-k/top-p sampling) and an HTTP inference service (FastAPIPOST /generate,GET /health, Pydantic request/response models, CORS, Docker deployment) that can serve text generation directly. - End-to-end pipeline: data preparation → training → checkpoint management → inference service; a single project demonstrates the full engineering path “from raw corpus to a callable text-generation service”.
- Reproducible and configurable: YAML separates model/training/data configs, and training and inference share the same keys and loading logic, making experiments easy to reproduce and hyperparameters easy to switch.
- Character-level simplicity: a small vocabulary and simple implementation allow fast validation of architecture and hyperparameters on small corpora like Tiny Shakespeare, ideal for teaching and for walking through in interviews.
My Role and Contributions#
I handled full-stack development of the model, training, inference, and serving, with a focus on:
- Implementing the LLM components: built the decoder-only Transformer’s token embedding, GroupedQueryAttention, RoPE, sliding-window attention, SwiGLU, RMSNorm, and LM head from scratch, including weight initialization and options like tie_word_embeddings; every submodule is hand-written with no model-library dependencies.
- Providing the inference service: implemented local inference (
infer.py+ greedy_decode / sample_decode with temperature, top_k, top_p) and an HTTP inference service (FastAPIPOST /generate,GET /health, Pydantic request/response models, model loading in lifespan, CORS, Docker deployment) offering text generation and health checks out of the box. - Data and training: CharTokenizer, CharDataset, collate_fn; a Trainer (gradient accumulation, checkpoint management, validation), AdamW, learning-rate scheduling, and loss/optimizer wrappers.
| Category | Technology | Purpose |
|---|---|---|
| Deep learning framework | PyTorch | Model implementation, training, and inference |
| Model architecture | Decoder-only Transformer | Backbone of the character-level LM |
| Attention | GQA + RoPE + sliding window | Grouped queries, positional encoding, long-context optimization |
| FFN / normalization | SwiGLU + RMSNorm | Feed-forward network and Pre-Norm |
| Training | AdamW + Cosine/Linear LR + gradient accumulation | Optimizer and learning-rate scheduling |
| Configuration | YAML (configs/) | Model, training, and data configs |
| API service | FastAPI + Uvicorn | Text generation and health checks |
| Deployment | Docker (optional) | Containerized runtime |
Background#
Implementing a language model from scratch builds a deep understanding of every part of a Transformer: embeddings, attention, positional encoding, FFN, loss, and sampling. Simply calling existing libraries makes it hard to master the details of GQA, RoPE, and sliding windows, or their impact on memory and long sequences. This project implements every LLM component from scratch and ties “model architecture → data pipeline → training loop → inference service (local + HTTP API)” into a single runnable, configurable codebase, making it easy to experiment with different layer counts, window sizes, and sampling strategies. It serves as a flagship project demonstrating deep understanding of Transformers and LLM engineering, highlighting the full-stack ability to “implement every component and ship an inference service” on a resume or portfolio.
Why character-level? Character-level modeling needs no BPE/WordPiece; the vocabulary is derived from the data and is small, so implementation and debugging are simple, and architecture and hyperparameters can be validated quickly on small corpora like Tiny Shakespeare. To move to subword tokenization later, only the tokenizer and vocabulary path need to change; the existing training and inference pipelines can be reused.
Choosing character-level over subword/word-level prioritizes interpretability and fast experiment iteration in this project. For larger corpora and stronger results, subword tokenization and larger configurations can be plugged into the same codebase.
System Architecture and Design Choices#
The overall design is single-machine training with a single-process service.
Data flow: raw text (e.g., data/raw/tiny_shakespeare.txt) is read by tokenize_data.py → CharTokenizer.build_vocab(text) counts characters to build the vocabulary (<unk>=0 + characters sorted by frequency) → tokenizer.encode(text) produces a 1D array of token IDs → split into training/validation segments by val_ratio → saved as train.npy, val.npy (dtype int32), and char_vocab.json.
Training flow: CharDataset(data_path, tokenizer, context_window) slides a window over the 1D array; each sample is a contiguous slice of length context_window+1, with the first context_window tokens as input_ids and the last context_window as target_ids (targets are the inputs shifted right by one). collate_fn stacks equal-length samples into (batch_size, seq_len), padding with pad_token_id=0 if lengths differ. Before each batch enters the model, a causal mask (L, L) is built from seq_len; the forward pass yields logits (B, L, vocab_size), and the loss is CrossEntropyLoss(logits, target_ids) (with ignore_index=-1 support). Within the accumulation window only backward runs; once gradient_accumulation_steps is reached, gradient clipping, optimizer.step, and scheduler.step are executed. Every eval_steps the model is evaluated on the validation set and a best checkpoint is saved if validation loss improves; every save_steps a periodic checkpoint is saved, capped by save_total_limit.
Inference/serving flow: on startup, infer.py or serve reads the three YAML files via load_all_configs(config_dir), takes vocab_dir from the data config to locate char_vocab.json, and loads the CharTokenizer. It builds the Transformer from the model config (injecting vocab_size=tokenizer.vocab_size), then loads weights with load_model_only(model, checkpoint_path). At inference time the prompt is encoded (an empty prompt starts from [[0]], i.e., <unk>), greedy_decode or sample_decode is chosen based on temperature, and the output is decoded to text. The server performs this loading inside FastAPI’s lifespan, and incoming requests call the generation logic directly; results are currently returned all at once (non-streaming).
Design rationale: PyTorch makes it easy to implement each module from scratch and to do gradient checks and shape debugging. GQA reduces the number of KV heads, enabling longer sequences or larger batches at the same parameter count. RoPE encodes position into Q/K, removing the need for an explicit positional embedding table. Sliding windows limit the attention span, lowering memory and compute for long sequences. SwiGLU and RMSNorm are standard in modern LLMs and help convergence and numerical stability. FastAPI and Pydantic make it easy to define request/response models and auto-generate OpenAPI docs, simplifying integration and frontend hookup.
Architecture alignment: the attention and FFN design in this project (GQA, RoPE, sliding window, SwiGLU, RMSNorm) matches open-source LLMs such as Llama and Mistral, so the understanding gained here transfers to tuning and extending larger models and existing libraries (e.g., Hugging Face).
Key Design Rationale#
GQA (grouped-query attention)
Multiple query heads share one set of key/value heads, preserving expressiveness while reducing the KV cache and compute. This allows larger batches or sequence lengths under limited memory and matches current open-source LLM practice. Configured via
num_attention_headsandnum_key_value_heads(num_heads must be divisible by num_kv_heads).RoPE (rotary position embeddings)
Position is encoded as rotations of Q and K, so relative position is represented by angles. This supports extrapolation and longer contexts, needs no positional embedding table in the embedding layer, and is compatible with GQA and sliding windows. Configured via
max_position_embeddingsandrope_theta(default 10000.0); RoPE is built perhead_diminside each block and passed into the attention layer.Sliding-window attention
Each position attends only to tokens within a local window, reducing attention complexity from quadratic in sequence length to a function of window size, which significantly cuts memory and compute for long-sequence training and inference. Implementation: compute QK^T to get scores, then call
apply_sliding_window_mask. Withsliding_window_overlap=Truethe window is symmetric (position i attends to [max(0, i-w/2), min(L, i+w/2+1)]); withFalseit is an asymmetric causal window ([max(0, i-w+1), i+1]). The window mask is ANDed with the causal lower-triangular mask, invisible positions in scores are set to -inf, then softmax is applied and multiplied by V.SwiGLU + RMSNorm
SwiGLU strengthens the FFN via element-wise multiplication of a gate and an up-projection (gate_proj → SiLU, up_proj → element-wise multiply → down_proj). RMSNorm replaces LayerNorm to reduce compute and dependencies, and the Pre-Norm structure (norm before each sublayer, then residual) improves stability in deep networks, consistent with Llama and similar architectures.
YAML configs + unified loading
Model, training, and data parameters are separated into configs/model.yaml, train.yaml, and data.yaml, and loaded into a single dict (keys “model”, “train”, “data”) via
load_all_configs(config_dir). Training, inference, and serving share the same paths and keys, making it easy to reproduce experiments and switch device, step counts, window size, and other hyperparameters.
Features#
Health Checks#
The service exposes GET /health for liveness probes and load balancing:
- Response body (Pydantic model HealthResponse):
status(healthy / model_not_loaded),model_loaded(boolean),device(e.g., cuda:0 or cpu),model_info(optional; includes num_layers, hidden_size, total_params, vocab_size) - Startup: lifespan loads the config, tokenizer, model, and checkpoint in order; if any step fails an exception is raised and the service does not start
- Errors: if the model is not loaded at runtime,
POST /generatereturns 503 with detail “model not loaded”
Data Preparation#
- download_data.py: downloads the Tiny Shakespeare raw text from Karpathy’s char-rnn repository, saves it to
data/raw/tiny_shakespeare.txt, and prints the character count (URL:https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt). - tokenize_data.py: reads
val_ratio(default 0.1) fromconfigs/data.yaml; reads the raw text; instantiatesCharTokenizer()and callsbuild_vocab(text)(counts characters with Counter,<unk>=0, the rest sorted by frequency); saves the vocabulary todata/vocab/char_vocab.json;encode(text)produces the full list of token IDs; splits it proportionally into training/validation segments, converts them to int32 numpy arrays, and saves them asdata/processed/train.npyanddata/processed/val.npy. Input/output paths are hard-coded in the script, matching the “download first, then tokenize” workflow.
CharTokenizer (llm/data/tokenizer.py): char_to_id, id_to_char, vocab_size; build_vocab(texts) builds the vocabulary from text; encode(text) looks up each character to produce an ID list, mapping unknown characters to <unk> (id 0); decode(ids) converts an ID list back to a string; save_vocab/load_vocab read and write JSON. The vocabulary JSON is a character-to-id mapping.
Outputs: train.npy, val.npy (1D int32 arrays) and char_vocab.json (character → id). Data statistics (character count, token count, train/validation split, vocabulary size) are printed at the end of tokenize_data.
Model Architecture#
- llm/model/transformer.py: the main model. Input
input_ids(batch_size, seq_len) passes through TokenEmbedding to get (B, L, hidden_size), then N TransformerBlocks, a final RMSNorm, and finally the LM head (or the transposed shared embedding) to produce logits (B, L, vocab_size). Supportstie_word_embeddings: when True there is no separate lm_head and the output isF.linear(x, embedding.weight). - llm/model/block.py: a single TransformerBlock. First
attn_norm(x)→ GQA (with RoPE, sliding_window, mask) → residual; thenffn_norm(x)→ FFN (SwiGLU) → residual. Input and output shapes are both (B, L, hidden_size). - llm/model/attention.py:
GroupedQueryAttention(independent Q heads; shared K/V heads repeated to match the number of Q heads) andMultiHeadAttention.scaled_dot_product_attentioncomputes scores, applies the sliding-window mask (if configured), then the causal/general mask, and multiplies the softmax by V.create_causal_mask(seq_len)returns a lower-triangular boolean mask;apply_sliding_window_mask(scores, window_size, seq_len, overlap)combines the window mask with the causal mask and writes it into scores. - llm/model/ffn.py: SwiGLU: gate_proj(x) → SiLU, up_proj(x) → element-wise multiply → down_proj; output keeps the input hidden_size.
- llm/model/rope.py: RoPE built from head_dim, max_seq_len, and theta, applying rotary position encoding to Q and K.
- llm/model/norm.py: RMSNorm(dim, eps); forward computes x / sqrt(mean(x^2)+eps) * weight.
- llm/model/embedding.py: token embedding, vocab_size → hidden_size.
Core features: Pre-Norm (RMSNorm before each sublayer), GQA, RoPE, sliding window (symmetric/asymmetric), SwiGLU, RMSNorm, and optional tying of word embeddings and LM head (tie_word_embeddings). Weight initialization can be enabled via llm.utils.init.apply_llm_init (init_weights and init_std in model.yaml).
Training Pipeline#
- scripts/train.py: loads the configs (model, train, data), builds CharDataset(train_path, tokenizer, context_window) and CharDataset(val_path, …), with DataLoaders using
collate_fn(from llm.data.collate); instantiates the Transformer (with vocab_size injected into the config) and the Trainer; runs Trainer.train(). - llm/training/trainer.py:
train_step(batch): batch is (input_ids, target_ids), both shaped (batch_size, seq_len); creates a causal mask (seq_len, seq_len);logits = model(input_ids, mask=causal_mask);loss = criterion(logits, target_ids), divided by gradient_accumulation_steps before backward; when (global_step+1) % gradient_accumulation_steps == 0, runs gradient clipping (max_grad_norm, default 1.0), optimizer.step, zero_grad, and scheduler.step; global_step increments by 1 each step.evaluate(max_batches): runs a no-grad forward pass over val_loader and averages the loss.save_checkpoint(is_best): writes the file via utils.checkpoint.save_checkpoint; when is_best, names it best_model_step_.pt and deletes the old best; otherwise names it checkpoint_step_.pt and maintains a saved_checkpoints queue, deleting the oldest file when save_total_limit is exceeded. Training loop: iterates over num_epochs, running train_step per batch; every eval_steps runs evaluate and calls save_checkpoint(is_best=True) if validation loss beats best_val_loss; every save_steps, if no best was saved at that step, calls save_checkpoint(is_best=False); supports early termination via max_steps. - llm/training/optim.py: AdamW (learning_rate, weight_decay, beta1, beta2, eps); learning-rate scheduler options cosine, linear, and constant, with warmup_steps.
- llm/training/loss.py: CrossEntropyLoss(logits, targets) flattens logits and targets and calls F.cross_entropy, with ignore_index support (default -1).
CharDataset: __len__ = len(data) - context_window; __getitem__(idx) takes data[idx:idx+context_window+1], with the first context_window tokens as input_ids and the last context_window as target_ids (targets are inputs shifted right by one), returning two 1D LongTensors.
Checkpoints: best_model_step_.pt (best validation loss; only the latest is kept) and checkpoint_step_.pt (periodic; count limited by save_total_limit), both saved in checkpoint_dir from train.yaml (default checkpoints).
Inference and Sampling#
- llm/inference/generate.py:
greedy_decode(model, input_ids, max_length, device, stop_token_ids): at each step, runs a forward pass on the current generated_ids, takes the argmax of logits[:, -1, :] as the next token, and appends it to the sequence; supports a stop_token_ids list, marking a sequence finished when hit and exiting early once all are finished.sample_decode(model, input_ids, max_length, temperature, top_k, top_p, device, stop_token_ids): with temperature=0 it calls greedy_decode directly; otherwise each step divides logits[:, -1, :] by temperature, optionally applies top_k (keep the k largest logits) and top_p (nucleus: sort by probability and cut positions whose cumulative probability exceeds p), then applies softmax and samples with multinomial; also supports stop_token_ids. The current implementation runs a forward pass over the whole sequence at each step and does not use a KV cache. - scripts/infer.py: checkpoint, prompt, max_length, temperature, top_k, top_p, config_dir, device_str, and num_samples are configurable at the top of the script. It loads the config and vocabulary (vocab_dir from data_config, path vocab_dir/char_vocab.json), creates the model, and loads weights with load_model_only. A non-empty prompt is encoded and reshaped to (1, seq_len); an empty prompt uses input_ids = [[0]] (
as the start token). Greedy or sampling decoding is chosen by temperature; if num_samples>1, multiple samples are generated in a loop. After decoding, if there is a prompt and the generated text starts with it, the “full” and “new” parts are printed separately; otherwise the full text is printed, along with character and token counts.
Sampling strategy: temperature=0 means greedy; temperature>0 means sampling, combined with top_k (0 disables) and top_p (0.0 disables) to balance diversity and coherence.
Text Generation API#
- serve.py: a FastAPI app whose lifespan loads the configs, CharTokenizer(vocab_path), Transformer(model_config), and load_model_only(model, checkpoint_path), storing the model and tokenizer as globals. CONFIG sets checkpoint_path, config_dir, vocab_path, device (cuda if available, else cpu), host (0.0.0.0), and port (8000).
POST /generate: the request body is GenerateRequest (Pydantic): prompt (required, e.g. “First Citizen:\n”), max_length (default 100, ge=1, le=1000), temperature (default 0.8, ge=0, le=2), top_k (default 50, ge=0, le=200), top_p (default 0.9, ge=0, le=1), num_samples (default 1, ge=1, le=5). The response body is GenerateResponse: prompt, samples (a list, each with text and tokens), and model_info (checkpoint, device, total_params, num_layers, hidden_size). temperature=0 calls greedy_decode, otherwise sample_decode; the output is decoded with tokenizer.decode, packed into a list of GeneratedSample, and returned. CORS is configured (allow_origins=["*"]) for cross-origin frontend access. Results are currently returned all at once, not streamed.GET /returns a welcome message and the endpoint list (generate, health, docs, redoc). - If the model is not loaded,
/generatereturns 503 with detail “model not loaded”; errors during generation return 500 with detail “generation failed: …”.
Running and Deployment#
Typical workflow: run download_data.py to download Tiny Shakespeare, then tokenize_data.py to produce train.npy, val.npy, and char_vocab.json; then run scripts/train.py to train, with checkpoints written to checkpoint_dir (default checkpoints/). After training, use scripts/infer.py for local inference, or start serve.py to expose POST /generate and GET /health. For Docker deployment, see the docker/ directory (Dockerfile, docker-compose.yml, build.sh).
Quick start: after training, run python serve.py and open http://localhost:8000/docs to try POST /generate (text generation) and GET /health (health check) interactively; the root path GET / returns a welcome message and the endpoint list.
Helper Scripts and Configuration#
- scripts/check_model.py: inspects model structure or weights (e.g., parameter count, shapes) for debugging and alignment.
- configs/model.yaml: num_hidden_layers, hidden_size, num_attention_heads, num_key_value_heads, intermediate_size, max_position_embeddings, rope_theta, sliding_window, sliding_window_overlap, rms_norm_eps, tie_word_embeddings, init_weights, init_std, etc.
- configs/train.yaml: learning_rate, weight_decay, beta1, beta2, eps, lr_scheduler (cosine/linear/constant), warmup_steps, max_steps, batch_size, gradient_accumulation_steps, max_grad_norm, save_steps, eval_steps, save_total_limit, device, num_epochs, checkpoint_dir, use_amp, amp_dtype, etc.
- configs/data.yaml: processed_dir, vocab_dir, context_window, num_workers, pin_memory, val_ratio, etc.
Technical Challenges and Solutions#
During implementation, targeted designs addressed masking and causality, training stability and long sequences, and a unified entry point for inference and serving, enabling stable training on limited resources and a consistent external service interface.
Sliding-Window Attention and Causal Masking#
Challenge: the mask must guarantee causality (each token sees only the past) while restricting attention to a local window, so it has to satisfy both the lower-triangular and window-boundary constraints and fit the Q/K/V shapes of RoPE and GQA. If window and causal masks are applied separately, ordering and broadcasting must be consistent to avoid missing or duplicate masking.
Solution: in scaled_dot_product_attention, first compute QK^T / sqrt(head_dim) to get scores (B, H, L, L). If sliding_window is configured, call apply_sliding_window_mask(scores, window_size, seq_len, overlap), which builds window_mask (symmetric window [i-w/2, i+w/2+1] when overlap=True, asymmetric [i-w+1, i+1] when False), ANDs it with the causal lower-triangular mask, and sets invisible positions in scores to -inf. If an external mask (e.g., a causal mask) is also passed, scores are masked_fill’ed according to its boolean values (False becomes -inf). This applies the window first, then causality, and an (L, L) mask is unsqueezed to (1, 1, L, L) to match the scores dimensions. Each block passes sliding_window and sliding_window_overlap from the config, and the attention layer uniformly uses GQA + RoPE + this masking logic.
Training Stability and Long Sequences#
Challenge: deep Transformers and longer context_window values can cause unstable gradients or out-of-memory errors; large batches are infeasible on a single GPU, and the loss curve needs to be smooth.
Solution: Pre-RMSNorm plus gradient clipping (max_grad_norm, default 1.0), running clip_grad_norm_ before each optimizer.step. AdamW (with weight_decay) and a cosine learning-rate schedule with warmup_steps avoid an overly high early learning rate. gradient_accumulation_steps accumulates gradients over smaller per-step batches before updating, emulating a larger batch while keeping per-step memory under control. Sliding windows limit the attention span, allowing a larger context_window or batch_size within the same memory. The config optionally enables use_amp and amp_dtype (e.g., bfloat16) to further save memory and speed things up. The loss can use ignore_index (e.g., -1) for padding positions; in the current dataset equal-length samples are stacked directly by collate, so with no padding nothing needs to be ignored. If variable lengths are supported in the future, pad positions in target_ids can be filled with -1.
Unified Entry Point for Inference and Serving#
Challenge: both infer.py and serve.py must load the same model and tokenizer, consistent with the config and vocabulary path used in training, to avoid scattered configs causing load failures or dimension mismatches (e.g., vocab_size or hidden_size inconsistent with the checkpoint).
Solution: both read the three YAML files via load_all_configs(config_dir), yielding model, train, and data sub-dicts. For the vocabulary path, infer takes vocab_dir from data_config and joins vocab_dir / "char_vocab.json", while serve’s CONFIG hard-codes vocab_path (e.g., data/vocab/char_vocab.json). Before constructing the model, model_config[“vocab_size”] is set to tokenizer.vocab_size and passed into Transformer(config), keeping it consistent with training (the training script likewise injects vocab_size from the tokenizer). The checkpoint path is configured separately in serve’s CONFIG and in the variables at the top of infer, and must match the training output filename (e.g., best_model.pt or best_model_step_*.pt). Training, inference, and serving thus share the same config keys and loading logic, differing only in the entry point (script variables vs. CONFIG).
Summary#
As a flagship personal project, Manshu LLM highlights two things: first, implementing every LLM component from scratch (token embedding, GQA, RoPE, sliding-window attention, SwiGLU, RMSNorm, LM head, etc.) without relying on model libraries; second, providing a complete inference service (local infer.py plus FastAPI POST /generate and GET /health) with greedy/sampling decoding and health checks, ready to serve text generation. The project covers the full pipeline from data preparation, training, and checkpoint management to the inference service, with YAML configuration and Docker deployment options.
Future directions: add a KV cache to inference and the API to speed up long-sequence generation; support streaming responses (e.g., SSE); wrap the model and tokenizer in Hugging Face-style from_pretrained/save_pretrained; and add more datasets and evaluation metrics to compare architectures and hyperparameters.