Manshu Finetune: Chinese Medical QA LLM Fine-Tuning with Unsloth#
#LLM; #Finetune; #Unsloth; #LoRA; #Qwen
LoRA fine-tuning of Qwen2.5-7B-Instruct with Unsloth on a Chinese medical Q&A dataset to strengthen domain-specific question answering.
Repository: (not yet public; local project manshu_finetune)
The flowchart below summarizes the main pipeline: data preparation → fine-tuning → inference validation.
graph TD
A[Chinese-medical-dialogue-data] --> B[Data Cleaning & Format Conversion]
B --> C[Alpaca-format JSON]
C --> D[Dataset.from_json]
D --> E[Tokenize & Format]
E --> F[Unsloth FastLanguageModel]
F --> G[LoRA Adapter Injection]
G --> H[SFTTrainer Fine-Tuning]
H --> I[Save LoRA Adapter]
I --> J[Merge into Full Model]
J --> K[Inference Validation]
K --> L[GGUF Export]
Project Overview#
Manshu Finetune (v0.1.0) is one of my flagship projects. It focuses on two things: LoRA fine-tuning of Qwen2.5-7B-Instruct with the Unsloth efficient fine-tuning framework, and using a real-world medical Q&A dataset (Chinese-medical-dialogue-data, about 790K Q&A pairs) to improve professional question answering in the medical domain. The project covers the full pipeline of data cleaning → format conversion → efficient fine-tuning → model merging → inference validation → GGUF export, demonstrating a solid understanding of LLM fine-tuning engineering.
Tech stack: Unsloth as the efficient fine-tuning framework (2-5x faster training, 50%-80% less GPU memory); Qwen2.5-7B-Instruct as the base model (Alibaba’s Tongyi Qianwen family, Chinese and English); LoRA/QLoRA for parameter-efficient fine-tuning (only about 0.06% of parameters trained); Hugging Face Transformers + TRL for SFTTrainer and data processing; bitsandbytes for 4-bit quantized loading; training data from Chinese-medical-dialogue-data (covering six departments: internal medicine, surgery, pediatrics, obstetrics & gynecology, andrology, and oncology); export to GGUF for local inference with llama.cpp / Ollama.
Highlights#
- Efficient fine-tuning: Unsloth’s FastLanguageModel trains 2-5x faster than vanilla Transformers and cuts GPU memory by 50%-80%, so LoRA fine-tuning of a 7B model fits on a single RTX 4090 (24GB).
- Real medical data: The Chinese-medical-dialogue-data dataset contains about 790K real medical Q&A pairs across six departments: internal medicine (220K), surgery (116K), pediatrics (102K), obstetrics & gynecology (184K), andrology (95K), and oncology (76K).
- Parameter efficiency: LoRA (r=16, alpha=16) trains only about 42M parameters (0.06% of the total), injecting medical domain knowledge while preserving the base model’s general capabilities.
- End-to-end engineering: Covers data cleaning, format conversion, tokenization, fine-tuning, model merging, inference validation, and GGUF export, making it easy to reproduce and transfer to other domains.
- Multi-format export: Saves to Hugging Face format (for further training or deployment) and GGUF format (for local inference with llama.cpp / Ollama).
My Role and Deliverables#
I was responsible for full-stack development of data processing, fine-tuning, and inference validation, with emphasis on:
- Data cleaning and format conversion: Cleaned the raw CSV data (four columns: department, title, question, answer) and converted it to Alpaca format (instruction, input, output), handling null values, malformed characters, and overly long text.
- Fine-tuning: Loaded a 4-bit quantized Qwen2.5-7B-Instruct with Unsloth FastLanguageModel, injected a LoRA adapter (rank=16, alpha=16, target_modules including q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), and ran supervised fine-tuning with SFTTrainer.
- Inference and export: Wrote inference scripts to validate fine-tuning results, and added support for merging LoRA into a full model and exporting GGUF (q4_k_m / q8_0) for local deployment.
| Category | Choice | Purpose |
|---|---|---|
| Base model | Qwen2.5-7B-Instruct | Chinese/English instruction-tuned base |
| Fine-tuning framework | Unsloth | Efficient fine-tuning (2-5x speedup, 50%-80% memory savings) |
| Fine-tuning method | LoRA / QLoRA | Parameter-efficient fine-tuning (only 0.06% of parameters trained) |
| Quantization | bitsandbytes 4-bit | Memory optimization; trains a 7B model on a single 24GB GPU |
| Training tool | TRL SFTTrainer | Supervised fine-tuning trainer |
| Dataset | Chinese-medical-dialogue-data | 790K Chinese medical Q&A pairs |
| Export format | GGUF | Local inference with llama.cpp / Ollama |
Background#
General-purpose LLMs often fall short in the medical domain: limited understanding of medical terminology, and answers that may be inaccurate or not specific enough. Asking ChatGPT or a general Qwen model medical questions can yield overly generic or unprofessional advice. By fine-tuning on large-scale real medical Q&A data, this project teaches the model the phrasing, terminology, and reasoning patterns of medical answers, so it gives more professional and targeted responses in medical consultation scenarios.
Why Qwen2.5-7B-Instruct? The Qwen family performs strongly on Chinese tasks, and a 7B model can be fine-tuned on a single 24GB GPU with 4-bit quantization + LoRA, balancing quality and resource cost. The Instruct version is already instruction-tuned with good instruction-following ability, making it a good starting point for specializing in medical Q&A.
I chose Unsloth for its seamless compatibility with the Hugging Face ecosystem and its significant efficiency gains: 2-5x faster training and 50%-80% less GPU memory under the same configuration. This lets a single GPU handle training that would otherwise require multiple GPUs, lowering the barrier to experimentation and iteration.
Architecture and Technology Choices#
The system uses a single-machine training setup; the entire pipeline runs on a single RTX 4090 (24GB).
Data flow: Raw data (six department CSV files from Chinese-medical-dialogue-data) is read by the cleaning script → the department, title, question, and answer columns are extracted → converted to Alpaca format as instruction (“You are now a {department} doctor. Please give advice based on the patient’s question:”), input (title + question concatenated), and output (answer) → saved as JSONL for training.
Training flow: Unsloth’s FastLanguageModel.from_pretrained loads a 4-bit quantized Qwen2.5-7B-Instruct → get_peft_model injects the LoRA adapter (r=16, lora_alpha=16, target_modules covering the attention and FFN projection layers) → SFTTrainer (from TRL) runs supervised fine-tuning with an Alpaca prompt template (learning_rate=2e-4, batch_size=2, gradient_accumulation=4, max_seq_length=2048, num_train_epochs=1) → the LoRA adapter and tokenizer are saved after training.
Inference/export flow: Load the base model and LoRA adapter → optionally call merge_and_unload to merge into a full model → run inference with Hugging Face generate or Unsloth’s FastLanguageModel.for_inference → optionally export to GGUF (q4_k_m / q8_0) for llama.cpp / Ollama.
Design rationale: Unsloth uses Triton kernel optimizations and memory management to greatly improve training efficiency without changing the model architecture, and it is fully compatible with Hugging Face Transformers, simplifying later deployment and migration. LoRA trains only low-rank incremental matrices, injecting domain knowledge while preserving the base model’s capabilities; adapters can be saved and loaded independently, making it easy to switch between tasks or domains. 4-bit quantization (NF4 + double quantization) reduces memory requirements to about 1/4 of native float16, making a 7B model trainable within 24GB of GPU memory.
Relationship to Manshu LLM: Manshu LLM focuses on implementing each Transformer module from scratch to understand the underlying principles; Manshu Finetune focuses on using existing pretrained models and efficient fine-tuning tools to quickly iterate toward a usable domain model on real data. The two are complementary, covering “deep understanding” and “practical engineering” respectively.
Key Technology Choices#
Unsloth
A Triton-based efficient training framework that achieves 2-5x faster training and 50%-80% memory savings through kernel fusion and memory optimization. Fully compatible with Hugging Face Transformers: no model code changes are needed, only a different loading call. Supports mainstream architectures such as Llama, Mistral, and Qwen.
LoRA (Low-Rank Adaptation)
Injects low-rank incremental matrices (A @ B) alongside the pretrained weights and trains only the two small matrices A and B (a few hundred KB per layer at rank=16). At inference time they can be merged back into the original weights with no extra latency, or kept separate for easy task switching. Compared with full fine-tuning, it trains roughly 1000x fewer parameters, greatly reducing memory and time costs.
Qwen2.5-7B-Instruct
A 7B instruction-tuned model from Alibaba’s Tongyi Qianwen family with strong performance on Chinese tasks. The Instruct version already supports instruction following and multi-turn dialogue, making it a suitable base for medical Q&A. At 4-bit quantization the 7B weights need about 4-5GB; adding LoRA gradients and optimizer states brings total usage to about 15-18GB, which fits on a single 24GB GPU.
4-bit quantization (NF4 + double quantization)
bitsandbytes’ NF4 (Normal Float 4-bit) quantization compresses weights to 4 bits, and double quantization (quantizing the quantization constants) further reduces memory. Quantization only affects weight storage; gradients and optimizer states remain in float16/float32, so the impact on training accuracy is limited.
Alpaca format
The standard format for instruction tuning, with three fields: instruction, input (input/context), and output (expected output). A prompt template concatenates them into a complete training sample, and the model learns to generate the output from instruction + input. The format is simple and general, making it easy to adapt to different data sources.
Functional Modules#
Data Cleaning and Format Conversion#
The raw data comes from Chinese-medical-dialogue-data and consists of CSV files for six departments:
| Department | File | Q&A Pairs |
|---|---|---|
| Internal Medicine | IM | 220,606 |
| Surgery | Surgical | 115,991 |
| Pediatrics | Pediatric | 101,602 |
| Obstetrics & Gynecology | OAGD | 183,751 |
| Andrology | Andriatria | 94,596 |
| Oncology | Oncology | 75,553 |
| Total | — | 792,099 |
Cleaning steps:
- Read each department’s CSV and normalize the encoding to UTF-8
- Filter nulls: drop rows where question or answer is empty
- Text cleaning: strip leading/trailing whitespace and replace malformed characters (e.g., mojibake, unusual Unicode)
- Length filtering: drop samples where question + answer exceeds 1500 characters (to avoid overly long sequences hurting training)
- Deduplication: deduplicate by question, keeping the first entry (the same question may have several similar answers)
Format conversion:
{
"instruction": "You are now an internal medicine doctor. Please give advice based on the patient's question:",
"input": "Can people with high blood pressure take codonopsis? I have high blood pressure, and when my son-in-law visited these past two days he brought me some codonopsis to steep in water. Hello, can people with high blood pressure take codonopsis?",
"output": "Patients with high blood pressure can take codonopsis orally. Codonopsis helps lower blood lipids and blood pressure and can clear waste from the blood, which offers some stabilizing and preventive benefit for patients with coronary heart disease and cardiovascular disease, so taking codonopsis regularly can help keep the risks of high blood pressure, high blood lipids and high blood sugar at bay. In addition, codonopsis replenishes qi and nourishes the blood, has a calming effect on the central nervous system, regulates digestive function, and strengthens the spleen and lungs. Thank you for your question; I hope my explanation helps."
}
After cleaning, the data is split 9:1 into training and validation sets and saved as JSONL.
Model Loading and LoRA Injection#
Load the 4-bit quantized Qwen2.5-7B-Instruct with Unsloth:
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Qwen2.5-7B-Instruct-bnb-4bit",
max_seq_length=2048,
dtype=None, # auto-detect
load_in_4bit=True,
)
Inject the LoRA adapter:
model = FastLanguageModel.get_peft_model(
model,
r=16,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_alpha=16,
lora_dropout=0,
bias="none",
use_gradient_checkpointing="unsloth",
random_state=3407,
)
Configuration notes:
- r=16: LoRA rank, controlling the expressiveness and parameter count of the incremental matrices
- lora_alpha=16: Scaling factor, typically set equal to r
- target_modules: Modules that receive LoRA, covering attention (Q/K/V/O) and FFN (gate/up/down)
- use_gradient_checkpointing=“unsloth”: Unsloth-optimized gradient checkpointing for further memory savings
Training Configuration and Execution#
Supervised fine-tuning with TRL’s SFTTrainer:
from trl import SFTTrainer
from transformers import TrainingArguments
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
dataset_text_field="text",
max_seq_length=2048,
args=TrainingArguments(
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
warmup_steps=100,
num_train_epochs=1,
learning_rate=2e-4,
fp16=not torch.cuda.is_bf16_supported(),
bf16=torch.cuda.is_bf16_supported(),
logging_steps=10,
eval_steps=500,
save_steps=500,
output_dir="outputs",
optim="adamw_8bit",
),
)
trainer.train()
Training hyperparameters:
| Parameter | Value | Notes |
|---|---|---|
| per_device_train_batch_size | 2 | Per-GPU batch size |
| gradient_accumulation_steps | 4 | Gradient accumulation; effective batch size = 8 |
| learning_rate | 2e-4 | Learning rate |
| warmup_steps | 100 | Warmup steps |
| num_train_epochs | 1 | Training epochs (790K samples; one epoch is about 100K steps) |
| max_seq_length | 2048 | Maximum sequence length |
| optim | adamw_8bit | 8-bit AdamW to save optimizer-state memory |
Inference Validation#
After training, load the LoRA adapter for inference:
FastLanguageModel.for_inference(model)
inputs = tokenizer(
[alpaca_prompt.format(
"You are now an internal medicine doctor. Please give advice based on the patient's question:",
"I've been feeling dizzy often lately and my blood pressure is a bit high. What should I pay attention to?",
""
)],
return_tensors="pt"
).to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Evaluation metrics:
- Human evaluation: sample 100 validation examples and compare the professionalism and relevance of answers before and after fine-tuning
- BLEU-4 / ROUGE-L: text similarity to reference answers (for reference only, since medical Q&A has no single correct answer)
Model Export#
Save the LoRA adapter:
model.save_pretrained("lora_model")
tokenizer.save_pretrained("lora_model")
Merge into a full model:
model.save_pretrained_merged("merged_model", tokenizer, save_method="merged_16bit")
Export to GGUF (for llama.cpp / Ollama):
model.save_pretrained_gguf("gguf_model", tokenizer, quantization_method="q4_k_m")
Supported quantization methods: q4_k_m (4-bit, recommended), q8_0 (8-bit, higher precision), f16 (half precision, lossless).
Technical Challenges and Solutions#
GPU Memory Optimization#
Challenge: Full fine-tuning of a 7B model requires 60-80GB of GPU memory, far beyond a single 24GB card; even with LoRA, loading and training with vanilla Transformers still needs about 30GB.
Solution:
- 4-bit quantized loading: bitsandbytes NF4 quantization compresses model weights from float16 (14GB) to 4-bit (about 4GB)
- Unsloth optimizations: Triton kernel fusion reduces intermediate activation memory, and optimized gradient checkpointing further lowers peak memory
- 8-bit AdamW: Optimizer states are stored in 8-bit, saving 75% of memory compared with float32
- Gradient accumulation: Small batches plus gradient accumulation achieve a large effective batch while lowering per-step peak memory
Training ran stably on an RTX 4090 (24GB) with peak memory of about 18GB.
Data Quality and Format#
Challenge: The raw medical Q&A data is noisy (empty answers, duplicate questions, overly long text, encoding issues), which would degrade model quality if used directly.
Solution:
- Multi-stage cleaning: null filtering → encoding repair → length filtering → deduplication
- Per-department statistics: record sample counts before and after cleaning for each department to keep the data distribution reasonable
- Standardized Alpaca format: a unified prompt template ensures the instruction clearly specifies the doctor role and task
About 750K high-quality samples remain after cleaning, covering all six departments.
Training Stability#
Challenge: Large-scale data and long-sequence training are prone to exploding gradients or non-converging loss.
Solution:
- Warmup: The learning rate rises linearly from 0 to 2e-4 over the first 100 steps to avoid early gradient instability
- Gradient clipping: max_grad_norm=1.0 to prevent exploding gradients
- bf16 mixed precision: Use bf16 on supported GPUs for a wider numeric range and better stability
- Periodic evaluation: Evaluate on the validation set every 500 steps to monitor overfitting
Training loss decreased steadily, and validation loss stabilized in the 0.8-1.0 range.
Summary#
As a flagship personal project, Manshu Finetune highlights two things: first, an efficient fine-tuning approach based on Unsloth + LoRA that completes domain fine-tuning of a 7B model on a single 24GB GPU; second, the use of a real medical Q&A dataset (790K pairs) to strengthen medical question answering. The project covers the complete pipeline from data cleaning, format conversion, model loading, LoRA injection, and training to inference and GGUF export, and is reproducible and transferable to other domains.
Future directions: add fine-tuning on multi-turn dialogue data (currently single-turn Q&A); introduce DPO / RLHF to further align with human preferences; extend to other medical data sources (e.g., medical literature, clinical note summaries); deploy as an API service or integrate with a RAG system for knowledge-augmented medical Q&A.