[Maeumjigi 畢業設計開發日誌 #4] 全面數據預處理: Llama 3.1 對話模板與自主 JSONL 數據流水線
· 閱讀時間約 4 分鐘
April 1, 2026, was the day our engineering team launched the automated preprocessing pipeline for the massive counseling corpora that would form the foundation of Maeumjigi. Training a high-performance open-source Large Language Model (LLM) like Meta Llama 3.1 requires rigorous data hygiene: converting unstructured multi-party dialogues into clean, token-bounded, and special-token-aligned instruction datasets.