Skip to main content

[Maeumjigi Capstone Dev Log #11] Evaluating M6 Model Trained on Synthetic Data (v5)

· 3 min read
Jihwan Park (jaybee)
Jihwan Park (jaybee)
AI & Robotics Engineer

Cover Image

April 19 is the day we evaluated the performance of the model (M6) trained on the virtual synthetic dataset (v5) we painstakingly created, and wrote the 7th-week weekly report based on it. Was the data we generated while enduring a $10 bill bomb worth it?

A Real Showdown of Model Performance: M0 vs M5 vs M6​

  • Problem: To persuade the advising professor and team members, we needed a weekly report that included quantitative test results and intuitive comparison examples, rather than simply claiming "it got better."
  • Process:

    "Please rewrite the report. The main point is the comparison between M5 based on v4 and M6 based on v5... Include examples of each dataset." "When describing the test results, include explanations with examples of the differences between the un-tuned base model (M0) and M6."

  • Result: The Artificial Intelligence (AI) analyzed the text results generated by the M0, M5, and M6 models and extracted a comparison file (m0_m6_compare.json). The results showed that the base Llama 3.1 model (M0) was somewhat awkward in Korean counseling, and while M5 (trained on refined data v4) improved, it lacked depth. In contrast, the M6 model trained on high-quality synthetic data (v5) perfectly commanded the warm and professional tone typical of a psychological counselor, deeply empathizing with the user's emotions.

The 7th-Week Weekly Report Written by AI (DOCX)​

  • Process:

    "Please write the 7th-week report focusing on the results and significance in two versions (a concise version under 5 pages, and a detailed version). Then generate docx files for each based on the two md files."

  • Result: With a single instruction, two versions (7th-week_report_v1_concise.md, 7th-week_report_v2_detailed.md) were completed in a flash, and even neatly converted into Word files.

Introducing RAG Architecture: FAISS Database (DB) Construction Evaluation​

  • Process:

    "I performed FAISS_DB construction and wrote FAISS_DB_construction_report.docx. Please evaluate our plan and the results of the report."

  • Result: In addition to fine-tuning, which tweaks the model's weights, we introduced a FAISS vector database for RAG (Retrieval-Augmented Generation) to reduce hallucinations by referencing external knowledge. I asked the AI to evaluate the structure I built, and received positive feedback (FAISS_DB_evaluation.md).

Terminal Screenshot

Today was a day when all the hard work paid off. The thrill of confirming that the trained AI model actually became noticeably smarter is incredible! With model performance proven and the FAISS DB successfully integrated, the Maeumjigi chatbot is one step closer to completion.