Pular para o conteúdo principal

[Diário de Bordo Maeumjigi Capstone #11] Avaliação do Modelo M6 com Dados Sintéticos v5

· 3 min para ler
Jaybee
Jaybee
AI & Robotics Engineer

Cover Image

April 19 is the day we evaluated the performance of the model (M6) trained on the virtual synthetic dataset (v5) we painstakingly created, and wrote the 7th-week weekly report based on it. Was the data we generated while enduring a $10 bill bomb worth it?

A Real Showdown of Model Performance: M0 vs M5 vs M6​

  • Problem: To persuade the advising professor and team members, we needed a weekly report that included quantitative test results and intuitive comparison examples, rather than simply claiming "it got better."
  • Process:

    "Please rewrite the report. The main point is the comparison between M5 based on v4 and M6 based on v5... Include examples of each dataset." "When describing the test results, include explanations with examples of the differences between the un-tuned base model (M0) and M6."

  • Result: The Artificial Intelligence (AI) analyzed the text results generated by the M0, M5, and M6 models and extracted a comparison file (m0_m6_compare.json). The results showed that the base Llama 3.1 model (M0) was somewhat awkward in Korean counseling, and while M5 (trained on refined data v4) improved, it lacked depth. In contrast, the M6 model trained on high-quality synthetic data (v5) perfectly commanded the warm and professional tone typical of a psychological counselor, deeply empathizing with the user's emotions.

The 7th-Week Weekly Report Written by AI (DOCX)​

  • Process:

    "Please write the 7th-week report focusing on the results and significance in two versions (a concise version under 5 pages, and a detailed version). Then generate docx files for each based on the two md files."

  • Result: With a single instruction, two versions (7th-week_report_v1_concise.md, 7th-week_report_v2_detailed.md) were completed in a flash, and even neatly converted into Word files.

Introducing RAG Architecture: FAISS Database (DB) Construction Evaluation​

  • Process:

    "I performed FAISS_DB construction and wrote FAISS_DB_construction_report.docx. Please evaluate our plan and the results of the report."

  • Result: In addition to fine-tuning, which tweaks the model's weights, we introduced a FAISS vector database for RAG (Retrieval-Augmented Generation) to reduce hallucinations by referencing external knowledge. I asked the AI to evaluate the structure I built, and received positive feedback (FAISS_DB_evaluation.md).

Terminal Screenshot

Today was a day when all the hard work paid off. The thrill of confirming that the trained AI model actually became noticeably smarter is incredible! With model performance proven and the FAISS DB successfully integrated, the Maeumjigi chatbot is one step closer to completion.

Summary & Key Takeaways​

  • Comparative Evaluation of Synthetic Fine-Tuning: Conducted qualitative and quantitative benchmarking across un-tuned base model M0, filtered-data model M5, and synthetic-dataset model M6.
  • Superior Empathetic Dialogue: Model M6, trained on synthetic dataset v5, demonstrated professional counseling tone and nuanced emotional resonance without generic platitudes.
  • Automated Reporting Pipeline: Prompted AI to generate comprehensive and concise 7th-week academic reports (.docx) with empirical comparison tables (m0_m6_compare.json).
  • RAG Architecture with FAISS Vector DB: Integrated an external FAISS vector retrieval database alongside fine-tuned weights to ground responses in verified clinical knowledge and curb hallucination.
PUBLICIDADE