Aller au contenu principal

[Journal de bord Maeumjigi Capstone #5] Nettoyage de 100 000 données émotionnelles et contraintes de coûts d'API LLM

· 3 minutes de lecture
Jaybee
Jaybee
AI & Robotics Engineer

Cover Image

This is the record of April 3, the day we began in earnest the task of refining the training conversation data to feed into the deep learning model (Llama 3.1). It was also the day we ran a massive pipeline that makes the Artificial Intelligence (AI) read the text and judge the emotion on its own, going beyond simple data merging.

Reclassifying Data Emotions (v3 -> v4)​

  • Problem: The emotion labeling of the existing dataset (Emotion_train_dataset_v3) did not perfectly align with our specific psychological counseling goals. Out of a total of 1 million data points, we needed to filter only the meaningful ones and accurately map their emotions.
  • Process:

    "The scope is the entire 1 million... Because Claude, as a psychological counseling expert, will read the conversation and directly map the emotion. We are filtering only what falls under our emotion classification and excluding the rest."

  • Result: We established v4_plan.md and started the work. Initially targeting 100,000 lines of data, we wrote and ran a script that integrates the Claude API to grasp the conversational context and reclassify emotions.

The Horror of Large Language Model (LLM) API Billing​

  • Problem: As we threw massive amounts of text data at an LLM (Claude) for evaluation, the API usage (token count) increased exponentially.
  • Process:

    "I'm already using Claude with a Pro subscription, do I need a separate API?" "Is there a way to use Claude Code instead of the Anthropic API? The billing is burdensome." "The actual cost came out to $10.27."

  • Result: We painfully realized that, completely separate from the Claude Pro subscription ($20/month), the Anthropic API called by the Python script for automated batch processing is billed on a pay-as-you-go basis depending on usage. Just processing 100,000 test lines instantly cost us about $10.27. Realizing that processing all 1 million records would cost a fortune, it was a day we learned firsthand the importance of model optimization and data sampling.

Terminal Screenshot

Today was the day we hit the realistic wall of 'computing costs' inevitably encountered during deep learning data preprocessing. Although we successfully obtained high-quality, refined data (v4_report.md), it was a valuable experience that left us deeply contemplating how to balance cost and efficiency as we proceed with fine-tuning!

Summary & Key Takeaways​

  • Context-Aware Emotion Reclassification: Transitioned dataset v3 to v4 by utilizing LLMs to read conversational context and map multi-turn dialogues into target counseling emotion tags.
  • API Pay-As-You-Go Economics: Discovered firsthand that scripted automated batch ingestion incurs pay-per-token API charges independent of personal Pro subscriptions.
  • Empirical Cost Realities: Sustained a $10.27 expenditure for processing the first 100,000 records, exposing the fiscal impossibility of processing the full 1-million sample corpus naively.
  • Pivoting to Stratified Sampling: Promptly re-evaluated pipeline architecture to balance training efficiency with budget, prioritizing high-leverage samples and prompt compression.
PUBLICITÉ