45 vulnerabilities (15 moderate, 15 high, 15 critical)
To address all issues (including breaking changes), run:
  npm audit fix --force

Seeing words like Critical, Remote Code Execution (RCE), and Prototype Pollution triggers an immediate instinct: type npm audit fix --force to make the scary red warnings go away.

However, in the world of Static Site Generators (SSG) like Docusaurus, Next.js, or Hugo deployed to GitHub Pages or AWS S3, doing so is often an engineering mistake that breaks production builds without improving your actual security posture.

In this article, we dissect the static site threat model, explain why bundler vulnerabilities differ fundamentally from server vulnerabilities, and share a production-tested 4-Layer Security Hardening Blueprint to protect what actually matters: your secrets, client privacy, and repository integrity.

1. Executive Summary: The Core Engineering Trade-off​

1. Build-Time Tools != Production Runtime: Packages flagged in npm audit (such as tinypool, serialize-javascript, postcss-selector-parser) are node worker and bundler dependencies executed locally on your machine during npm run build.
2. Zero Live Server on CDN: Once compiled, GitHub Pages serves static, immutable HTML, CSS, and client-side JavaScript. There is no running Node.js daemon, no backend port listening, and no database connection in production.
3. The Danger of --force: Running npm audit fix --force blindly downgrades core frameworks (e.g., rolling Docusaurus back to older releases), breaking React 19 compatibility, MDX v3 pipelines, and future flag architectures.
4. Where Real Static Risks Lie: The genuine security risks of static blogs are credential/API key leakage in public Git history, PII exposure in datasets, and client-side cross-origin leakage.

2. Anatomy of the 45 Vulnerabilities: Build vs Runtime​

Let's examine the actual CVEs reported in our audit log:

A. Tinypool Prototype Pollution (GHSA-5gmw-xhrv-c9v3 - Critical)​

• Vulnerability: If untrusted user input is passed into the worker initialization options of tinypool, an attacker can pollute Object.prototype.
• Reality in SSG: tinypool is used internally by @docusaurus/faster to parallelize bundling tasks across CPU cores during build time. The input parameters come solely from your own docusaurus.config.ts. No external visitor on your website can send HTTP payloads to your local CPU worker pool.

B. Serialize-JavaScript RCE (GHSA-5c6j-r48x-rmvq - High)​

• Vulnerability: Remote Code Execution via crafted RegExp flags or serialized dates.
• Reality in SSG: Webpack uses this to serialize internal compilation chunks into static .js bundles. Once written to the build/ directory, JavaScript files are passive assets served over HTTPS. No server-side deserialization takes place.

C. PostCSS Selector Parser CPU Exhaustion (GHSA-rj75-hqrm-r3gf - Moderate)​

• Vulnerability: Quadratic complexity in flat selector parsing allows Denial of Service (DoS) via maliciously crafted CSS strings.
• Reality in SSG: The CSS files being parsed are your own theme stylesheets (custom.css). Unless an attacker has write access to your Git repository, your build server will never parse untrusted external CSS.

Running npm audit fix --force in this situation attempts to downgrade @docusaurus/core from 3.10.2 to 3.7.0. This breaks Docusaurus v4 future flags and React 19 rendering, introducing breaking bugs into your codebase to patch an exploit that cannot physically occur in your architecture.

3. What Actually Matters: The 4-Layer Static Security Blueprint​

If npm bundler vulnerabilities are low risk, where should static site engineers invest their security budget? Here is the multi-layered defense pipeline implemented in this blog.

Layer 1: Automated Pre-Commit Secret Scanning​

The single greatest catastrophic risk for public technical blogs is accidentally committing an OpenAI key (sk-...), a GitHub Personal Access Token (ghp_...), or an AWS access credential (AKIA...).

We created an automated audit script (tools/audit_security.py) that utilizes Git's object database to scan every tracked file against high-entropy regex patterns:

PATTERNS = [
    (r'(?i)(api[_-]?key|secret|token|password|auth|bearer)\s*[:=]\s*["\']([^"\']{8,})["\']', "Potential Key Assignment"),
    (r'ghp_[a-zA-Z0-9]{36}', "GitHub Personal Access Token"),
    (r'sk-[a-zA-Z0-9]{32,}', "OpenAI API Key"),
    (r'AIzaSy[a-zA-Z0-9_-]{33}', "Google API Key"),
    (r'AKIA[0-9A-Z]{16}', "AWS Access Key"),
    (r'-----BEGIN\s+(RSA\s+)?PRIVATE\s+KEY-----', "Private Key Certificate"),
]

Running this audit script before every release ensures zero hardcoded tokens ever reach the remote repository.

Layer 2: Personally Identifiable Information (PII) Scrubbing​

When publishing engineering articles on AI datasets (such as our 23-part Maeumjigi clinical counseling series or 5-part ANPR Vision AI series):

• Synthetic Data First: All counseling records were synthetically generated via LLM pipelines (synth_gen) rather than using real clinical transcripts, guaranteeing zero medical privacy breach.
• Regex PII Audit: Our scanner automatically validates markdown text against Korean national identity numbers (\d{6}-[1-4]\d{6}) and phone numbers (010-\d{4}-\d{4}).

Layer 3: Hardened .gitignore Configuration​

A standard .gitignore often ignores .env.local while leaving .env exposed. We hardened .gitignore to comprehensively block sensitive files, private certificates, and raw analytics dumps:

# Environment & Secret Protection
.env
.env.*
*.pem
*.key
*.pfx
*.cert
*credential*
*.secret

# Analytics & CSV dumps
*.csv

# Archived template files
_old/

Layer 4: Client-Side Privacy & Iframe Sandboxing​

Even without a backend server, static sites must respect user privacy and prevent data leakage:

1. Strict Referrer Policy: In docusaurus.config.ts, we inject a strict referrer header into <head>:

   themeConfig: {
     metadata: [
       { name: 'referrer', content: 'strict-origin-when-cross-origin' },
     ],
   }
   

   This ensures that when a reader clicks an external documentation link, your blog's specific full URL parameters are truncated, preventing path leakage.

2. Giscus Discussion Sandboxing: We host comments via GitHub Discussions. In BlogPostItem/index.tsx, Giscus is configured with:

   • emitMetadata="0": Prevents post-message iframe broadcasts from exposing discussion details to third-party browser scripts.
   • loading="lazy": Prevents iframe resource execution until the reader scrolls to the bottom of the article.

4. Key Takeaways for Developers​

• Do Not Panic at npm audit: Always distinguish between Build-Time Bundler Dependencies and Production Runtime Dependencies.
• Beware of --force: Automated downgrades often introduce real, breaking application bugs while solving theoretical vulnerabilities that cannot be triggered on a static CDN.
• Focus on the Real Attack Surface: Automate secret detection, scrub sensitive data from datasets, enforce strict client-side referrer policies, and protect your Git history.

📌 Key Takeaways (TL;DR)

• Problem: Single-frame OCR inevitably suffers from transient misreadings (e.g., 64아1511, 24가1511) due to vehicle yaw, glare, and vibration.
• Solution: Implemented Korean plate regex validation (^\d{2,3}[가-힣]\d{4}$) coupled with a temporal sliding window majority voting engine across a 7-frame buffer (plate_confirm_threshold=3).
• Result: Advanced through 11 system iterations (v01 to v11), transitioning from initial gibberish (ACCIIXOY-1Z01...) on June 5 to deterministic confirmation of 64가1511 on June 7 with 99.4% reliability.

1. Korean ANPR Syntax & EasyOCR Tuning​

Korean vehicle registration plates follow strict regulatory syntax:

Syntax: [2 or 3 Digits] + [1 Hangul Syllable] + [4 Digits]
Example: 64가1511, 123가4567

import easyocr
import re
from collections import deque, Counter

reader = easyocr.Reader(['ko', 'en'], gpu=True)

def parse_korean_plate(plate_img):
    results = reader.readtext(plate_img, detail=0)
    raw_text = "".join(results).replace(" ", "")
    
    # Korean License Plate Regex Match
    pattern = r'(\d{2,3}[가-힣]\d{4})'
    match = re.search(pattern, raw_text)
    return match.group(1) if match else None

2. Temporal Majority Voting Engine​

To overcome transient single-frame OCR flicker, we maintain a sliding history queue:

plate_history = deque(maxlen=CONFIG["plate_history_size"])  # 7
confirmed_plate = None

def update_plate_voting(detected_text):
    global confirmed_plate
    if detected_text:
        plate_history.append(detected_text)
        
    if len(plate_history) >= CONFIG["plate_confirm_threshold"]:
        counts = Counter(plate_history)
        most_common_plate, count = counts.most_common(1)[0]
        
        # Confirm plate if 3 or more votes agree
        if count >= CONFIG["plate_confirm_threshold"]:
            confirmed_plate = most_common_plate
            
    return confirmed_plate

3. The 11-Iteration Evolution: June 5 to June 7​

Our log.txt preserves the raw chronological milestones:

4. Conclusion & Future Horizons​

Across this 5-part series, we designed an end-to-end computer vision system integrating dynamic lane tracking, polygon IoU intrusion detection, hierarchical object localization, CSRT tracking, EDSR_x4 super-resolution, and temporal voting OCR. This pipeline provides a proven template for production-grade intelligent transportation systems.

📌 Key Takeaways (TL;DR)

• Problem: Cropped license plates from dashcam footage often measure under 40x20 pixels with heavy H.264 macroblocking; standard bicubic interpolation fails to resolve character contours.
• Solution: Integrated EDSR_x4 deep residual networks (EDSR_x4.pb) that omit batch normalization to preserve dynamic range, combined with temporal multi-frame image stacking across 3 to 5 consecutive frames.
• Result: Quadrupled plate pixel density and sharpened stroke edges, lifting downstream EasyOCR accuracy on Korean characters from 32% to over 91%.

1. Resolution Limitations & The Need for Deep Super-Resolution​

Even with precise license plate localization, optical limitations thwart standard OCR:

1. Failure of Bicubic Scaling: Interpolation algorithms blur edges further without hallucinating high-frequency contours.
2. H.264 Compression & Sensor Noise: Dashcam compression creates severe pixelation that distorts alphanumeric strokes.

We integrated EDSR_x4 via the OpenCV DNN Super-Resolution module to reconstruct high-frequency details:

import cv2
from cv2 import dnn_superres

def init_super_resolution_engine(model_path="EDSR_x4.pb"):
    sr = dnn_superres.DnnSuperResImpl_create()
    sr.readModel(model_path)
    sr.setModel("edsr", 4)
    sr.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
    sr.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)
    return sr

2. Multi-Frame Image Stacking for Noise Reduction​

I_stacked(x, y) = (1 / N) * Σ Warp(I_i(x, y))

This temporal stacking cancels random sensor noise while boosting text-background contrast.

3. Looking Forward​

In Post #5, we conclude the series with Korean ANPR OCR tuning, regex syntactic validation, and the temporal majority voting engine, recounting our engineering evolution from initial gibberish to finalized recognition (64가1511).

📌 Key Takeaways (TL;DR)

• Problem: Searching for tiny plates (30x15 pixels) across the raw 640x480 frame generates massive false positives from roadside signs and drops inference speed below 10 FPS.
• Solution: Designed a two-stage hierarchical architecture: broad vehicle detection (YOLOv3-tiny) crops the vehicle ROI, within which a fine-tuned YOLOv5 (lp_det.pt, conf=0.6) locates the license plate. Integrated OpenCV CSRT correlation filter tracking with a 15-frame redetection interval.
• Result: Eliminated 98% of background false positives and slashed GPU overhead by 80%, sustaining real-time 50+ FPS continuous tracking.

1. The Pitfalls of Global Detection & The Hierarchical Solution​

Running plate detectors across the full 640x480 image causes two critical flaws:

1. Background Clutter: Street advertisements, road billboards, and store signs trigger high false-alarm rates.
2. Resolution Downsampling Loss: Resizing 640x480 to typical model inputs (e.g., 640x640) squashes tiny 30-pixel plates into sub-feature extinction.

Our two-stage hierarchical pipeline addresses this by constraining the search space:

vehicle_boxes = detect_vehicles_yolov3(frame, net=vehicle_net, conf_thresh=0.5)

for box in vehicle_boxes:
    x1, y1, x2, y2 = box
    vehicle_roi = frame[y1:y2, x1:x2]
    plate_boxes = plate_model(vehicle_roi)
    for pbox in plate_boxes:
        px1, py1, px2, py2 = pbox
        global_plate_box = [x1 + px1, y1 + py1, x1 + px2, y1 + py2]

2. Real-Time Latency Mitigation: CSRT Tracker Integration​

if frame_count % CONFIG["detection_interval"] == 0:
    boxes = detect_vehicles_yolov3(frame)
    tracker = cv2.TrackerCSRT_create()
    tracker.init(frame, tuple(boxes[0]))
else:
    success, tracked_box = tracker.update(frame)
    if not success:
        tracker = None

3. Looking Forward​

In Post #4, we explore deep EDSR_x4 super-resolution to reconstruct crisp alphanumeric strokes from blurry low-resolution plate crops, combined with multi-frame image stacking.

📌 Key Takeaways (TL;DR)

• Problem: Dashcam footage contains heavy pavement specular reflection, tar cracks, and road shadows that induce false lane lines; simple coordinate checking cannot reliably catch the exact moment of lane encroachment.
• Solution: Implemented vanishing-point trapezoidal ROI masking, slope thresholding (|slope| > 0.5), and a 7-frame temporal moving average queue. Formulated intrusion as geometric IoU between vehicle tire footprints and lane boundaries.
• Result: Filtered out >95% of road texture noise while capturing vehicle lane crossing events with sub-frame accuracy.

1. Dynamic Lane Detection: Canny & Probabilistic Hough Transform​

def detect_lanes(frame):
    height, width = frame.shape[:2]
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    blur = cv2.GaussianBlur(gray, (5, 5), 0)
    edges = cv2.Canny(blur, 50, 150)
    
    mask = np.zeros_like(edges)
    roi_polygon = np.array([[
        (int(width * 0.1), height),
        (int(width * 0.45), int(height * 0.6)),
        (int(width * 0.55), int(height * 0.6)),
        (int(width * 0.9), height)
    ]], np.int32)
    cv2.fillPoly(mask, roi_polygon, 255)
    masked_edges = cv2.bitwise_and(edges, mask)
    
    lines = cv2.HoughLinesP(
        masked_edges,
        rho=1,
        theta=np.pi / 180,
        threshold=20,
        minLineLength=20,
        maxLineGap=10
    )
    return lines

2. Slope Filtering & Temporal Smoothing​

Raw line segments are filtered using slope criteria:

Slope m = (y2 - y1) / (x2 - x1)

• Left Lane: m < -0.5
• Right Lane: m > 0.5
• Horizontal Noise: |m| <= 0.5 (crosswalks and bumps are rejected)

A deque(maxlen=7) temporal moving average smooths lane equations across consecutive frames.

3. Geometric Polygon IoU Lane Intrusion Criterion​

Footprint = [x_min, y_min + 0.7*h, x_max, y_max]
IoU = Area(Footprint ∩ Lane_Polygon) / Area(Footprint)

def check_lane_intrusion(vehicle_box, lane_polygon, threshold=0.05):
    vx1, vy1, vx2, vy2 = vehicle_box
    vh = vy2 - vy1
    footprint = Polygon([(vx1, vy1 + 0.7*vh), (vx2, vy1 + 0.7*vh), (vx2, vy2), (vx1, vy2)])
    
    intersection = footprint.intersection(lane_polygon).area
    iou = intersection / footprint.area
    return iou > threshold, iou

When IoU > 0.05, the system flags a violation event and triggers YOLOv5 license plate localization.

4. Looking Forward​

In Post #3, we explore hierarchical object detection with fine-tuned YOLOv5 (lp_det.pt) and real-time CSRT tracking integration to slash GPU inference overhead by 80%.

📌 Key Takeaways (TL;DR)

• Problem: Low-resolution dashcam footage suffers from severe vehicle vibration, nighttime backlighting, and heavy motion blur, making direct license plate recognition and lane violation tracking extremely error-prone.
• Solution: Designed a unified 6-stage vision architecture: Canny/Hough lane extraction -> YOLOv3 vehicle detection -> polygon IoU lane violation calculation -> YOLOv5 plate localization -> EDSR_x4 super-resolution -> EasyOCR temporal voting.
• Result: Constructed an end-to-end Python pipeline achieving robust vehicle tracking and lane boundary monitoring even under degraded dashcam conditions.

1. Project Background: Why Dashcam Vision is Hard​

In intelligent transportation systems (ITS) and autonomous traffic enforcement, real-time lane intrusion detection and automatic number plate recognition (ANPR) are foundational capabilities. However, real-world dashcam footage presents severe optical challenges:

1. Extreme Low Resolution & Blurred Textures: The license plate of a leading or overtaking vehicle often spans merely 30x15 to 50x25 pixels across a 640x480 frame, causing alphanumeric strokes to merge into indistinct blobs.
2. Vehicle Vibration & Rolling Shutter Distortions: Road bumps, sudden decelerations, and perspective shifts distort straight lane boundaries.
3. Real-Time Latency Bottlenecks: Executing heavy full-frame vision transformers per frame results in fatal FPS drops, disabling real-time tracking.

2. 6-Stage Hierarchical Pipeline Architecture​

To achieve both high precision and high throughput, we modularized the system into a 6-stage decoupled pipeline:

flowchart TD
    Raw["Raw Dashcam Frame 640x480"] --> L1["Stage 1: Canny + Hough Dynamic Lane Extraction"]
    Raw --> L2["Stage 2: YOLOv3-tiny Vehicle Detection"]
    L1 & L2 --> L3["Stage 3: Vehicle Footprint & Lane Polygon IoU Intrusion Metric"]
    L3 -- If Intrusion Triggered --> L4["Stage 4: Localized YOLOv5 Plate Detection in Vehicle ROI"]
    L4 --> L5["Stage 5: EDSR_x4 Deep 4x Super-Resolution"]
    L5 --> L6["Stage 6: EasyOCR + Regex + Temporal Majority Voting"]
    L6 --> Result["Confirmed Violation Plate: 64가1511"]

Decoupled Processing Responsibilities​

1. Lane Extraction: Trapezoidal dynamic ROI masking, Canny edge detection, and slope filtering isolate driving corridor bounds.
2. Vehicle Detection: Lightweight YOLOv3-tiny outputs candidate vehicle bounding boxes in milliseconds.
3. Intrusion Criterion: Computes geometric Intersection-over-Union (IoU > 0.05) between vehicle tire footprint and the lane boundary polygon.
4. Plate Localization: YOLOv5 (lp_det.pt) evaluates only the cropped vehicle ROI, saving immense compute cycles.
5. Super-Resolution: The cropped plate patch is passed through a deep EDSR_x4 neural network, reconstructing high-frequency character contours.
6. OCR & Majority Voting: EasyOCR extracts alphanumeric tokens, which undergo Korean syntax regex validation and temporal multi-frame voting to confirm the plate number.

3. Core System Parameters (CONFIG)​

CONFIG = {
    "output_width": 640,
    "output_height": 480,
    "conf_threshold_vehicle": 0.5,
    "nms_threshold_vehicle": 0.4,
    "min_vehicle_w": 70,
    "min_vehicle_h": 70,
    "lane_iou_threshold": 0.05,
    "detection_interval": 15,
    "min_plate_area_for_ocr": 1200,
    "plate_history_size": 7,
    "plate_confirm_threshold": 3,
    "sr_model": "EDSR_x4.pb",
    "yolo_plate_model": "lp_det.pt"
}

4. Looking Forward​

In Post #2, we dive deep into Canny edge filtering, probabilistic Hough transform mathematics (HoughLinesP), slope-based outlier rejection, and the geometric polygon IoU intrusion criterion.

As we entered June, the Maeumjigi team completed the major developments of the 1st semester Capstone project and focused entirely on building safety nets and preparing for the final presentation.

The Victory of the 2:1 Ratio and the Safety Wrapper​

"What was our chosen model?" "We decided on 2:1. Now design a method to use a safety_wrapper as a safety lock for our Main Large Language Model (LLM) (2:1)."

As a result of the 4th fine-tuning, we selected the model mixed with a 2:1 ratio of 'emotional' to 'daily' chat as our final LLM. However, given the nature of a suicide prevention chatbot, we could not allow even a single hallucination or fatal wrong answer.

To prevent this, we introduced a Safety Wrapper. It verifies the input and output of the LLM, and if a high-risk situation (Active Suicidality) is detected, the architecture instantly switches to a pre-defined crisis response protocol.

Cloud Server (EC2) Budget and Final Presentation​

"We need an AWS EC2 quote. What's the budget for 5 months of g5.2xlarge?" "Draft the 1st_semester_final_presentation.pptx based on our progress!"

We calculated the Amazon EC2 GPU server budget to run the verified chatbot for 5 months during the summer vacation.

And on June 18th, we gathered all the achievements of our backend, frontend, and Artificial Intelligence (AI) to complete the 1st Semester Final Presentation (PPTX)!

The long and arduous 1st semester of the Maeumjigi Capstone development has successfully concluded. We will deploy the system to the server over the summer and return with a more advanced service in the 2nd semester!

In the final week of May (27th-30th), we dedicated our efforts to wrapping up the results of the highly anticipated 4th Fine-Tuning (fine_tuning_4) and solidifying the system's framework.

Final Checkpoint 4400 and Graph Analysis​

"Qwen35-9B_mixed_7_3/checkpoints/checkpoint-4400/adapter_config.json"

The 4th fine-tuning model, trained with a mix of daily casual chat and emotional counseling data, finally finished training and yielded the optimal weight: checkpoint-4400.

In previous fine-tunings, there was an overfitting issue where the model would respond in an overly serious counseling mode even to casual chat. As a result of experimenting with the data mix ratio, the model has finally been reborn as a 'perceptive' chatbot that perfectly distinguishes between and appropriately responds to daily conversation and serious crisis counseling!

We visualized all evaluation metrics with matplotlib graphs to prove the performance differences at a glance.

Chatbot System Architecture Design and Final Report​

"chatbot_architecture.html generation complete" "Please write the results of fine_tuning_4 as a report (.docx)."

To serve the verified model as a real app, we designed the entire system architecture and Knowledge Graph integration plan under the name chatbot_architecture.html.

And on May 30th, with the Artificial Intelligence (AI)'s help, we automatically generated the fine_tuning_4_report.docx, successfully concluding Phase 1 of our fine-tuning project. The report bluntly contains the actual response text comparisons between the original base model and our fine-tuned model, proving that all our hard work was not in vain!

With this, the breathless May dev logs of the Maeumjigi project are all wrapped up. What new challenges await in June?

On May 21st and 22nd, the Maeumjigi Capstone team embarked on massive data generation and preparation for the 4th Fine-Tuning (fine_tuning_4).

The Tug-of-War Between Emotional and Daily Conversations​

While testing the 3rd fine-tuning model, I discovered a critical issue.

"When fine-tuning with the haiku_pred dataset, the responses seem too heavily skewed towards emotional counseling. How should we fine-tune to maintain the original model's daily conversation abilities?"

Our goal is a 'suicide prevention counselor conveying warm empathy', but the model was showing signs of overfitting, giving overly serious and emotional responses even to mundane questions like "How's the weather?" or "What should I eat?".

To solve this, we planned an experiment to train the model by mixing emotional data and daily chat data at appropriate ratios.

Generating 10,000 v6 Emotion Dataset Entries via Application Programming Interface (API)​

"Unify to 512 tokens / Average 500 tokens per entry / 20 Personas / File: v6.jsonl"

To create the best dataset, I used the Anthropic API (Haiku) Batch mode to generate a staggering 10,000 entries of emotional conversation data (Emotion_train_dataset_v6.jsonl). Though there was a frustrating moment when the API batch ran in the background without showing progress, I filled in the missing entries with an additional script (fill_v6.py) to complete the 10,000 count.

Finding the Golden Ratio (fine_tuning_4)​

"Emotion_train_dataset_v6.jsonl is complete. Now make a 7:3 ratio dataset, a 2:1 ratio, and a 5:5 ratio dataset." "This 4th fine-tuning will be based on the training dataset mix ratio (mixed_2_1, mixed_5_5, mixed_7_3) instead of alpha values."

The 4th fine-tuning has now become an arena to verify which data ratio perfectly harmonizes the Artificial Intelligence (AI)'s 'daily casualness' with 'professional counseling'. We generated three versions of the dataset and started injecting them into the Qwen3.5-9B model!

📌 Key Takeaways (TL;DR)

• Problem: Severe computational constraints on microcontrollers (Arduino Uno ATmega328P with only 2KB SRAM), non-linear wheel slip, noisy ultrasonic sensing, and the classical Reality Gap between Webots simulation physics and physical hardware.
• Solution: Designed a distributed I/O architecture (HC-06 Bluetooth at 115,200 baud) where a host PC runs OpenCV vision processing and Q-Table inference while the Arduino handles low-level PWM motor actuation. Overhead camera lens distortion correction and ArUco marker homography (vision.py) deliver sub-centimeter real-time (x, y, θ) tracking.
• Result: Zero-shot transfer of the simulation-trained 648-state Q-Table achieved an 86.7% target reaching success rate (26/30 trials) in a physical 1.2m × 1.2m arena, successfully closing the reality gap without real-world fine-tuning.

1. The Challenge of Sim-to-Real: Navigating the Reality Gap​

In virtual Webots simulations, surface friction coefficients remain idealized, wheel motors rotate with deterministic precision, and simulated distance sensors yield noiseless measurements. Real hardware, however, introduces unpredictable physical perturbations:

1. Severe Memory & Compute Bottlenecks: The Arduino Uno (ATmega328P) operates with only 32KB flash memory and 2KB SRAM. Running deep neural networks or heavy matrix operations directly on-chip is physically impossible.
2. Sensor Noise & Blind Angles: Inexpensive ultrasonic rangefinders suffer from specular reflection at shallow incidence angles, reporting 0 or spiking to maximum range when pinging smooth acrylic boundaries.
3. Wheel Slip & Odometry Drift: Minor variations in wheel friction and motor gearboxes cause differential-drive robots to veer off-course even when commanded to drive straight, making dead-reckoning odometry completely unreliable.

2. Distributed Control Architecture: Host PC (Brain) ↔ Arduino (Muscle)​

To overcome microcontroller limitations, we decoupled high-level reasoning from low-level actuation: "PC as the brain, Arduino as the muscle."

flowchart TD
    subgraph CeilingCam["Ceiling Vision System (Overhead Camera)"]
        Cam["FHD Webcam (30 FPS)"]
        Aruco["ArUco Marker Detection (DICT_4X4_50)<br>Perspective Homography (homography.npy)"]
        Cam --> Aruco
    end

    subgraph PC["Host PC (Python Engine)"]
        Pose["Robot Pose Estimation (x, y, θ)"]
        Disc["648-bin Discretizer"]
        QL["Trained Q-Table (ε=0)"]
        BT_Host["HC-06 Bluetooth SPP (115200 baud)"]
        
        Aruco --> Pose --> Disc --> QL --> BT_Host
    end

    subgraph Robot["Physical Mobile Robot (Real Hardware)"]
        BT_Slave["HC-06 Bluetooth Receiver"]
        MCU["Arduino Uno (Firmware)"]
        Motors["L298N Dual H-Bridge + Geared DC Motors"]
        
        BT_Host <-->|"Action Byte [0..3]<br>Ack/Status"| BT_Slave --> MCU --> Motors
    end

• Host PC Pipeline: An overhead webcam captures the complete arena top-down, tracking an ArUco fiducial marker mounted on top of the robot at 30 FPS. The robot's planar pose (x, y, θ) is calculated and mapped into the discrete 648-state space to retrieve the optimal greedy action from the Q-Table.
• HC-06 Serial Communication: To minimize Round-Trip Time (RTT) over the Bluetooth Serial Port Profile (SPP), the transmission baud rate was upgraded from the default 9,600 to 115,200 bps, sustaining a low-latency 30~50Hz control loop.
• Arduino Firmware (arduino_firmware.ino): Upon receiving single-byte action commands (0: Forward, 1: Backward, 2: Turn Left, 3: Turn Right), the microcontroller modulates PWM pins on the L298N motor driver for precise tick-based open-loop pulses.

3. Overhead Camera Homography & ArUco Vision Tracking​

To measure robot positioning and obstacle proximity with millimeter accuracy across the real arena, we built an OpenCV-based top-down computer vision pipeline.

📐 Perspective Rectification & Homography (calib_auto.py)​

1. Checkerboard Distortion Calibration: To eliminate barrel distortion from wide-angle webcam lenses, intrinsic camera matrix (K) and distortion coefficients (D) were pre-computed using a standard 9×6 checkerboard.
2. Perspective Homography: To project the angled camera viewpoint into a canonical 1.2m × 1.2m square bird's-eye view, a 3×3 perspective homography matrix (homography.npy) was established from the 4 marked arena corner coordinates.

💻 ArUco Marker Pose Tracking Implementation (vision.py)​

import cv2
import numpy as np

# Load ArUco dictionary and detection parameters
aruco_dict = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_50)
parameters = cv2.aruco.DetectorParameters()
detector = cv2.aruco.ArucoDetector(aruco_dict, parameters)

def get_robot_pose(frame, homography_matrix):
    """Calculates planar coordinates (x, y) and heading angle (theta) from rectified image."""
    warped = cv2.warpPerspective(frame, homography_matrix, (600, 600))
    corners, ids, _ = detector.detectMarkers(warped)
    
    if ids is not None and len(ids) > 0:
        c = corners[0][0]
        center_x = float(np.mean(c[:, 0]))
        center_y = float(np.mean(c[:, 1]))
        
        # Calculate heading angle (theta) from marker front edge vector
        front_x = (c[0][0] + c[1][0]) / 2.0
        front_y = (c[0][1] + c[1][1]) / 2.0
        theta = np.arctan2(front_y - center_y, front_x - center_x)
        
        return center_x, center_y, theta
    return None

4. Hardware Troubleshooting & Real-World Deployment Results​

Transitioning from simulation to physical operation revealed critical edge cases addressed through robust engineering countermeasures:

🏆 Final Real-World Benchmark​

• Autonomous Goal Reaching Rate: Tested across 30 independent trials in the physical 1.2m × 1.2m obstacle arena, the robot achieved an 86.7% success rate (26/30 successful arrivals).
• Leveraging just 2,592 pre-computed discrete Q-values (~2.5 KB), the agent navigated complex physical obstacles without any real-world gradient updates.

5. Concluding the RL Q-Learning Dev Log Series​

With this final Sim-to-Real deployment, our 5-part RL Q-Learning Robotics series reaches its conclusion:

1. #1: 648-Bin State Discretization & Webots TCP Socket IPC Architecture
2. #2: Tabular Q-Learning ε-Decay Exploration & Multi-Objective Reward Shaping
3. #3: Overcoming Arena Local Minima in 12-Wall Environments via Headless Pipelines
4. #4: Quantitative Benchmark with evaluate_qlearning vs. SARSA & DQN
5. #5: Sim-to-Real Hardware Transfer via Arduino Uno, Bluetooth SPP & ArUco Vision

From foundational simulator modeling and algorithmic tuning to multi-seed automation and physical hardware deployment, this series documented the complete lifecycle of reinforcement learning for embedded robotics. Stay tuned for future deep dives into our next projects!

Over the two days of May 19th and 20th, the automation of evaluation for the 3rd fine-tuning (fine_tuning_3) and the enhancement of backend/frontend features for building the actual app service proceeded in parallel.

Concluding the 3rd Fine-Tuning and Automation Scripts​

Based on the insights gained from the previous fine-tuning, the training for the 3rd fine-tuning, which was set up with concepts of knowledge graphs and ontologies in mind, was completed.

"Select the checkpoint with the Best Eval from the experimental models in the fine_tuning_3 folder, replace the files in the adapter folder, and report the results." "Write a run_test_20_30.sh file that runs inference and evaluation in parallel for fine_tuning_3 as well."

Instead of manually digging through folders to copy checkpoints like before, I instructed the Artificial Intelligence (AI) to write a script that automatically extracted the best performing model weights (Best Checkpoint) and set them up in the adapter folder. Then, I executed the parallel inference script, run_test_20_30.sh, throwing the evaluation for the 4 Alias models to the background server.

Backend/Frontend: Chatbot Database (DB) Saving and Community Board Features​

While the models were hard at work running inferences on the server, I accelerated the mental-health DB and app integration work in my local environment.

"Summarize seed_data.sql. What is the pw for users? Is it Test1234!? What was the db I was trying to apply this sql to?"

To break through the login issue that caused the frustrating 401 Unauthorized error last time, I thoroughly examined seed_data.sql to verify the password hashing and user data.

As a result, I was able to leave a meaningful Git commit on the afternoon of May 20th: feat: Added chatbot DB save, board pagination, and emotion diary filter

Finally, the conversation logs of our fine-tuned Large Language Model (LLM) chatbot officially started being saved to the DB! In addition, the pagination feature for the community board where users can communicate, and the emotion diary filter feature for recording daily moods were successfully added to the frontend.

It was a thrilling moment when the massive, invisible fine-tuned model finally began to be served in the beautiful vessel of a mobile app.

📌 Key Takeaways (TL;DR)

• Objective: Measure the pure, deterministic performance of the learned greedy policy (ε=0) without exploration noise, validating real-world deployability.
• Methodology: Implemented evaluate_qlearning.py running 30 independent trials across 5 round-robin evaluation seeds, rigorously tracking success rates, path efficiency vs. theoretical shortest routes, and obstacle near-miss counts.
• Result: Tabular Q-Learning attained a 93.3% deterministic success rate in the 12-wall arena with superior path smoothness and ultra-low 0.02ms inference latency, outperforming heavier DQN and SARSA baselines.

1. Quantitative Evaluation Pipeline (evaluate_qlearning.py)​

Training reward curves are inherently noisy due to ε-greedy exploration. To evaluate the true deployment capability of our agent, we must test the purely deterministic policy ($\epsilon = 0$).

We built an evaluation runner that loads the saved Q-table (qlearning_qtable.npy) and evaluates the agent across 5 distinct test seeds (EVAL_SEEDS = (0, 1, 2, 3, 4)).

flowchart LR
    Model["Trained Q-Table<br>(qlearning_qtable.npy)"] --> Agent["Greedy Agent<br>(ε = 0 Fixed)"]
    EvalSeeds["5 Test Seeds<br>(EVAL_SEEDS 0..4)"] --> Webots["Webots Simulator<br>(30 Evaluation Trials)"]
    Agent <--> Webots
    Webots --> CSV["Trace & Metric Logging<br>(eval_log.csv / traces)"]

💻 Evaluation Loop Architecture​

# evaluate_qlearning.py excerpt
agent = QLearningAgent(seed=0)
agent.load(model_path) # Load trained Q-Table

for ep in range(n_episodes):
    env_seed = EVAL_SEEDS[ep % len(EVAL_SEEDS)]
    raw, info = env.reset(seed=env_seed, return_info=True)
    s = discretize(raw)
    
    for t in range(1, max_steps + 1):
        a = agent.act(s, deterministic=True) # Pure argmax Q(s, a)
        raw_next, r, done, info = env.step(a)
        s = discretize(raw_next)
        
        # Log trajectory coordinates and near-misses (dist <= 0.15m)
        record_metrics(info)
        if done: break

2. Comprehensive Evaluation Metrics​

1. Success Rate: Percentage of runs reaching the target radius (0.10m) within 150 steps without collision.
2. Path Efficiency: Ratio of theoretical shortest path distance (L_shortest) to actual traversed path length (L_actual). Path Efficiency (%) = (Shortest Path / Actual Path) × 100%
3. Near-Miss Frequency: Number of steps where obstacle distance dipped below 0.15m.
4. Oscillation Count: Metric penalizing back-and-forth steering jitter.

3. Algorithm Benchmark: Tabular Q-Learning vs. SARSA vs. DQN​

Comparative evaluation in the identical Webots 12-wall arena across the three reinforcement learning algorithms revealed clear performance trade-offs:

🔍 In-Depth Behavioral Analysis​

• Why Q-Learning Excels: By evaluating max_a' Q(s', a'), off-policy Q-Learning ignores transient exploratory collisions during training, settling on the shortest feasible detour path.
• SARSA's Conservative Bias: On-policy updates factor in exploration penalties, causing SARSA to over-correct and choose excessively long, inefficient detours to avoid walls at all costs.
• DQN's Trade-off: DQN handles continuous inputs well, but training stability requires replay buffers and target networks, while its 2.15ms latency and GPU requirement impose severe constraints on lightweight microcontrollers like Arduino.

4. Evaluation Takeaways​

Our discrete 648-bin Tabular Q-Learning model requires just 2.5 KB of memory, yet matched and exceeded deep neural network models in success rate (93.3%) and real-time execution speed.

5. What's Next?​

With simulation benchmarks thoroughly concluded, the ultimate milestone remains: transferring policy to reality (Sim-to-Real).

In our final Episode #5, we will detail the hardware integration: deploying our policy to a 2-wheel differential Arduino Uno robot, communicating via HC-06 Bluetooth, and tracking ground-truth position via ceiling webcam checkerboard calibration and ArUco markers!

📌 Key Takeaways (TL;DR)

• Problem: When a goal lies directly behind an obstacle, the myopic nature of distance-delta rewards (+0.1 × Δdist) pulls the robot into head-on collisions, trapping it in severe Local Minima.
• Solution: Integrated 3-way head-scan obstacle perception (front/left/right) with long-horizon Bellman value propagation (γ=0.99), alongside an automated PowerShell Webots Headless pipeline (headless_run.ps1) executing multi-seed training across seeds 0, 1, and 2.
• Result: The agent successfully learned robust detour policies in the 12-wall arena, achieving a 90%+ goal success rate after 600 episodes.

1. The Obstacle Challenge: The Local Minima Trap​

In an empty arena (no_obstacle), navigating is simply a matter of steering toward the target heading. However, introducing 12 interior walls (obstacle12) unveiled a critical bottleneck in reward design:

flowchart TD
    Start["Start (Robot)"]
    Wall["Obstacle Wall"]
    Goal["Target Goal"]
    
    Start -->|"Direct drive (greedy distance reward)"| Wall
    Wall -.->|"Collision & deadlocked (Local Minima)"| Wall
    Start ==>|"Detour Policy<br>Accepts temporary distance penalty"| Goal

• The Dilemma: When the target is directly on the other side of an interior wall, driving straight ahead leads directly to collision.
• Reward Conflict: Circumnavigating the wall requires moving temporarily away from the goal (negative distance delta). Under naive exploration, the agent enters a deadlock—spinning or timing out against the wall.

2. Solution Strategy: Head-Scan Perception & Value Backpropagation​

Overcoming this trap relies on coupling 3-directional head-scan perception with a high discount factor (γ = 0.99):

1. Perceptual State Discretization:
   • Attempting to advance while front_distance = Danger yields a heavy collision penalty (-1.0).
   • Sweeping the sensor head left and right reveals free avenues (left_distance = Safe or right_distance = Safe), encouraging lateral turns.
2. Long-Term Temporal Difference Credit Assignment:
   • Although turning away temporarily depresses the step reward, successfully rounding the wall and reaching the target yields the substantial terminal reward (+1.0).
   • Backpropagated through Q(s, a) with γ = 0.99, the agent gradually learns that detour steps possess higher expected return than head-on approaches.

3. Webots Headless Automation Pipeline​

Rigorous reinforcement learning requires multi-seed validation (Seeds 0, 1, 2). Running 1,500 episodes interactively in the Webots GUI three times would take many hours.

We automated this with a headless_run.ps1 PowerShell script, launching Webots in high-speed, no-render mode (--mode=fast --no-rendering):

💻 Automated Pipeline (headless_run.ps1)​

# Launch Webots in background headless fast mode
$webotsProc = Start-Process -FilePath $WebotsExe `
    -ArgumentList @("--mode=fast", "--no-rendering", "--stdout", "--stderr", $WorldFile) `
    -PassThru

# Train agent over socket IPC
python train_qlearning.py --scenario obstacle12 --seed 0 --episodes 1500

# Cleanup process and generate performance plots
Stop-Process -Id $webotsProc.Id -Force
python -m qlearning.plot_curves --log-dir logs/obstacle12/seed_0

4. Empirical Evaluation: Cross-Environment Learning Curves​

The graph above contrasts convergence across 3 random seeds between the unobstructed world (no_obstacle) and the 12-wall obstacle world (obstacle12).

📊 Benchmark Metrics​

• Extended Exploration: In obstacle12, discovering viable detour corridors requires approximately 200 additional episodes of random trials, keeping the average reward depressed until episode 400.
• Detour Mastery: Beyond episode 800, all seeds stabilize above 90% success, confirming that the agent reliably maneuvers through tight gaps without collisions.

5. What's Next?​

Our 648-bin Tabular Q-Learning model successfully navigates maze-like obstacle configurations.

In Episode #4, we will load the saved model checkpoints (models/*.npy) to benchmark greedy, deterministic performance using evaluate_qlearning.py and analyze comparative metrics against DQN and SARSA!

📌 Key Takeaways (TL;DR)

• Problem: Sparse reward setups (+1.0 at goal, -1.0 at collision) cause severe exploration delays, causing the agent to wander or spin for thousands of episodes before stumbling upon the target.
• Solution: Implemented a multi-objective Dense Reward Shaping mechanism combining potential-based Euclidean distance delta rewards (+0.1 × Δdist) with step (-0.01), turning (-0.01), and backward (-0.05) penalties, paired with exponential epsilon decay (1.0 → 0.05).
• Result: The moving average reward shifted into a steady positive regime (0.7+) within 500 episodes in the no_obstacle scenario, achieving a 98%+ goal success rate.

1. Tabular Q-Learning Update Formula & Agent Architecture​

Tabular Q-Learning is an off-policy Temporal Difference (TD(0)) algorithm grounded in the Bellman Optimality Equation.

The agent maintains an action-value table Q(s, a) structured as a 2D NumPy array of shape $(648, 4)$. Using the experience transition tuple $(s, a, r, s', done)$ streamed from the Webots socket server, the $Q$-values are updated in real time via:

📐 Q-Learning TD(0) Update Rule​

Target = r + γ * max_a' Q(s', a') * (1 - done)
TD Error = Target - Q(s, a)
Q(s, a) <- Q(s, a) + α * TD Error

• Learning Rate (α = 0.1): Balances the rate of incorporating new feedback versus retaining historical estimates.
• Discount Factor (γ = 0.99): Values long-term future returns, critical for multi-step navigation.
• Terminal Masking (1 - done): Clamps future expectations to zero upon goal arrival or collision, isolating the terminal reward.

💻 Agent Core Implementation (qlearning_agent.py)​

import numpy as np

class QLearningAgent:
    def __init__(self, n_states=648, n_actions=4, alpha=0.1, gamma=0.99,
                 eps_start=1.0, eps_min=0.05, eps_decay=0.995, seed=0):
        self.rng = np.random.default_rng(seed)
        self.Q = np.zeros((n_states, n_actions), dtype=np.float64)
        self.alpha = float(alpha)
        self.gamma = float(gamma)
        self.epsilon = float(eps_start)
        self.eps_min = float(eps_min)
        self.eps_decay = float(eps_decay)
        self.n_actions = int(n_actions)

    def act(self, s: int, deterministic: bool = False) -> int:
        """ε-Greedy behavior policy: Exploration vs. Exploitation"""
        if (not deterministic) and self.rng.random() < self.epsilon:
            return int(self.rng.integers(self.n_actions))
        return int(np.argmax(self.Q[s]))

    def update(self, s: int, a: int, r: float, s_next: int, done: bool) -> float:
        """TD(0) Off-policy Q-Table update"""
        target = r if done else r + self.gamma * float(np.max(self.Q[s_next]))
        td_error = target - self.Q[s, a]
        self.Q[s, a] += self.alpha * td_error
        return float(td_error)

    def decay_epsilon(self):
        """Exponential decay executed at the end of each episode"""
        self.epsilon = max(self.eps_min, self.epsilon * self.eps_decay)

2. Exponential Epsilon-Greedy Decay Strategy​

At the start of training, the agent has zero domain knowledge, requiring ε = 1.0 (100% random exploration) to discover the arena boundaries.

However, an uncurbed exploration rate induces perpetual collisions even after the optimal path has been resolved. We apply per-episode exponential decay:

ε_(ep+1) = max(ε_min, ε_ep × 0.995)

• Episode 1: ε = 1.0 (Full exploration)
• Episode 300: ε ≈ 0.22 (Shift toward exploiting high Q-values)
• Episode 600: ε = 0.05 (Floor exploration rate ε_min, 95% greedy execution)

3. Dense Reward Shaping for Sparse Navigation​

A fundamental challenge in mobile robotics is sparse reward feedback. In a 1.2m × 1.2m arena, moving randomly yields zero reward until the robot coincidentally intersects the $0.10m$ goal tolerance.

To guide convergence, we engineered a Dense Reward Shaping schedule inside the Webots controller (RL_controller.py):

🔍 Mathematical Principle of Distance Delta​

Δdist = dist_prev - dist_curr

• Driving $0.05m$ toward the target awards $+0.1 \times 0.05 = +0.005$ immediately.
• Conversely, driving away incurs an instant penalty, allowing the agent to align its heading with the target within the first 100 episodes.

4. Empirical Evaluation: Reward and Success Curves​

Training for 1,500 episodes in the no_obstacle world demonstrated remarkable convergence matching our theoretical expectations.

📈 Convergence Milestones​

1. Episodes 0 ~ 200 (Exploration Phase):
   • Frequent wall collisions and 150-step timeouts keep the moving average reward in the negative range (-0.4 ~ -0.2).
2. Episodes 200 ~ 500 (Ascent Phase):
   • As distance-delta rewards propagate through the Q-Table, the policy establishes a strong forward-driving preference whenever goal_heading is centered.
   • Success rate surges from 20% to over 80%.
3. Episodes 500 ~ 1,500 (Asymptotic Convergence):
   • With ε settled at 0.05, average steps per episode plummet from 150 (timeout) to 15 ~ 22 steps.
   • The 100-episode moving average success rate converges above 98.5%.

5. What's Next?​

We have proven that Tabular Q-Learning over 648 states solves empty arena navigation cleanly.

However, real-world environments are packed with walls and obstacles. In Episode #3, we will investigate the obstacle12 and obstacle_hard maps, examining how our agent handles local minima traps and running multi-seed parallel training!

I must correct my previous assumption that "May 13th was the last of Maeumjigi's May development"! On May 17th, a massive wave of backend integration testing and report drafting hit us.

Automating the Fine-Tuning Report using Artificial Intelligence (AI)​

There was simply not enough time for a human to manually analyze the massive result graphs from the 2nd fine-tuning (fine_tuning_2).

"Create a prompt to write a report comparing the inference results of the models. Save it as report_prompt.md and write the report."

I attempted prompt engineering to have the AI interpret the metrics and automatically draft the report. Ultimately, the AI analyzed the pros and cons of each model and the differences based on Alpha values, generating an excellent Word (.docx) report draft.

Backend DB Integration and the Swamp of 401 Unauthorized​

Right after the report, I started pushing backup.sql data into the local PostgreSQL DB to test the system stability of the frontend and backend.

"I logged in successfully after running the backend and frontend, but nothing responds when I click menus." "Email or password incorrect... INFO: POST /auth/login 401 Unauthorized"

I successfully pushed the DB dump and accessed the web, but fell into a hell of 401 Unauthorized login errors. I created debug_login.py and endlessly restarted the uvicorn server for troubleshooting.

Contemplating Knowledge Graphs​

"Techniques using ontology and knowledge graphs are rising. Using these might create a chatbot with better stability than just focusing on a single Large Language Model (LLM)."

Late at night, to overcome the limitations of simple LLM fine-tuning, I began designing chatbot_design_prompt.md to explore fusing ontologies and Knowledge Graphs. As we pass mid-May, the scale of the project is growing massive!

📌 Key Takeaways (TL;DR)

• Problem: Continuous sensor readings (distance & heading) cause a state-space explosion for tabular Q-learning, and synchronous IPC with the Webots simulator requires low-latency coordination.
• Solution: Designed a geometrically partitioned 5-dimensional continuous sensor mapper producing 3 × 8 × 3 × 3 × 3 = 648 discrete states (discretizer.py) and a line-delimited JSON IPC socket pipeline over TCP port 5000.
• Result: Achieved O(1) fast Q-Table indexing and reliable 32ms physics synchronization between the simulator and Python agent.

1. Project Background: Why Webots and Q-Learning?​

Enabling an autonomous differential-drive robot to navigate an unknown indoor arena and reach a target while avoiding obstacles is a foundational robotics challenge. While Deep Reinforcement Learning (DQN, PPO) handles complex inputs, it demands high computational overhead and lacks straightforward interpretability.

In this project, we apply classical Tabular Q-Learning (Off-policy TD(0)) to a mobile robot in the Webots 3D physics engine. Our objective is to establish an optimal, mathematically verifiable navigation and obstacle avoidance policy in a 1.2m × 1.2m indoor arena.

2. System Architecture: Webots Controller ↔ Python RL Engine​

The system is decoupled into two clean tiers: the Webots simulation server and the Python RL client.

flowchart LR
    subgraph Webots["Webots 3D Simulator (localhost:5000)"]
        Robot["Supervisor Robot<br>(Ultrasonic Sensor + Wheels)"]
        HeadScan["Head-scan Mechanism<br>(Left/Right 1.1 rad servo)"]
        Server["TCP Socket Server<br>(Line-delimited JSON)"]
        Robot -→ HeadScan -→ Server
    end

    subgraph Python["Python RL Engine (Client)"]
        Env["WebotsEnv (gym-like)<br>reset() / step()"]
        Disc["Discretizer<br>(Continuous → 0~647 Bin)"]
        Agent["QLearningAgent<br>(Q-Table 648x4)"]
        Env -→ Disc -→ Agent
    end

    Server <-→|"request: {type: step, action: 0..3}<br>response: {state: [...], reward, done}"| Env

• Webots Controller (RL_controller.py): Executes action commands for 480ms (15 simulation ticks × 32ms) and rotates a head-mounted ultrasonic sensor left and right by 1.1 rad ($\approx 63°$) to eliminate blind spots.
• Socket Protocol: Lightweight line-delimited JSON over TCP localhost:5000:
  • Request: {"type": "step", "action": 0} (0: Forward, 1: Backward, 2: Turn Left, 3: Turn Right)
  • Response: {"state": [0.45, 0.12, 0.35, 0.28, 0.40], "reward": 0.05, "done": false, "info": {...}}

3. Discretizing Continuous Sensors into 648 States​

To apply tabular Q-learning, continuous floats must be mapped into finite discrete states. We selected 5 crucial observation variables based on kinematic relevance:

Total State Count: N_states = 3 × 8 × 3 × 3 × 3 = 648

The resulting Q-Table requires only 648 × 4 = 2,592 floating-point entries, allowing sub-millisecond memory updates and zero GPU overhead.

💻 State Discretization Code (discretizer.py)​

import numpy as np

# Dimension boundaries
N_GOAL_D, N_GOAL_H, N_FRONT, N_LEFT, N_RIGHT = 3, 8, 3, 3, 3
N_STATES = N_GOAL_D * N_GOAL_H * N_FRONT * N_LEFT * N_RIGHT  # 648

_HEADING_BINS = np.linspace(-np.pi, np.pi, 9)[1:-1]
_NEAR_BINS    = np.array([0.15, 0.30])
_DIST_BINS    = np.array([0.6, 1.2])

def discretize(state_dict: dict) → int:
    """Maps 5-tuple continuous sensor readings to an integer in [0, 647]."""
    gd = int(np.digitize(state_dict["goal_distance"], _DIST_BINS))
    gh = int(np.digitize(state_dict["goal_heading"], _HEADING_BINS))
    df = int(np.digitize(state_dict["front_distance"], _NEAR_BINS))
    dl = int(np.digitize(state_dict["left_distance"], _NEAR_BINS))
    dr = int(np.digitize(state_dict["right_distance"], _NEAR_BINS))

    # Compute flattened 1D index
    index = ((((gd * N_GOAL_H + gh) * N_FRONT + df) * N_LEFT + dl) * N_RIGHT + dr)
    return int(index)

4. Head-Scan Mechanism & Collision Logic​

A fixed forward-facing sensor leaves lateral blind spots. We implemented an active Head-Scan routine:

1. Step Execution: Drive motors at 3.0 rad/s for 480ms.
2. Head Rotation: Sweep head servo +1.1 rad to read left distance, then -1.1 rad for right distance.
3. Re-center: Return head to 0 rad and refresh front distance.
4. Terminal Conditions:
   • Goal Reached: Euclidean distance ≤ 0.10m → success (done = True, reward $+1.0$)
   • Collision: Any distance ≤ 0.08m → failure (done = True, reward $-1.0$)

5. What's Next?​

With the simulator environment, TCP socket IPC, and 648-bin state discretization established, our agent is ready for reinforcement learning.

In Episode #2, we will cover the mathematical implementation of Tabular Q-Learning, ε-greedy exploration decay, and reward function tuning!

On May 13th, after fixing the hardcoding bugs discovered yesterday, we finally resumed proper inference testing for the 2nd round fine-tuned models.

The Cause of Short Answers: Increase the Token Length!​

While meticulously reviewing the inference results, I felt that the answers from the Llama33-70B_a8 model were still strangely short and rigid.

"This is one of Llama33-70B_a8's answers. I feel that max_seq_length and max_new_tokens are too small. I want to increase them. Looking at the VRAM status, how much can I increase them?"

After calculating the available VRAM space (39.8GB at the time) with the Artificial Intelligence (AI), we boldly increased max_new_tokens (the maximum number of words generated) to 384. The result was a massive success!

[Before] "It must be really hard to feel that way. It is important to talk to a professional."
[After] "It must be really hard to feel that way. However, having these thoughts is not a sign that you are weak, but a signal that you are carrying too much burden right now. I highly recommend slowly unraveling these emotions with a professional."

By simply increasing the token allowance, the model maintained its context to the end and began pouring out much deeper and warmer sentences of empathy. The model had been holding its tongue because it was trapped in a limited token count.

Alpha Value Comparison Analysis (a=8 vs a=16)​

The results for the biggest objective of the 2nd fine-tuning—the performance difference based on the LoRA hyperparameter alpha—were also out. By analyzing the logs piled up in the fine_tuning_2 folder, we generated graphs like 05_alpha_comparison_daily.png. With rank=16 fixed, we clearly visualized how the model's overfitting patterns and final answer quality differed when given alpha=8 versus alpha=16.

Ultimately, models granted the correct hyperparameters and ample token length were finally getting closer to the true 'Maeumjigi' (Heart-keeper) we had originally planned. It seems our mid-term presentation preparations will wrap up very smoothly!

The 2nd round of fine-tuning (fine_tuning_2) parallel training, which was ambitiously started to correct the LoRA hyperparameter mistake (alpha/rank setup error), finally ended on May 12th!

Extracting the Best Checkpoint and Building the Inference Pipeline​

The fine-tuning for 3 models with 6 hyperparameter combinations was all completed. I immediately compared the eval_loss to extract the best performing checkpoints and moved them to the adapter folders for each model.

I also created a new run_test_20_30.sh script to run evaluation and result comparison all at once in the background on the server.

"Currently, the HCX-Text-1p5B models generated 12 outputs in 15 minutes. GPU Util is 1.6GB per graphics card..."

However, the inference speed was still hopeless. Even the lightest 1.5B model couldn't even generate 1 answer per minute. The VRAM usage was only 1.6GB, meaning a bottleneck occurred because the Batch Size was set to 1. On top of this, server storage issues forced us to urgently migrate the project's base path from ~/home/ to ~/home/data/temporary/, causing a huge fuss to modify all script paths.

Falling into the Trap of Hardcoding​

After painstakingly running all the evaluation codes, I discovered a horrifying fact.

"Why is MODEL_ALIAS hardcoded to 'Qwen35-9B_a16' in 20_eval_ft_models.ipynb?" "So in the server logs, even for 20_eval_HCX-Text-1p5B_a8.ipynb, MODEL_ALIAS was defined as 'Qwen35-9B_a16'. You're telling me it's normal that the filename and model name are different?"

Yes. While writing the evaluation automation script, I accidentally hardcoded MODEL_ALIAS. The script diligently copied files to test the HCX model, but inside the code, it kept repeatedly loading only the Qwen35-9B_a16 model and doing the same inference over and over!

It was a day that made me realize once again that fine-tuning is not a science, but a battle of meticulousness. I'll have to fix this hardcoded bug and run the inferences again tomorrow.

May 8th was the day we were building our capstone design mid-term presentation (PPT) based on the fine-tuning results. However, while happily filling in the slides, I discovered a massive, chilling mistake.

Discovering the Fatal LoRA Hyperparameter Mistake​

I was refining the model structure diagrams and hyperparameter info with the Artificial Intelligence (AI) for the PPT.

"Looking at the fine-tuning examples, the answers are shorter than the base model. I think this is because setting LoRA's r to 8 and alpha to 16 reduced the intermediate features too much..." "Wait, it was a/r? I thought it was r/a...??? I really applied LoRA greedily..."

When fine-tuning with LoRA, it's standard to set the alpha value equal to or twice the rank value. However, I completely misunderstood the scaling ratio equation, setting it in a way that either severely suppressed or excessively amplified the weight updates. No wonder the fine-tuned models' answers were unnaturally short and chopped off—the fault lay right at my fingertips.

The Decision for 2nd Round Fine-Tuning (fine_tuning_2)​

Although we already had the 1st round results, we couldn't just move on for a perfect 'Maeumjigi'. I immediately created a fine_tuning_2 folder and decided on a 2nd round of fine-tuning.

"The models to use this time are HCX-Text-1p5B, Llama33-70B, and Qwen35-9B. Train on MIG-0 with rank=16, alpha=16, and on MIG-1 with rank=16, a=8." "Extract the top 2000 longest entries from Daily_train_dataset.jsonl and create a file named Daily_train_dataset_long.jsonl."

To prevent the answers from becoming too short, we extracted only the top 2000 datasets with the longest text length. We also split our MIG (Multi-Instance GPU) resources to run parallel training, directly comparing the performance differences based on the rank and alpha ratios.

Starting Frontend (App) Development While the Backend Trains​

While the models fell back into the cycle of training, I didn't just sit around. I finally started developing the frontend app where actual users will meet 'Maeumjigi'!

"I need to show a QR code to run maeum-app... but when I type npx go start, I only get an error message. How and where can the QR code be shown?"

I created the maeum-app project using React Native (Expo) and struggled to launch the local server and connect it to my smartphone. I even mistakenly typed npx go start instead of expo, but I finally succeeded in pulling up the Expo QR code and setting up the app icons (assets/logo.png). It was a thrilling day where the brain of the backend (AI) and the face of the frontend (App) were being built simultaneously.

May 7th was the day we synthesized the fine-tuning results of our 5 models (Llama3.3-70B, DSR1-70B, HCX-Omni-8B, Qwen35-9B, HCX-Text-1.5B) and proceeded with full-scale visualization and local environment setup.

Visualization and Local Setup for Model Comparison​

To comfortably review the result comparison code (which was previously running on the heavy server) on my personal laptop, I created a new notebook, 31_comparison_local.ipynb.

"Can't 30_comparison.ipynb run on my laptop too?" "Please tell me how to install pandas, matplotlib, seaborn, and bert_score on the currently selected kernel."

While setting up the local Jupyter kernel in VSCode and installing the necessary libraries, I faced frequent Kernel Crashes and annoying errors like ModuleNotFoundError: No module named 'ft_utils'. However, I resolved them step-by-step by fixing folder paths. I also successfully recovered the missing ft_metrics.json files for some models (like DSR1-70B) using a script.

5-Model Comprehensive Report and Graph Visualization​

The most important task was visualizing the performance of these 5 models so they could be compared at a glance. Together with the AI, we established a plan named 5MODELS_ORGANIZATION_PLAN_present.md and began systematically organizing the data.

"What is the difference between the faint line and the thick line? Bring the training loss and eval loss for each model and visualize them as graphs." "Remove the thick line and make the faint line clearer and save it."

I removed the unnecessary trend lines (thick lines) from the matplotlib graphs generated by the AI, finally extracting 03_training_loss_all.png and 04_eval_loss_all.png which neatly gathered the Training and Eval Loss curves of all models in one place. We also perfectly documented graphs analyzing the LoRA Rank Decomposition for each model, as well as a GPU memory usage chart (02_gpu_memory.png).

Now, based on these graphs, we can beautifully decorate our capstone design mid-term presentation materials for next week. Which of these 5 models will ultimately be selected as the best quality 'Maeumjigi' chatbot?