Boosting Software Efficiency : A Case Study of 100% Performance Improvement in ... - Gili Kamma

Summary of technological concepts, product features, and analysis (case study)

Legacy embedded system context (smart metering water domain)

• The device collects wireless water meter readings (meter sends data every ~16 seconds depending on configuration).
• A central backend (“ready manager”) performs billing; data loss directly means money loss and fines.
• The embedded unit runs on Linux (older kernel, openembedded/LEOP-like distro not fully modern) with limited resources (~32 MB RAM).
• Software stack/constraints:
  • C++17 (but effectively C++11-style implementation; e.g., limited/absent modern safety features like smart pointers).
  • Qt-based application (high-level embedded framework; encourages allocations without enforcing timely release; no garbage collector).
  • Dynamic allocations + memory management issues were central to crashes/resets.

Scaling failure mode

• Hardware was claimed/sold to support up to ~7,500 meters.
• Reality: increasing from <1,000 meters to thousands caused instability:
  • Crashes and unexplained resets
  • Out-of-memory behavior (memory issues despite theoretical capacity)
  • Occasional data loss, especially during network instability
• Errors were hard to debug because crashes and data loss didn’t reproduce in the office environment.

---

Goals and overall strategy

• Primary goal: reach zero data loss and make the system stable.
• Approach:
  1. Diagnose memory and concurrency issues.
  2. Build internal tooling to reproduce real-world load conditions.
  3. Improve message sizing/encoding/compression behavior.
  4. Add reset-time log shipping to recover debugging visibility.
  5. Redesign networking/data flow to be resilient to network failures.
  6. Add ongoing monitoring + automation.
  7. Build stronger testing pipelines and release discipline.

---

Key engineering tools and methods introduced

1) Custom memory profiler via operator overloading

• Overloaded new and delete in C++ to track allocations/releases.
• Added bookkeeping:
  • Stored allocation size in extra bytes so release could update profiler accurately.
• Produced metrics to detect:
  • Allocation rate growth
  • Current vs. maximum bytes increasing over time
  • Per-size allocation counts changing over time
• Used these insights to find and eliminate small memory leaks, though the biggest crash cause was elsewhere.

2) Internal load simulator embedded inside the unit

• Implemented a simulator mode within firmware (enabled by a file flag).
• For each real incoming meter message, it generated up to ~100 fake meter messages with unique IDs.
• Result: the system “believed” it had thousands of meters (simulated up to ~6,000) and could reproduce crashes reliably.
• Aggressive configuration:
  • Tightened message intervals (e.g., hourly → every 5 minutes) to force worst-case load.
• Advantage:
  • No special external hardware; just ~10 lines of simulation logic and no separate build.

---

Four major problems and how they were solved

Problem A: Unexplained crashes at large site scale (>5,000 meters)

• Used the simulator + a conditional breakpoint (stop when pointer == null) to catch failure before further damage.
• Root cause:
  • Two threads tried to create very large messages (~5 MB each) concurrently.
  • One message failed due to not enough memory → null pointer access → crash/reset.
• Fixes:
  1. Reduced concurrency: removed one of the simultaneous threads (changed async to more sequential processing).
     • Memory peak reduced from ~10 MB → ~5 MB, eliminating crashes.
  2. Split large JSON payloads into batches:
     • Message size scaled with number of meters.
     • Instead of one giant encrypted/compressed message, process in chunks/batches (e.g., 1,000 meters at a time).
  3. Reduced memory usage in Qt string/data handling:
     • Optimized only the message-preparation function/data (not the rest of Qt usage).
     • Memory peak reduced further (e.g., 1 MB → 0.5 MB).
  4. Outcome: >2× more meters supported (from ~5,000 toward 10,000 later).

Problem B: Missing diagnostic info after resets (limited logging visibility)

• Existing event mechanism sent partial logs during normal operation, but:
  • During crashes/resets, event queues could be empty, losing context.
• Fix:
  • On wake-up after reset, send the last ~100 lines of nonvolatile/application logs as an event to the backend.
• Result:
  • Better visibility into what happened just before reset, enabling faster root-cause analysis.
• Emphasis:
  • “Good enough” engineering—tolerate minor inefficiency to ensure critical diagnostics always arrive.

Problem C: Occasional data loss during network instability or resets

• Behavior:
  • Data waited in RAM awaiting network transmission/processing.
  • With unstable network, pending data accumulated until memory pressure caused crashes.
  • If a reset occurred during transmission, RAM-held data could be lost.
• Fix: decouple logic from networking with a queue + nonvolatile storage
  • Thread 1: always generate data at scheduled times and store messages in a nonvolatile queue (survives resets).
  • Thread 2: continuously attempts to send queued messages.
    • On success: delete/advance.
    • On failure (timeout/no ack): keep message, retry later.
• Outcome:
  • Maximum data loss bounded (e.g., if data is collected hourly and deleted only after confirmed receipt, loss is limited to at most ~one interval).
  • Tolerates long network outages; data eventually transmits when network returns.

Problem D: Sudden “no meters detected” / receiver alignment failure

• Symptom:
  • Another component (“RF receiver unit” near antenna) stopped producing frames.
  • Main unit still connected to backend but received nothing.
  • System didn’t automatically recover.
• Fixes evolved in steps:
  1. Implemented a silence detection timer and performed a controlled software reset
     • Initially around 1 hour missing data → later tightened to ~4 minutes.
  2. Identified deeper cause:
     • The RF receiver unit reset itself, producing junk frames.
     • UART/driver couldn’t realign to subsequent valid frames.
  3. Final solution:
     • Instead of full unit reset, reinitialize UART upon detecting silence.
     • Controlled reset had worked only because it forced full driver reinitialization after wake-up.
• Outcome:
  • Improved customer-visible reliability by shortening recovery time and correcting the underlying low-level failure mode.

---

Monitoring and testing improvements (process/productivity engineering)

Proactive monitoring using automated event analysis

• Problem: complaints-based debugging; manual log inspection was too slow and reactive.
• Solution:
  • Export events to CSV via a Python script for Excel analysis (sorting/filtering; detect gaps and patterns).
  • Graph trends over days/weeks/months:
    • message timing gaps
    • memory usage over time
    • error counters across fleets of units
  • Fleet-scale summary:
    • When managing thousands of concentrators, summarize per-unit signals so anomalies stand out (e.g., firmware mismatch or abnormal error counts).
  • Automated nightly runs + email reports to reduce manual triage.

Testing automation with pipeline separation

• Replaced a very large manual test plan (100 pages) with automated tests.
• Added two CI pipelines:
  • Short tests vs long tests so long tests don’t block short feedback loops.
• Ran tests 24/7 repeatedly to eliminate flaky behavior:
  • If a test became flaky once (red/unstable), remove and fix before trusting it.
• System-level test strategy:
  • Maintain two groups of units:
    • 24/7 with aggressive simulator load
    • 24/7 with standard configuration
  • Use monitoring to verify both:
    • negative outcomes (crash/errors)
    • positive outcomes (correct scheduling/behavior)

---

Results achieved

• After ~8 months: product stabilized with no resets and scaled to ~10,000 water meters (vs ~5,000 before).
• After ~2 years: testing automation + semi-automated monitoring enabled releases every 2–3 months (previously much slower), and confidence increased enough to safely add features.

---

Main speakers/sources (as stated in subtitles)

• Speaker: Gili Kamma (Gili/Gilli Kamma)
• Video title reference: “Boosting Software Efficiency : A Case Study of 100% Performance Improvement in … - Gili Kamma”