Video summary

DSAI HDA AVS Tech Skill 4: Data Discovery 101 Finding, Cleaning, and Preparing Data for AI Models

Main summary

Key takeaways

Educational

Main ideas / lessons (what the speaker conveys)

  1. AI systems in industry require strong “data discovery → exploration → cleaning → preparation”

    • Data is needed to support models/assistants; raw data rarely works as-is.
    • The process is iterative: explore to understand patterns and issues, clean to fix quality problems, then prepare for model/RAG use.

  2. Solve real operational problems with AI helpdesks/ticketing automation

    • The video centers on a ticketing use case where first-response SLAs are missed and many requests are repetitive.
    • Goals include faster responses, reduced human effort, and more consistent troubleshooting.

  3. Data sources for training/knowledge bases

    • Use a mix of:
      • Public data from the internet (e.g., Kaggle, Hugging Face, Google, ML repositories)
      • Internal employee feedback (internal discussion + Q&A capture)
      • Scraping from historical ticketing systems
      • Surveys (e.g., using Google Forms)

  4. Industry-oriented “AI adoption” principles

    • Emphasizes reducing human dependence for critical data handling.
    • Uses automation triggered by events (e.g., onboarding/offboarding).
    • Adds audit trail/anomaly detection to catch inconsistencies before business impact.

  5. Practical exploration analytics before training

    • Detect outliers/anomalies (e.g., HTTP errors like 404/401 from scraping)
    • Detect missing values (data completeness varies over time/internet sources)
    • Find relationships/correlations among features and target variables
    • Decide which features are most useful and how to handle empty/zero values
    • Optionally remove features that don’t relate to the target

  6. Cleaning improves downstream model quality

    • Remove irrelevant/inappropriate content (e.g., emoji/noise, wrong-language characters)
    • Correct/delete inaccurate or incomplete data so the dataset quality is maintained.

  7. Preparing data for AI: RAG and ML/DL frameworks

    • Mentions using tools/frameworks such as:
      • TensorFlow
      • Chroma/ChromaDB for vector storage (for LLM/RAG)
      • scikit-learn for classic ML experiments
    • Suggests experimenting with traditional ML vs deep learning, and comparing results.

  8. Example real system architecture (IT helpdesk automation)

    • Uses RAG-style flow:
      • Receive incoming message/ticket (from Trustdes → ticket)
      • Capture via n8n
      • Retrieve relevant vector context using embeddings + similarity (mentions BM25/similarity index/RG retrieval)
      • Build a prompt and generate an answer with a tuned model
    • Includes branching workflows for different intents (e.g., monitoring alerts vs onboarding/offboarding vs password reset vs general requests).

  9. Operational concerns: avoid unsafe or costly “auto-reply”

    • Direct auto-replies can be:
      • Expensive (token costs)
      • Unreliable (hallucinations / wrong pricing / incorrect answers)
      • Risky for account health (users reporting spam → account can be blocked)
    • Proposed safer approach:
      • Use a system that prepares responses based on knowledge base + context, rather than blindly auto-replying.

  10. End-user workflows across departments

    • Examples of departmental AI assistant roles:
      • CMO/Marketing: content scheduling/creation; integrates with platforms like Tokopedia/Shopee/Meta Ads/Google Ads
      • COO: meeting transcription to text and notes
      • CFO: analytics/dashboard support for finance
      • CHRO/HR: CV upload → convert to vector DB → rank candidates via similarity/search
      • CTO/IT: semi-coding assistance; reporting app/dashboard generation; code assistance using coder LLMs

Methodology / step-by-step instructions presented (detailed bullets)

A) Data pipeline for AI readiness (as described in the video)

  1. Data discovery / collection

    • Search for relevant datasets externally (e.g., Kaggle, Hugging Face, Google, ML repositories).
    • Use internal discovery:
      • Interview employees / observe recurring user questions that humans repeatedly handle.
      • Convert collected issues into a Q&A dataset.
    • Use scraping from prior ticket systems (e.g., historical helpdesk/ticket logs).
    • Conduct surveys (e.g., Google Forms or an internal app) to ask what users need.

  2. Internal discussion & problem cataloging

    • Identify what users commonly ask.
    • Identify what users can solve themselves vs what actually needs assistance.
    • Consolidate into structured Q&A.

  3. Exploratory Data Analysis (EDA)

    • Recognize patterns in existing datasets before investigation/training.
    • Detect:
      • Outliers / anomalies (unexpected HTTP errors like 404/401; unusual records)
      • Missing values (count empty/NaN values; handle incomplete internet data)
    • Identify:
      • Correlations / relationships between features and the target variable.
      • Filter/remove irrelevant feature relationships.
    • Understand which features are:
      • Most relevant to the outcome
      • Worth keeping vs dropping
    • Handle empty/zero values (decide a cleaning/handling strategy).

  4. Data cleaning

    • Detect and correct/delete “dirty” or inaccurate/incomplete records.
    • Examples of cleaning actions:
      • Remove irrelevant emojis/noise from support text.
      • Remove questionable/unwanted characters/language contamination (e.g., Chinese characters in an otherwise intended dataset).

  5. Data preparation for model/RAG usage

    • Convert cleaned data into forms usable by:
      • Vector database + RAG pipeline (e.g., embeddings stored in ChromaDB)
      • ML experiments (e.g., scikit-learn classification)
      • Deep learning experiments (mention TensorFlow and neural networks)

  6. Training / knowledge base building

    • Train a knowledge base / RAG index on cleaned data.
    • Support truly “raw” inputs (video claims it can accept JSON/HTML/markdown-table-like data for training).
    • Evaluate by:
      • Testing retrieval (“best three” similar items)
      • Running RAG with context and language detection (Indonesian vs English)
    • Generate output in a troubleshooting format using a prompt template.

B) IT helpdesk RAG automation workflow (system-level steps)

  1. Trigger

    • User creates/updates a ticket (from Trustdes / ticketing flow).
    • System captures the message via n8n webhook.

  2. Pre-checks / intent routing

    • Determine whether the request matches:
      • monitoring alert
      • onboard/offboard
      • password reset
      • general troubleshooting
    • If not matched, create an AI-assisted ticket workflow.

  3. Summarization (if needed)

    • Summarize long messages to reduce resource use.

  4. Retrieval (RAG)

    • Use trained vector store / embedding retrieval:
      • Retrieve similar knowledge entries using embeddings + similarity index (mentions BM25/similarity).
    • Build the prompt with retrieved context.

  5. Response generation

    • The model answers using troubleshooting steps format.
    • If RAG context is insufficient, return a default fallback reply.

  6. Code assistance (when requested by users)

    • For IT code errors, route to a coder model (mentions “2.5 coder/em coder” usage).

  7. Execution on branches/workflows

    • For alerts: follow alert workflow; avoid duplicate alerts within a time window (mentions “within last 1 hour”).
    • For offboard: delete/unlink data if required.
    • For general RAG troubleshooting: use RAG pipeline.

C) Safer WhatsApp assistant approach (vs naive auto-reply)

  • Avoid direct auto-reply from Open Cloud to WhatsApp immediately (risks high token cost and potential account blocking).
  • Use n8n + a custom controlled application/webhook flow:
    • Route incoming messages to n8n
    • Retrieve context (RAG) and generate replies
    • Send reply using message payload containing recipient + text (or media)

Further safer tactic

  • Prefer “auto recommendation” or context-aware assistant instead of unconditional auto-reply.
  • Use a UI/extension/app that reads context from previous messages and group/name/time to improve personalization.

Speakers / sources featured (as mentioned in the subtitles)

  • Mr. Masarf (also rendered as Masah / Marsada / Masad / Mr. Syaden / Mr. Sadan / Mas Yusuf in various subtitle fragments): the main speaker presenting the technical session.
  • Unnamed audience / students (participants asking questions during Q&A).
  • TAS Cosmos Indonesia: company/source of the described industrial system and workflows.
  • Transdes: application mentioned for the ticketing module.
  • Trustdes: application mentioned for becoming a member; also referenced as a source for ticketing flow.
  • Open Source ticketing systems referenced
    • OTicket / OS Tiket (rendered inconsistently)
    • GLPI (referenced as a ticketing/source system)
  • Platforms/tools referenced
    • Google Forms
    • Kaggle
    • Hugging Face
    • Chroma / ChromaDB
    • TensorFlow
    • scikit-learn
    • n8n
    • Grafana
    • BM25 / similarity index (retrieval components)
    • OpenAI-compatible / Open-compatible / OpenELLM family (general ecosystem reference)
    • Anthropic-compatible (mentioned)
    • WhatsApp / Meta (used in the Q&A and discussion of risks)
    • GitHub (model/app link mentioned as private on GitHub)
    • Cloud/Open Cloud (as referenced in Q&A)
    • Framebox notjs / baayis socket (WhatsApp bot integration mentioned, with uncertain spelling)

Original video