Narzędziownik AI V 3/5 2025

Summary of “Narzędziownik AI V 3/5 2025”

This extensive training video focuses on local AI models, their installation, configuration, usage, and fine-tuning, targeting both individual users and companies. The session is highly technical and educational, covering multiple tools, platforms, and concepts related to offline large language models (LLMs), retrieval-augmented generation (RAG), and AI deployment in business contexts.

---

Key Technological Concepts and Product Features

1. Local AI Models and Offline Usage

• Importance of offline/local models for privacy, data security, and control.
• Tools to run local models include:
  • Ollama: A lightweight hypervisor for managing local models as container-like micro virtual machines. Supports downloading verified models, running them as services, and interacting via command line.
  • Open Web UI: A web-based graphical interface to interact with local models managed by Ollama. Requires Docker (Docker Desktop on Windows) for containerization.
  • LM Studio: A popular local AI model interface with a graphical UI, easy for beginners. Supports downloading and running various models, especially Chinese models like DeepSik and Kuven.
  • Use Anything LLM: A more advanced hypervisor combining features of LM Studio and Open Web UI, allowing extensive configuration, agent creation, and integration with multiple LLM providers.
• Model types discussed include LLaMA, DeepSik, Kuven, Mistral, Gemini, and more.
• Model formats: GPTQ, GGUF, GGML, quantized vs full precision (float16, float32).
• Hardware requirements vary by model size and precision; examples given for RAM, VRAM, CPU cores, and GPUs (NVIDIA recommended).

2. Model Management and Configuration

• Downloading models via commands like ollama pull or lmstudio download.
• Managing models locally includes loading, unloading, setting parameters (temperature, top-p, top-k), clearing context, and saving/loading sessions.
• License considerations: MIT licenses common but always verify before commercial use.
• Segmentation of users and permissions in Open Web UI for enterprise deployment.
• Support for multi-language, though some smaller models struggle with Polish tokenization.
• Integration with Active Directory for enterprise user management.

3. Retrieval-Augmented Generation (RAG)

• RAG combines a retriever model (lightweight embedding-based search) with a generative model.
• The retriever indexes documents, code, or datasets; the generative model uses retrieved context to answer queries.
• RAG is a cost-effective alternative to full fine-tuning.
• Handling large codebases (e.g., 90,000 lines) with chunking, embedding, and indexing.
• Popular retriever models: BG BGE, Instructor, CodeBERT.
• RAG improves accuracy and reduces hallucinations by grounding answers in provided documents.

4. Fine-Tuning and Training Local Models

• Overview of training methods:
  • Retraining from scratch (very expensive, e.g., $20M for LLaMA 3).
  • Fine-tuning (training parts of the model, more accessible with LoRA adapters).
  • RAG (augmenting with external knowledge bases).
  • Chat contextual training (ephemeral session-based learning).
• Use of LoRA adapters and Unslot AI library for efficient fine-tuning on smaller datasets.
• Training demonstrated using Google Colab with GPU (T4 recommended).
• Dataset preparation, tokenization, and standardization are critical.
• Training progress monitored via loss metrics; aim to reduce randomness and hallucinations.
• Fine-tuning allows models to specialize (e.g., legal, cooking, programming).
• Ability to save fine-tuned models or adapters and deploy locally.
• Discussion on data privacy when using cloud services for training.

5. Integration and Automation

• Local models can be exposed via web interfaces, APIs, or integrated into business workflows.
• Mention of automation tools like Zapier, Make, and N8N for workflow automation (covered in next sessions).
• Use of vector databases and embedding engines for efficient document search and retrieval.
• Support for text-to-speech (TTS) and speech-to-text (STT) modules, with caution about data privacy when using web APIs.
• Potential for multi-agent systems, though currently limited.

6. Additional Features and Considerations

• Model evaluation and rating (ELO rating system) for selecting best-performing models in production.
• Handling document parsing and OCR via tools like Tika.
• Security best practices: verifying sources of models, firewalling, and monitoring API traffic.
• Licensing and copyright considerations for AI-generated content.
• Hardware recommendations for various use cases (gaming PCs, servers, Mac M1/M2, Xeon processors).
• Discussion of hallucination and mitigation via model parameters and RAG.
• Examples of practical use cases: contract checking, programming assistance, chatbot deployment, and office automation.

---

Guides and Tutorials Provided

• Step-by-step installation and setup of Ollama and Open Web UI on Windows/Linux/Mac.
• Using Docker Desktop for containerizing Open Web UI.
• Command-line usage for downloading and running models.
• Setting up user roles, groups, and permissions in Open Web UI.
• Demonstration of loading and querying models locally.
• Preparing datasets and running fine-tuning scripts in Google Colab.
• Explanation of LoRA adapters and how to apply them for efficient fine-tuning.
• Managing context windows, session memory, and prompt engineering.
• Handling RAG knowledge bases: uploading files, creating searchable spaces, and querying.
• Tips on hardware requirements and model selection based on system specs.
• Overview of integrating local models with APIs and external services securely.
• Brief mention of upcoming sessions covering automation (Zapier, N8N) and business use cases.

---

Main Speakers and Sources

• Tomasz Turba (referred to as “Uncle Tomek”): Primary speaker and trainer. Expert with 16+ years in cybersecurity, AI, and hacking. Works for Securitum and Sekurak.pl. Leads the AI Toolmaker training series.
• Supporting colleagues and community members providing answers and clarifications during Q&A.
• References to external projects and companies:
  • OpenAI (GPT models, Codex)
  • Ollama (local model hypervisor)
  • LM Studio (local AI UI)
  • Use Anything LLM (advanced local LLM manager)
  • Hugging Face (model and dataset repository)
  • Unslot AI (fine-tuning library)
  • Docker (containerization)
  • Tika (document parsing)
  • Google Colab (training environment)
  • Various AI models: LLaMA, DeepSik, Kuven, Mistral, Gemini, CodeLlama, Whisper (speech), Stable Diffusion (images)

---

Overall, the video is a comprehensive, in-depth tutorial and analysis of local AI model deployment, management, and training, emphasizing privacy, control, and practical business applications. It also serves as a guide for beginners and decision-makers interested in implementing AI solutions locally or within enterprises.