Developer Deskside | Building Apps on Kafka Streaming Data in Palantir Foundry

Summary of technological concepts & build steps

Goal / Use case

• Build an end-to-end factory machine monitoring solution using streaming data from Kafka in Palantir Foundry.
• Detect machine problems (e.g., temperature / production rate) and support collaboration via maintenance requests that can be created and resolved from both low-code and custom React apps.

---

1) Ingest streaming metrics from Kafka into Foundry

• Set up a Foundry data connection agent on a Linux VM to enable data ingestion.
• Create a Kafka Source in Foundry:
  • Uses the agent
  • Connects to Kafka (example uses localhost:9092)
  • Handles libraries via “Needs restart” → auto push libraries
• Create a Kafka topic and simulate events:
  • A Python Kafka producer generates JSON records with fields like:
    • machine ID
    • temperature
    • production rate
    • time
  • Sends metrics every ~5 seconds.

---

2) Stream processing with Pipeline Builder (“metrics cleaning”)

• Use a streaming sync to ingest Kafka records into Foundry.
• Observed issue: raw values arrive in a form described as binary/unusable.
• Pipeline Builder transforms the stream to a usable tabular schema:
  • Cast value to string
  • Parse JSON
  • Flatten nested structures
  • Normalize column names (convert to snake_case)
  • Cast time to timestamp
  • Drop unused fields like key/value columns
• Enrichment:
  • Create a Fusion table (“machines backing”) for metadata (machine ID → machine name, installed date, sector).
  • Join cleaned metrics with machine metadata in the pipeline.
  • Create a human-readable metric title (concatenates machine name + formatted timestamp).
• Output:
  • Write results back to a streaming dataset such as machine_metrics_clean.

---

3) Construct ontology (models) from datasets

Ontology Manager creates and links object types so the data works across Foundry.

Object types created

• Machine object type
  • Backed by the Fusion “machines” dataset
  • Primary key: machine_id
• Machine metric object type
  • Backed by the pipeline output dataset (e.g., machine_metrics_clean)
  • (Primary key mentioned as “title” in the demo; noted they could concatenate machine ID + timestamp for stability)
• Maintenance request object type
  • Backed by a Fusion-created maintenance requests dataset
  • Primary key: maintenance_request_id

Relationships / links

• Metric → Machine via machine_id
• Maintenance request → Machine via machine_id
• These links enable navigation and UI filtering in Workshop.

---

4) Low-code app in Foundry Workshop (monitoring + actions)

A Workshop module is built to:

• Browse machines
• Plot metrics over time
• Raise maintenance tickets
• Resolve tickets with comments

Workshop UI components

• Left: Object list of machines (sorted by installed timestamp)
• Right: Active machine drives:
  • Metric cards (machine properties like name/sector)
  • XY line chart:
    • X-axis bucketed by minutes
    • Temperature plotted as averaged series
    • Production rate plotted as another series
    • Hard-bound y-axis ranges (demo sets temperature between 0–100)

Maintenance request creation

• Adds Foundry Actions:
  • Requires first creating a right-back dataset for maintenance request updates (naming convention: dataset name + _edited)
  • Defines an action type like “create maintenance request”
    • Inputs: machine (object reference dropdown), title, request comment, status default open
    • UI includes text area input for comments
• In Workshop:
  • A button “raise maintenance request” calls the action
  • Submitting creates a maintenance request object that appears in the app after reload.

Maintenance request management view

• Adds Workshop tabs/pages:
  • Machines page
  • Maintenance requests page
• Shows requests using:
  • Object list
  • A filter list for status (demo filters e.g. “exclude Open tickets”)

Maintenance request resolution

• Adds another action like “resolve maintenance request”
  • Updates status → closed
  • Writes resolution comment
• In Workshop:
  • Adds “resolve request” button
  • Resolution comment is entered and submitted
  • Demo confirms updates propagate back into Workshop objects.

Planned/mentioned extensions

• More ticket states (e.g., investigation/in-progress/assigned)
• Better commenting flow (multiple comments objects)
• Automated alerts with Foundry rules
• Potential ML-based detection using Foundry GML

---

5) Custom React app using Foundry APIs (read + write)

Instead of using Workshop, the demo builds a separate web app that syncs with Foundry.

Backend: Express proxy server

• Creates an Express server using JavaScript
• Uses axios for calling Foundry APIs
• Adds endpoints:
  1. GET /requests
     • Uses OAuth client credentials flow to get an access token
     • Calls Foundry API to list maintenance request objects
  2. POST /resolve
     • Uses OAuth token
     • Calls Foundry API to apply the resolve action with parameters:
       • maintenance_request_id
       • resolution_comment

OAuth setup

• Registers a Foundry OAuth application for the server.
• Uses client credentials grant (service token).
• Grants permission for calling the specific ontology action:
  • Updates action submission criteria from “me” to the created service app identity (e.g., maintenance app).

Frontend: React + axios

• Creates a React app (TypeScript template).
• Fetches data from the Express server:
  • Calls GET /requests
• Displays maintenance requests in a list/table.
• Adds a “mark as resolved” UI:
  • Input for resolution comment
  • Calls POST /resolve with request ID + comment
• Addresses CORS issues:
  • Adds cors middleware in the Express server.
• Demo confirms:
  • React writes resolution comments back to Foundry
  • Workshop and API reflect updated status/comments after refresh.

---

Main speakers / sources

• Speaker: “Developer Deskside” (unnamed presenter; multiple “we” and “I’m going to” references)
• Primary source: Palantir Foundry documentation, plus Kafka (Apache) quick start (referenced during setup).