Summary of "Zui: Experimental projects with Generative AI"

This video presents an in-depth exploration of how generative AI technologies, particularly OpenAI’s GPT models, are being applied experimentally at Tāmaki Paenga Hira Auckland War Memorial Museum (TAPA) to enhance accessibility, description, and interaction with museum collections data.

Key Technological Concepts and Product Features

Custom GPTs and API Integration
- Introduction of customized GPTs (personalized versions of ChatGPT) that can be tailored via plain English instructions and by uploading files or linking to APIs.
- Use of TAPA’s public API (available since 2018) to access structured collections metadata under a Creative Commons BY 4.0 license.
- Development of an intermediary API and website (tonga.nz) to connect ChatGPT with TAPA’s collections API, enabling query prioritization and better formatting of data for AI consumption.
- The custom GPT acts as a friendly museum guide, transforming user queries into effective keywords to retrieve relevant collection data.
- Conversational AI model allows users to interact dynamically with collections data, including follow-up questions and multi-language support.
Generative AI for Automated Descriptions
- Pilot project focused on a collection of 1100 photographed silversmithing tools (19th-20th century) to generate detailed textual descriptions using OpenAI’s GPT-4 mini model (chosen for lower energy footprint).
- Initial AI output from images alone was basic and sometimes inaccurate (e.g., hallucinated object features or dimensions).
- Adding contextual metadata (title, classification, maker, dimensions, multiple images) significantly improved description quality.
- Use of two images per object helped AI understand 3D structure and reduce hallucinations.
- Careful crafting of system prompts and user prompts with explicit instructions (including many “do not” clauses) was essential to get accurate, concise, and relevant descriptions.
- AI-generated descriptions were saved both as JSON files (for programmatic analysis) and Word documents (for human verification).
Human Verification and Editing
- Despite AI’s capabilities, human review and editing remain necessary to correct hallucinations, remove speculative or verbose content, and ensure accuracy and trustworthiness.
- A secondary AI step was tested where the description was fed back to GPT as a “literary editor” to remove verbosity and speculation; however, this did not fully replace human intervention.
- Verification workflow involved either manual editing of Word documents or reviewing descriptions via PowerBI reports, though the latter did not allow direct editing.
- Average time for human fact-checking and editing was about 2 minutes per description.
Ethical, Legal, and Environmental Considerations
- Acknowledgement of AI’s environmental impact, with efforts to minimize energy use by choosing smaller models and tracking processing metrics.
- Discussion of copyright and ownership of AI-generated content:
  - OpenAI’s terms generally allow users to retain ownership of inputs and outputs, but legal status of AI-generated copyright is still uncertain under New Zealand law.
  - Caution advised when applying AI to collections with external copyright or cultural sensitivity.
- Importance of transparency and attribution to maintain audience trust, including tagging AI-generated content and providing concise platform attribution on collection websites.
- TAPA is developing an AI policy emphasizing human verification, trust, and transparency.
Future Directions and Potential Applications
- Expanding the model to query other trusted external sources (e.g., Wikipedia) to enrich responses.
- Creating standalone websites with custom GPTs to remove dependency on ChatGPT login or third-party platforms.
- Applying AI-generated descriptions to collections where images exist but rights are not yet cleared, potentially enabling early online access.
- Exploring other AI models, including open-source local models, especially for institutions lacking IT support.
- Considering the impact of AI on museum workforce roles, with a consensus that AI will augment rather than replace curatorial expertise.

Reviews, Guides, and Tutorials Provided

Step-by-step explanation of building a custom GPT connected to a museum API, including:
- Accessing and using TAPA’s API
- Creating an intermediary API and website (tonga.nz)
- Configuring custom GPT with plain English instructions and keyword extraction
- Handling data structuring for AI-friendly input
Pilot project workflow for AI-generated descriptions:
- Image and metadata input
- Prompt engineering (system and user prompts)
- Output formats (JSON and Word documents)
- Human verification and editing process
Q&A session addressing practical concerns such as accuracy, time