Speech-to-Action OS Assistant (LangGraph + AI Safety)
FeaturedTransform your voice and text commands into safe, confirmed browser actions. Experience intelligent orchestration, real-time streaming, and comprehensive audit trails—all in one beautiful interface
Overview
This project is a Speech-to-Action OS Assistant that transforms voice or text commands into safe, confirmed browser and system actions using a LangGraph-orchestrated agent workflow.
Rather than acting immediately on user input, the system emphasizes planning, safety, confirmation, and observability, making it suitable for real-world automation scenarios.
Domain: Agentic AI / AI Automation / AI Safety
Primary Users: Power users, developers, productivity-focused users
Core AI Capabilities: Intent classification, plan generation, tool execution, voice interaction
This project demonstrates how to build action-capable AI systems without sacrificing control or trust.
What it does
Transform voice and text commands into safe, confirmed browser actions using LangGraph orchestration with intent classification, plan generation, explicit confirmation, tool execution, and comprehensive audit trails, all with real-time streaming, voice commands, text commands, intent detection, plan generation, confirmation flow, tool execution, analytics, workflows, export/share, enhanced history, command suggestions, and command templates
Best for
Users who need to execute browser actions efficiently by speaking or typing commands naturally, understanding user intent accurately, generating detailed action plans with safety checks, confirming actions before execution, executing safe browser actions with proper validation, tracking all actions with comprehensive audit trails, and coordinating command processing workflows
Output
Full-stack voice and text command system with Next.js 16 (React 19.2, TypeScript, Tailwind CSS, shadcn/ui, Server-Sent Events SSE) frontend, FastAPI backend (Python 3.11+, LangGraph multi-agent AI orchestration), OpenAI GPT-4o-mini, OpenAI Whisper, Supabase PostgreSQL, Upstash Redis, and deployment on Vercel (frontend) and Railway (backend)
Problem
Most AI assistants that execute actions suffer from serious limitations:
- •
They execute commands too eagerly
- •
Users lack visibility into what will happen
- •
There is little or no audit trail
- •
Tool execution can be unsafe or opaque
- •
Voice interfaces often feel unreliable or brittle
The challenge was to design an assistant that:
- •
Understands natural language commands
- •
Plans actions explicitly before execution
- •
Requires user confirmation
- •
Enforces tool safety and allowlisting
- •
Provides full observability and auditability
- •
Feels fast, intuitive, and usable across devices
Solution
I designed a state-driven, LangGraph-orchestrated agent system that converts natural language into structured, safe action workflows:
- •
Voice or Text Input — Browser-based audio capture or typed commands
- •
Intent Classification — AI identifies command type and confidence
- •
Plan Generation — Step-by-step execution plan with rationale
- •
Safety Checks — Tool allowlisting, validation, SSRF protection
- •
Explicit Confirmation — User reviews and approves before execution
- •
Tool Execution — Client-side and server-side tools run safely
- •
Audit & Persistence — Full execution trail stored and queryable
This architecture ensures control, transparency, and trust at every step.
My Role
End-to-end AI Engineer (agent orchestration, backend, frontend, safety design, deployment)
Technical Architecture
AI & Orchestration
- •
LangGraph for state-based, multi-step agent orchestration
- •
OpenAI GPT-4.1-mini for:
- •
Intent classification
- •
Plan generation
- •
Response synthesis
- •
Whisper API for speech-to-text transcription
- •
Deterministic state transitions:
- •classify → plan → safety → finalize → execute
- •
Extensible graph design for adding new capabilities
Backend
- •
FastAPI (async Python) for orchestration and tool execution
- •
Pydantic v2 for strict schema validation
- •
Redis (Upstash) for:
- •
Job queuing
- •
Rate limiting
- •
URL caching
- •
Supabase PostgreSQL with:
- •
Custom
jarvisschema - •
Secure RPC functions
- •
Full audit trails for actions, plans, and tool calls
Frontend
- •
Next.js 16 + React 19
- •
Real-time streaming via Server-Sent Events (SSE)
- •
Execution flow visualization for LangGraph states
- •
Voice recording with live guidance and status indicators
- •
Workflow execution modals with formatted results
- •
Mobile-first, touch-optimized UX with dark/light mode
Deployment
- •
Vercel for frontend hosting
- •
Railway for backend API
- •
Cloud-native, production-ready setup
Impact
This project demonstrates how AI assistants can safely move from "chat" to "action":
- •
Enables natural language automation without blind execution
- •
Makes AI decision-making transparent and reviewable
- •
Provides enterprise-grade audit trails for every action
- •
Supports both voice and text interaction modes
- •
Establishes a reusable pattern for safe AI automation systems
This project reflects real-world automation and safety requirements.
Challenges & Key Decisions
- •
Safety first: Preventing unsafe execution via allowlists and confirmation gates
- •
UX trust: Making plans understandable before approval
- •
Voice reliability: Balancing transcription speed and accuracy
- •
State management: Designing a clear, extensible LangGraph workflow
- •
Observability: Ensuring every step is logged, searchable, and auditable
These challenges closely mirror real problems in AI automation and agent safety design.
What This Project Demonstrates
- •
Agentic AI orchestration with LangGraph
- •
Voice-enabled AI systems
- •
Safe tool execution and confirmation flows
- •
Event-driven, real-time streaming UX
- •
Full-stack AI product delivery
- •
AI safety, transparency, and observability by design