Building my own personal assistant

•matt

Since December of 2024 I've been dabbling in various AI tools. I started with pushing the boundaries of agentic coding with Cursor and Claude, from there I dabbled in MCP servers and MCP workflow engines, and most recently at work I've been building a multi-agent chatbot with MCP servers. Likewise, at home I've been building a personal assistant that's catered to me - unlike the chatbot that needs to deal with multitenancy and data access my personal assistant affords me some amount of simplication in terms of the implementation complexity.

I. Assistants: Agents of agents

I was pretty awestruck when I first used Claude Desktop and ChatGPT as personal assistants. I'm also somewhat endlessly fascinated with the concept of a personal assistant; it's one of those luxuries that most people could never justify paying for but that everyone would want or need. As impressed as I was I still wanted more out of a personal assistant than Claude Desktop and ChatGPT were able to provide so I set out to build my own.

I was a little skeptical of what I was building because it occurred to me that ChatGPT and Claude Desktop had the advantage of being first to market - why would they stop building on their personal assistant capabilities? Along the way I realized some challenges.

  1. Data residency, accuracy, and retrieval speed are all somewhat intersecting and opposing problems.
    • Vector databases (and RAG) are really useful tools when combined with text embedding and semantic search but they require pipelines that run in order to "fill the lake" with useful data so to speak.
    • MCP tools are great for retrieving live data but they often require an agent to have a context that's configured to use them well.
    • MCP servers are, in simple terms, probabilistic workflow engines. In order to function well they need rich prompts, resources, sampling, and contextualization.
  2. Data access in the era of AI is basically OAuth2 which, while secure, adds an order of complexity for every integration you add.
  3. Typically one agent simply is not enough.
    • An agent is effectively a model + tools + context + systems.
    • Models have limited context windows and large context windows are not befitting of every problem, which a personal assistant really need to be good at guaging work.
    • You end up in this situation where you have agents for specific capabilities and even agents managing those other agents.
    • Routing and task composition become really important to be good at.

In order to deal with these challenges I started thinking about a system with the following properties:

  1. Local and remote tool execution
    • Sometimes you probably do want your assistant doing a thing on your computer - or maybe the data is really sensitive and you don't want that stored on a remote computer.
    • Remote tool execution is really useful for leveraging the economies of scale for cost and performance.
  2. Agent routing and task execution through a workflow engine
    • Finite state machines are great for more deterministic tasks (and task management in general)
    • Directed acyclic graphs are great for more complex tasks that have dependencies.
  3. Data residency is really a trifecta
    • Running a database on the users machine is not great. It gets in the way of multi-platform use and a users machine is not dependable resource wise.
    • I changed the meaning of "local" to mean "local network" and "remote" to mean "not on the local network". This does open up the expectation that users would be able to setup and run some amount of software that the personal assistant can use.
    • Remote databases and tools are reserved for when the economies of scale for cost, performance, and privacy all align.

II. From concept to implementation

The design I ended up going with was a reflection of all of these properties.

The "assistant" is largely a collection of agents and systems that all message pass and complement each other. In the current iteration of my system many tools and systems run in-process - these systems are easily moved into "local network" tools and systems. "Remote" systems are run in Docker containers and configured into the graphical user interface of the assistant.

The assistant's graphical user interface

I still find myself decomposing settings and pushing the boundaries of how my personal assistant can be used and configured. The system I've crafted strives to maximize adaptability and extensibility.

III. My first successful exchange

Introspection is the ability of a language to evaluate its own code at runtime. These operations are often slow, somewhat fraught, but also very powerful. A lot of languages use these kinds of features under the hood to bridge a gap when more formalized, compile time features don't exist to support a given use case.

Naturally, I made introspection the first big challenge I wanted my assistant to tackle. Basically, look at what I've built so far and be a product owner: tell me what casual and business users would want out of my assistant. I like this problem because it challenges the model to use reasoning and tool use in various iterations in order to both understand the problem, the tasks involved, and to produce a proper response.

I'll let you look at it's raw feedback.

Great! I've analyzed the codebase and I'm excited to discuss opportunities for improvement. Let me share what I found and ideas for making Orbit even better for everyday users and business professionals.

Current Capabilities (What's Already Built)

Based on the codebase analysis, Orbit already has a solid foundation:

Core Features:

What Everyday People Really Need

🎯 High-Priority Additions:

  1. Voice Interaction
  1. Calendar & Scheduling Integration
  1. Email Assistant
  1. Smart Reminders & Tasks
  1. Quick Access Anywhere
  1. Personal Knowledge Base
  1. File & Document Help

What Business Professionals Need

💼 Business-Critical Features:

  1. Meeting Assistant
  1. Document Generation
  1. Data Analysis & Visualization
  1. CRM & Sales Tools Integration
  1. Project Management Integration
  1. Team Collaboration
  1. Compliance & Security
  1. Business Intelligence
  1. Workflow Automation
  1. Multi-language Support

Quick Wins (Easiest to Implement)

Based on your existing architecture, these would be relatively straightforward:

  1. System clipboard integration - Read/write to clipboard for easy pasting
  2. Export conversations - Already have SQLite, add PDF/Word export
  3. Conversation templates - Saved prompts for common tasks
  4. Browser bookmarklet - Send web pages to Orbit for analysis
  5. Keyboard shortcuts - Power user efficiency boost
  6. Tags and labels - Better conversation organization
  7. Search filters - Date ranges, topics, models used
  8. Cost tracking - Show API usage and costs per conversation

Architecture Suggestions

Given your current Wails + Go + SQLite + Weaviate setup:

  1. Plugin system - Let users/developers add integrations
  2. Webhook support - Receive data from external services
  3. API endpoints - Allow other apps to integrate with Orbit
  4. Background service - Run tasks when app is closed
  5. Cloud sync option - Optional backup/sync (encrypted)

My Recommendations (Prioritized)

Phase 1 - Make it indispensable:

  1. Quick access hotkey
  2. Smart clipboard integration
  3. Conversation templates
  4. Better export options

Phase 2 - Extend reach: 5. Calendar integration 6. Email drafting assistant 7. File/document analysis 8. Voice input

Phase 3 - Business value: 9. Meeting assistant features 10. Data analysis tools 11. Team collaboration 12. Integrations marketplace

Competitive Differentiators

What would make Orbit special:

Local-first + Privacy (you already have this!)

The key insight: People don't want another chatbot. They want an AI that integrates into their existing workflow and makes their daily tasks effortless. The less they have to think about "using the AI" and the more it just helps naturally, the better.

What resonates most with you? Which direction should we explore first?"

I was impressed by a few things I didn't expect in this response:

The systems, as they're designed, are all very loosely coupled and probabilistic. I can arrange and configure them in such a way that I think they'll have a high probability of being successful but it's really amazing when they are - especially the first time around.

IV. What's next

I'm going to continue productionalizing my assistant. I really like the idea of being able to run and host most of your own systems while having the option to use more centrally run systems. I think that's where the value in this product will really be; can the user bring their own models and can the user provide their own storage and tool execution systems as long as they adhere to a spec that the assistant can understand and use?

Back to top