Skip to content
← Back to projects
Case study · Applied AI

Pennyworth

AI with limits: the app measures and diagnoses, the model only explains

A free Windows app that explains your PC in plain language. The assistant answers about the person's own machine without inventing numbers and without acting on its own, using a small local model or the person's own AI account.

Role
Author and maintainer
Type
Desktop app · Windows
Code
Public on GitHub
Languages
English and Portuguese (Brazil)

Context

A beginner wants to ask "why does my PC freeze when I open this game?" or "is 16 GB of RAM enough?". A generic chatbot answers without knowing the machine, and a small local model readily invents numbers.

The app was already measuring storage, memory, processor and hardware before any assistant existed. The engineering question was how to use those measurements without giving the model a chance to invent a figure or touch someone's PC.

Architecture

Pennyworth assistant architectureThe path of an answer: measurements become deterministic facts and diagnoses, the model only writes, and the answer goes through a filter and a closed list of actions. Dashed lines exist only when the person sets up an AI account.App measurementsstorage, RAM, CPU, GPUand hardwareFacts + rulesdeterministic diagnosiswith unit testsContextpersona, facts, actions1,500 token targetAnonymiserpreview of whatleaves the PCLocal modelONNX Runtime + DirectMLSHA-256 verifiedIChatClientMicrosoft.Extensions.AIYour AI accountOpenAI, Anthropic,Gemini, AzureAdvice filterchecks before displayAction linkerlisted actions onlyApp screenspreview + confirm
The path of an answer: measurements become deterministic facts and diagnoses, the model only writes, and the answer goes through a filter and a closed list of actions. Dashed lines exist only when the person sets up an AI account.

Main flows

  1. The app collects facts from what it has already measured, and hand-written rules turn them into diagnoses with a severity and one allowed action.
  2. The model receives persona, facts, diagnoses and the list of actions, and is told to answer only from those and to say when it can't tell.
  3. With an AI account, the person sees what will be sent, with their name and personal folders replaced by tags such as `<user>`.
  4. The answer goes through the filter, sentence by sentence. Action tokens that survive validation become buttons to screens that already have a preview and a confirmation.

Decisions and trade-offs

  1. Facts first, no tool calling

    Decision
    The app computes everything and the model only explains. Nothing about the PC is left for the model to figure out, and the same mechanism works the same in the cloud and on the local model.
    Rejected
    Tool calling and autonomous agents. Small local models are unreliable at it, and an agent deleting files on a beginner's PC is a risk with no upside.
    Accepted cost
    The assistant executes nothing. It only points to an existing screen, and the person decides from there.
  2. Evals pick the model

    Decision
    An evaluation tool (`tools/AssistantEval`) runs scenarios in English and Portuguese against any backend and checks for invented numbers, unlisted actions, dangerous advice, length, language and persona. The criteria, in order: no invented facts, Portuguese quality, RAM use, CPU speed.
    Rejected
    Choosing by a model's reputation. The evaluation work overturned the initial plan: Phi-3.5-mini was the provisional default but doesn't run on DirectML, so the final catalog uses Phi-4-mini on GPUs with 6 GB or more and Phi-3.5-mini on CPU.
    Accepted cost
    The checks are deliberately strict and crude, so reports still need a human reading them.
  3. A core that knows neither network nor runtime

    Decision
    `Pennyworth.Assistant` holds facts, rules, prompts, filter and anonymiser, and talks only to `IChatClient`. Providers and the local model live in separate projects, loaded only when used.
    Rejected
    Putting SDKs and the runtime in the same project, which would hurt testing and Native AOT and weigh on people who never turn the assistant on.
    Accepted cost
    More projects to maintain. In return, the logic is tested with a fake client, and with no assistant set up the app makes no network calls and uses no extra memory.
  4. A deterministic output filter

    Decision
    Every answer goes through the filter before it appears, sentence by sentence while streaming. Advice such as editing the registry or turning off Windows Defender gets replaced.
    Rejected
    Relying only on the prompt's instructions, which a small model doesn't follow consistently.
    Accepted cost
    The blocked-advice list is curated by hand, in two languages, and has to grow with what the evaluations find.
  5. Privacy without a server of its own

    Decision
    No hosted service: the person runs locally or brings their own key, kept in the Windows Credential Manager. Before sending, the app shows exactly what leaves the PC.
    Rejected
    A Pennyworth cloud service, which would add a recurring cost and make the project the custodian of other people's data.
    Accepted cost
    More friction in setup. The local model is only downloaded on request, and the app warns, without blocking, when the PC is unlikely to run it well.

Results

  • With no assistant set up, the app behaves as before: no network, no download and no extra memory.
  • In one run with Phi-4-mini on GPU: 0 errors in 64 answers, 0 language and persona failures, and a median of 3.3 s per answer in English.
  • In that same run, 2 of the 64 answers suggested something dangerous. That's exactly the case the output filter exists to replace.
  • The model is downloaded from the official repository, with files checked against SHA-256 hashes pinned in the app. If verification fails, it is deleted and never loaded.

Known limits

  • Small models still fail the strict checks, mostly on answer length (39 of 64 in the run above), and do better in English than in Portuguese.
  • On PCs without a suitable graphics card, the local model takes about half a minute per answer.
  • Windows only for now.

Stack

  • C# / .NET 10
  • Avalonia
  • Native AOT
  • ONNX Runtime
  • DirectML
  • Microsoft.Extensions.AI
  • xUnit