🤖 Local AI Power: Vision and Function Calling with Microsoft Agent Framework and Ollama

⚠️ This blog post was created with the help of AI tools. Yes, I used a bit of magic from language models to organize my thoughts and automate the boring parts, but the geeky fun and the 🤖 in C# are 100% mine.

Hola friends!

One of the questions I get most often lately is: “Bruno, can I run a full-featured agent locally without sending everything to the cloud?”

The answer is a resounding YES. 🚀

Today, I want to show you how to use the new Microsoft Agent Framework to build a local agent that doesn’t just chat, but also calls functions and analyzes images (Vision!). All powered by Ollama and the Ministral-3 model.

You can start with the video, or the full description below:

🛠️ The Scenario

We are going to build a console application that:

Connects to a local instance of Ollama.
Uses the ministral-3 model (which is amazing for local tasks).
Implements a C# function to “get the weather” (simulated).
Processes a local image file to describe its content.

📝 The Code

The beauty of the Microsoft Agent Framework is how it integrates with Microsoft.Extensions.AI. Check out how clean this code is:

using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
using OllamaSharp;
using System.ComponentModel;

[Description("Get the weather for a given location.")]
static string GetWeather([Description("The location to get the weather for.")] string location)
    => $"The weather in {location} is cloudy with a high of 15°C.";

// 1. Setup the Local Client with Ollama
IChatClient chatClient = 
    new OllamaApiClient(new Uri("http://localhost:11434/"), "ministral-3");

// 2. Build the Agent with Function Calling support
var ollamaAgent = chatClient.AsBuilder()
    .UseFunctionInvocation()
    .Build()
    .CreateAIAgent(
        name: "OllamaAgent",
        instructions: "You are a useful agent",
        tools: [AIFunctionFactory.Create(GetWeather)]);

// 3. Ask a standard question
Console.WriteLine(await ollamaAgent.RunAsync("What is the capital of France?"));

// 4. Trigger Function Calling (Local!)
Console.WriteLine(await ollamaAgent.RunAsync("What is the weather like in Amsterdam?"));

// 5. Vision Analysis: Send an image as a Byte Array
var imageByteArray = File.ReadAllBytes("image01.jpg");
ChatMessage message = new(ChatRole.User, [
    new TextContent("What do you see in this image?"),
    new DataContent(imageByteArray, "image/png")
]);

Console.WriteLine(await ollamaAgent.RunAsync(message));

This is the image that I used for the sample:

And this is the complete response from the console app:

The capital of France is **Paris**.

The weather in **Amsterdam** is currently **cloudy**, with a high temperature of **-5°C**. It might be quite chilly, so bundle up! Let me know if you'd like updates or forecasts for another location.

This image depicts a charming and stylized scene featuring a raccoon. Here are the details:

1. **Raccoon**: The central figure is a raccoon sitting upright on its hind legs. It has a friendly expression and is adorned with a necklace featuring a red maple leaf pendant, which is symbolic of Canada.

2. **Background**: The raccoon is situated in a picturesque natural setting. Behind it, there is a serene lake with calm waters reflecting the surrounding scenery.

3. **Forest**: The lake is surrounded by a dense forest of tall evergreen trees, creating a lush green backdrop.

4. **Mountains**: In the distance, there are snow-capped mountains, adding to the scenic beauty of the landscape.

5. **Foreground**: The foreground includes a dirt path with some rocks and wildflowers, adding to the natural and tranquil ambiance of the scene.

Overall, the image combines elements of nature with a cute and anthropomorphic raccoon, creating a peaceful and inviting atmosphere.

💡 Why this matters

Running agents locally with the Microsoft Agent Framework gives you:

Privacy: Your images and data never leave your machine.
Latency: No round-trips to the cloud.
Cost: It’s essentially free (once you have the hardware!).

If you want to see this in action, I’ve recorded an 8-minute video walking through the entire setup. You can find the links to the resources and the video below!

📚 Resources

Happy coding!

Greetings

El Bruno