In the previous post, we showed how to train an image classification model using the Microsoft Custom Vision service as well as to perform a quick test on a new image. However, what if you want to integrate this model into one of your applications that is using C#? Whether it’s an app that runs on the desktop, mobile, or is a web API the Custom Vision SDKs are the best way to your Custom Vision model.

In this post, we’ll go over how to set up your project and use the Prediction SDK to perform predictions on your model within C#. We’ll look at how to predict both on image URLs and with image files you may have locally or that is user submitted to your application.

The complete code for this post is on GitHub.

Project Setup

Get Training and Prediction Keys
Before we can do anything with the SDKs we need to get the keys that we will use for the code to be able to access the custom vision service so we can make our predictions. The keys can be easily found in the settings page (the gear icon at the top right) of the custom vision service project that you want to use.

Install Packages
Once you have a new project, whether it’s a console, web, or web API app, you will need to bring in a few NuGet packages:

  • Microsoft.Cognitive.CustomVision.Training
  • Microsoft.Cognitive.CustomVision.Prediction
  • Microsoft.Rest.ClientRuntime

The training and prediction SDKs are separate and even though we’re only doing predictions we would still need the training SDK to get a reference to the project we want to be able to use that model. Also, we would need the ClientRuntime package to allow the use of the SDK.

Get Project Reference
With all the need packages installed we can start to use them. It was mentioned before that we need the training SDK to help us get a reference to the project we want to use for our predictions. To do that we need to instantiate an instance of the TrainingApi class and pass in the appropriate key. The GetKeys method just reads from a local JSON file that has the keys.

var keys = GetApiKeys();

var trainingApi = new TrainingApi { ApiKey = keys.TrainingKey };

With the trainingApi instantiated, we can use that to get all of the projects.

var projects = trainingApi.GetProjects();

And since that returns a collection, we can use some LINQ magic to select the one we want by the project name.

var herbProject = projects.FirstOrDefault(p => p.Name == "Herbs");

Now that we have a reference to our project, we will need it when we call the predict SDK methods.

Predicting on Images

To start predicting on new images, we first need to instantiate an instance of the PredictionEndpoint class and pass in the prediction key.

var predictionEndpoint = new PredictionEndpoint { ApiKey = keys.PredictionKey };

Predict on Image URL
In order to predict on an image URL, just use the PredictImageUrl method from the prediction endpoint class. This takes in a couple of parameters. First, it needs the project ID, which we can access from the herbProject variable we got earlier. And it also needs an ImageUrl instance. This instance can take in the actual URL of the image you want to predict on as a parameter to the constructor.

var result = predictionEndpoint.PredictImageUrl(herbProject.Id, new Microsoft.Cognitive.CustomVision.Prediction.Models.ImageUrl(imageUrl));

And that’s all we need to make a prediction. That’s very easy to use! We don’t need to worry about making HTTP calls and setting appropriate headers. The SDK does all of that for us.

For the results, we get back a collection of predictions. We can use that to tell us what tag name and probability the custom vision service predicted for us.

foreach (var prediction in result.Predictions)
{
    Console.WriteLine($"Tag: {prediction.Tag} - Probability: {String.Format("Value: {0:P2}.", prediction.Probability)}");
}

Let us run this with a test image such as this one of basil I randomly picked from Google Images and see what the custom vision service returns back to us.

It looks like it accurately predicted basil with a probability of 99.4%. Not bad at all with just a few images for training.

Predict on Image File
While predicting with image URLs is nice, a more likely situation would be to predict with images stored locally on the file system. This will be almost the same as predicting with an image URL, but we have to read the image before we can pass it on to the prediction SDK. To do that, we’ll just use the File.OpenRead method and pass in the path to the image.

var imageFile = File.OpenRead(imagePath);

From there, we can just call the PredictImage method on the prediction class and pass in the project ID as well as the result from the previous call to OpenRead. The OpenRead method actually returns back a FileStream object, but the PredictImage method takes in a Stream object. We can do this since FileStream inherits from Stream.

var result = predictionEndpoint.PredictImage(herbProject.Id, imageFile);

With that, we can read the results like we did with the image URL. The below image of cilantro is stored on the local machine where the code is run, so we’ll use that do a test run.

And let’s see what the results of that are.

It did correctly predict that it was cilantro, however, the probability is only at 4.3%. To help get this higher we would need to give the model more images similar to this one to train on.

In this post, we went over how to use the training and prediction Custom Vision SDKs to predict on an image URL and an image file. In our next post, we’ll look at how to use these same SDKs to add training images and to train a new iteration of our custom vision model.