Detecting a persons mood via an image with the Microsoft Emotion API

Introduction

In this post we are going to continue looking at the Microsoft Cognitive Services. Much of the core code for this will be similar to the last blog post on OCR, but this time we are going to get emotion data from the faces of an image.

Sign up for cognitive services

Just like in the previous post on OCR, the Emotion API that we are going to use is part of the Microsoft Cognitive Services. If you have not already, you will need to sign up to use these API’s here. We will be using the Emotion API so sign up for this subscription. It is a preview so is free for 30000 transactions a month, there are some other limits which you can read up on, but for our test it should be more than adequate. We will be using the keys from this subscription later on.

Get Mood from Emotion API

Before we call the Emotion API, we need to setup a data structure for the returned JSON. The data structures below should be pretty self explanatory and will allow us to iterate though multiple faces later on.

public class ImageData
{
    public FaceRectangle faceRectangle { get; set; }
    public Scores scores { get; set; }
}

public class FaceRectangle
{
    public int left { get; set; }
    public int top { get; set; }
    public int width { get; set; }
    public int height { get; set; }
}

public class Scores
{
    public decimal anger { get; set; }
    public decimal contempt { get; set; }
    public decimal disgust { get; set; }
    public decimal fear { get; set; }
    public decimal happiness { get; set; }
    public decimal neutral { get; set; }
    public decimal sadness { get; set; }
    public decimal surprise { get; set; }
}

With our data structure in place we can now call the Emotion API from https://api.projectoxford.ai/emotion/v1.0/reconize. The rest of the code is the same as our previous OCR example, although this time we will deserialize to a list of ImageData objects.

public async Task<List> GetOCRData(string filename)
{
    var client = new HttpClient();
    client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", "5f6067eea83497fdfaa4ce");
    var uri = "https://api.projectoxford.ai/emotion/v1.0/recognize";
    HttpResponseMessage response;
    using (var content = new ByteArrayContent(GetBytesFromFilepath(filename)))
    {
        content.Headers.ContentType = 
            new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");
        response = await client.PostAsync(uri, content).ConfigureAwait(false);
    }
    var data = JsonConvert.DeserializeObject<List>(
        await response.Content.ReadAsStringAsync());

    return data;
}

We added a helper class to get a byte array from a selected filepath. Although we could have set MediaTypeHeaderValue to “application/json” and sent a uri rather than sending the file bytes.

private byte[] GetBytesFromFilepath(string filePath)
{
    Image img = Image.FromFile(filePath);
    using (var stream = new MemoryStream())
    {
        img.Save(stream, System.Drawing.Imaging.ImageFormat.Jpeg);
        return stream.ToArray();
    }
}

Sample App – The Mood Elevator

The data returned from the API is a series of decimal values where the indicating the detected emotion. The closest to 1 is the indicated emotion.

To display the result of the returned emotion, I have created a basic MVC application to allow a user to browse and upload a file. I have then created a MoodAnalysis class to determine the mood value based on a set a set of criteria that I have chosen. In my case, I have assigned different values for happy based on decimal values 0.9, 0.8 etc.

public async Task Index(HttpPostedFileBase file)
{
    string guid = Guid.NewGuid().ToString() + ".jpg";
    if (file != null && file.ContentLength > 0)
    {
        var fileName = Path.GetFileName(file.FileName);
        var path = Path.Combine(Server.MapPath("~/Content/upload"), guid);
        file.SaveAs(path);
        PhotoReader pr = new PhotoReader();
        var rtn = await pr.GetOCRData(path);
        MoodAnalysis ma = new MoodAnalysis();
        int elScore = ma.GetMood(rtn.FirstOrDefault().scores);
        ViewBag.EmotionScore = elScore;
        ViewBag.EmotionMessage = ma.ConvertStepToElevatorName(elScore);
    }
    
    return View();
}

Using this information I am then able to display a result on a chart, as shown in the example below:

AngryMoodElevator

 

Conclusion

In this brief introduction to the Emotion API we have uploaded a file and retrieved the emotion scores for analysis. I have tried not to overcomplicate it by not showing my break down of the score results, but with the returned decimal values you will be able to experiment based on your applications needs. If you have any questions or comments, please add them below.

Author: Dave Green

Software developer focusing on Microsoft Technologies, MCSD in Windows Store Apps, full time developer at a real job, part time writer and terrible golf player. Follow on twitter @IntelligentLabs

Leave a Reply

Your email address will not be published. Required fields are marked *