CognitiveJ – Image Analysis for Java

CognitiveJ is an open source Java library that makes it easy to detect, interpret and identify faces or features contained within raw images.

Screen Shot 2016-05-11 at 14.03.25.png

Person Identification


Powered by Project Oxford, The library can suggest a persons age, gender and emotional state. Based on machine learning, the library can also attempt to interpret and describe what is contained within an image.

Its being released for public preview under the Apache 2 licence and at the time of writing, the features include;

Faces

Screen Shot 2016-05-11 at 11.54.02.png

Facial Detection with Age and Gender

  • Facial Detection – Capture faces, gender, age, associated facial features and landmarks from an image
  • Emotion Detection – Derive emotional state from faces within an image
  • Verification – Verify, with a confidence scale, on whether two different faces are of the same person
  • Identification – Identify a person from a set of known people
  • Find Similar – Detect, group and rank similar faces
  • Grouping – Group people based on facial characteristics
  • Person Group/Person/Face Lists; Create, manage and train groups, face lists and persons to interact with the
    identification/grouping/find similar face features

Vision

  • Image Describe – Describe visual content of an image and return real world caption to what the image contains
  • Image Analysis – Extract key details from an image and if the image is of an adult/racy nature
  • OCR – Detect and extract a text stream from an image
  • Thumbnail – Create thumbnail images based on key points of interest from an image

Overlay

  • Apply layers onto images to visually represent found features
  • Apply captions onto faces and images
  • Graphically illustrate the Faces/Vision feature sets
  • Pixelate faces within an image

 Other Features

  • Supports local and remote images
  • Validation of parameters
  • Image Grids
Screen Shot 2016-05-11 at 12.04.29.png

An Image Described

With the source code hosted on github, please log any issues/suggestions into the Issue Tracker.

Finally, before we dig into the detail of the new library, I would like to pay a massive thanks to the guys and girls at Microsoft who have delivered a truly amazing suite of tools to the community.

Getting Started 

Pre-requisites

  • Java 8
  • Subscription keys for the MS Cognitive libraries (free registration here)
  • The CognitiveJ library that is hosted on jcenter- see Github for maven/gradle dependency info

Structure

The CognitiveJ API is split into a number of sections and grouped by distinct concepts;

Chained Builders – The builders are simple lightweight wrappers over the MS Cognitive REST calls that manages the marshalling of parameters/responses, the HTTP communications and retry strategies. The builders are chained to allow for follow up manipulation on resources that have been created or retrieved & where applicable.

Scenarios – Scenarios are real world use case classes that greatly simplifies the interaction between the builders and the wrapper classes.

While there is no reason you can’t interact directly with the builders, scenarios have much of the boilerplate logic
in place to reduce burden.

Overlay – Allows for creating and writing new images based on the results from the queries

Wrappers

Simple domain wrappers around request/response/parameter objects (e.g. Face, FaceAttributes,Person etc)

So, lets get started (all examples below are in the blog section of the test code)

Faces API

 Face – Detect can detect faces from within an image and return the results as a collection of ‘Face’ results.

 Use Case: From an image URL, detect and mark faces within an image.

Screen Shot 2016-05-11 at 12.22.30.png

Face Detection With Multiple Faces

public static void main(String[] args){
    FaceScenarios faceScenarios = new FaceScenarios(
    getProperty("azure.cognitive.subscriptionKey"),
    getProperty("azure.cognitive.emotion.subscriptionKey"));
    ImageOverlayBuilder imageOverlayBuilder = ImageOverlayBuilder.builder(IMAGE_URL);
    imageOverlayBuilder.outlineFacesOnImage(faceScenarios.findFaces(IMAGE_URL),
    RectangleType.FULL, CognitiveJColourPalette.STRAWBERRY).launchViewer();
}

Face – Landmarks can detect faces from within an image and apply facial landmarks

 Use Case: From an image URL, detect and apply face landmarks

 

Screen Shot 2016-05-12 at 15.18.59.png

Face – with landmarks applied

public static void main(String[] args) throws IOException {
    FaceScenarios faceScenarios = new FaceScenarios(getProperty("azure.cognitive.subscriptionKey"),
    getProperty("azure.cognitive.emotion.subscriptionKey"));
    Face faces = faceScenarios.findSingleFace(IMAGE_URL);
    ImageOverlayBuilder.builder(IMAGE_URL).outFaceLandmarksOnImage(faces).
    launchViewer();
}

 

Face – Detect with Attributes displays associated attributes for detected faces

 Use Case: From an image URL, detect and mark faces along with gender, age and glasses within an
image.

Screen Shot 2016-05-11 at 12.56.15.png

Face – with detected attributes

 

public static void main(String[] args){
    FaceScenarios faceScenarios = new FaceScenarios(
    getProperty("azure.cognitive.subscriptionKey"),
    getProperty("azure.cognitive.emotion.subscriptionKey"));
    List<Face> faces = faceScenarios.findFaces(IMAGE_URL);
    ImageOverlayBuilder.builder(IMAGE_URL).outlineFacesOnImage(faces,RectangleType.CORNERED,
    CognitiveJColourPalette.MEADOW).
    writeFaceAttributes(faces,CognitiveJColourPalette.MEADOW).launchViewer();
}
    

Face – Verify will validate (with a confidence ratio) if 2 different faces are of the same persons.

Use Case: From 2 image URLs, each containing a single person; Verify if the 2 people are the same person

Screen Shot 2016-05-11 at 13.04.38.png

Validated match with confidence rating

Screen Shot 2016-05-11 at 13.06.31.png

Validated non-match with confidence rating

 

    public static void main(String[] args){
        FaceScenarios faceScenarios = new FaceScenarios(
        getProperty("azure.cognitive.subscriptionKey"),
        getProperty("azure.cognitive.emotion.subscriptionKey"));
        ImageOverlayBuilder.builder(CANDIDATE_1);
        imageOverlayBuilder.verify(CANDIDATE_2, faceScenarios
        .verifyFaces(CANDIDATE_1, CANDIDATE_2)).launchViewer();
    }

Face – Identify will identify a person (or people) within an image. Before the library can
identify, we need to provide the library with the sample set of candidates.

Use Case: Identify a group of people from a trained person group

Screen Shot 2016-05-11 at 14.00.55.png

All Persons Recognised

Screen Shot 2016-05-11 at 14.03.25.png

With Tommy removed from candidate dataset

 

    FaceScenarios faceScenarios = new FaceScenarios(getProperty("azure.cognitive.subscriptionKey"),
    getProperty("azure.cognitive.emotion.subscriptionKey"));
    ImageOverlayBuilder imageOverlayBuilder = ImageOverlayBuilder.builder(IMAGE);
    List<ImageHolder> candidates = candidates();
        People people = ScenarioHelper.
        createPeopleFromHoldingImages(candidates, ImageNamingStrategy.DEFAULT);
        String groupId = faceScenarios.
        createGroupWithPeople(randomAlphabetic(6).toLowerCase(), people);

        //grid drawing logic (see sourcecode)
        

Face – Pixelate will identify all faces within an image and pixelate them.

Use Case: Pixelate all faces in an image

Screen Shot 2016-05-11 at 19.23.23.png

        FaceScenarios faceScenarios = new FaceScenarios(getProperty("azure.cognitive.subscriptionKey"),
        getProperty("azure.cognitive.emotion.subscriptionKey"));
        ImageOverlayBuilder imageOverlayBuilder = ImageOverlayBuilder.builder(IMAGE);
        faceScenarios.findFaces(IMAGE).stream()
        .forEach(imageOverlayBuilder::pixelateFaceOnImage);
        imageOverlayBuilder.launchViewer();
        

Emotion – Detect will detect what emotion a face(s) is showing within an image.

Use Case: display all emotions shown on a face

Screen Shot 2016-05-11 at 14.26.42.png

Facial Emotion Detection (All Emotions Displayed)

        FaceScenarios faceScenarios = new FaceScenarios(getProperty("azure.cognitive.subscriptionKey"),
        getProperty("azure.cognitive.emotion.subscriptionKey"));
        ImageOverlayBuilder.builder(IMAGE_URL)
        .outlineEmotionsOnImage(faceScenarios
        .findEmotionFaces(IMAGE_URL)).launchViewer();
        
Screen Shot 2016-05-11 at 14.36.14.png

Dominant Emotion for each found face

        FaceScenarios faceScenarios = new FaceScenarios(getProperty("azure.cognitive.subscriptionKey"),
        getProperty("azure.cognitive.emotion.subscriptionKey"));
        ImageOverlayBuilder.builder(IMAGE_URL)
        .outlineDominantEmotionsOnImage(faceScenarios
        .findEmotionFaces(IMAGE_URL)).launchViewer();
        

 

Vision – Describe will analyse and describe the contents of an image in a human readable
caption.

Use Case: From an image, describe the contents and output the value as a human readable caption.

Screen Shot 2016-05-11 at 17.12.49.png

        ComputerVisionScenario computerVisionScenario =
        new ComputerVisionScenario(getProperty("azure.cognitive.vision.subscriptionKey"));
        ImageDescription imageDescription = computerVisionScenario.describeImage(IMAGE_URL);
        ImageOverlayBuilder.builder(IMAGE_URL).describeImage(imageDescription).launchViewer();

        

Vision – OCR will analyse and extract text from within an image into a computer understandable
stream.

Use Case: From an image, extract and overlay OCR words.

Screen Shot 2016-05-12 at 11.41.25.png

Found text within image

        ComputerVisionScenario computerVisionScenario =
        new ComputerVisionScenario(getProperty("azure.cognitive.vision.subscriptionKey"));
        OCRResult ocrResult = computerVisionScenario.ocrImage(IMAGE_URL);
        ImageOverlayBuilder.builder(IMAGE_URL).ocrImage(ocrResult).launchViewer();
        

One thought on “CognitiveJ – Image Analysis for Java

  1. Pingback: CognitiveJ Release: Image Analysis for Java - Artificial Intelligence Online

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s