Azure: Grids, Hubs, Busses, Topics, and Queues… Oh My!

Messaging between application components and devices is why engineers invented computer networks. The ultimate computer network is the Internet, which has practically connected all the world’s devices to a single network where any one device can send a message to another device. For this reason, there are tons of solutions that provide messaging infrastructure to…

Azure Databricks Series - Part 6
Batch and Streaming Pipelines

One of the best features of Databricks is that it facilitates transitioning batch pipelines to streaming pipelines. Depending on your transformation logic it may be as easy as changing two lines of code. In this post we will review some batch and streaming concepts and show how to write a notebook that can be run…

ML & AI for Software Developers - Part 28
Object Detection

My previous post introduced two popular algorithms for detecting faces in photographs: Viola-Jones, which relies on machine learning, and MTCNNs, which rely on deep learning. Face detection is a special case of object detection, in which computers detect and identify objects in images. Identifying an object is an image-classification problem, something at which CNNs excel.…

ML & AI for Software Developers - Part 27
Face Detection

My previous post demonstrated how to use transfer learning with a CNN trained on millions of facial images to build a facial-recognition model that is remarkably adept at identifying faces. The images used to train the model were carefully cropped so that faces were consistently sized and positioned in the frame. Suppose you wanted to…

ML & AI for Software Developers - Part 25
Audio Classification with CNNs

You are the leader of a group of climate scientists concerned about the planet’s dwindling rainforests. The world loses up to 10 million acres of old-growth rainforests each year, much of it due to illegal logging. Your team plans to convert thousands of discarded smart phones into solar-powered listening devices and position them throughout the…

ML & AI for Software Developers - Part 24
Data Augmentation

My previous post demonstrated how to use transfer learning to build a model that with just 300 training images can classify photos of three different types of Arctic wildlife with 95% accuracy. One of the benefits of transfer learning is that it can do a lot with relatively few images. This feature, however, can also…

Azure Databricks Series - Part 5
Databricks Jobs

Today’s post will cover Databricks Jobs. There is also the concept of a Spark Job which will be covered briefly to try to avoid confusion. Spark Job When running a Spark application there is the concept of a Spark job. At runtime, the Spark driver converts your Spark application into a job that is transformed…

ML & AI for Software Developers - Part 23
Transfer Learning

My post introducing convolutional neural networks (CNNs) used a dataset with photos of Arctic foxes, polar bears, and walruses to train a CNN to recognize Artic wildlife. Trained with 300 images – 100 for each of the three classes – the CNN achieved an accuracy around 60%. That’s not sufficient for most purposes. Imagine you’re…

ML & AI for Software Developers - Part 22
Pretrained CNNs

Given a set of images with a relatively high degree of separation between classes, it is perfectly feasible to train a CNN to classify those images on a typical laptop or PC. A great example is the famous MNIST dataset, which contains 60,000 training images of scanned, handwritten digits, each measuring 28 x 28 pixels,…

Azure Databricks Series - Part 4
Databricks Clusters

This is a continuation of my series of posts on Databricks where we most recently reviewed the Workspace & Notebooks. Now let’s get more familiar with the concept of clusters. Clusters Databricks breaks clusters into multiple categories: All-Purpose Clusters Job Clusters Pools Spark clusters consist of a single driver node and multiple worker nodes. The…

ML & AI for Software Developers - Part 21
Convolutional Neural Networks

Computer vision is a branch of AI in which computers extract perceptual information from images. Real-world uses include identifying objects in photos, removing inappropriate images from social-media sites, flagging defective parts coming off an assembly line, and facial recognition. Computer-vision models can even be combined with natural language processing (NLP) models to caption photos. I…