Microsoft Seeing AI: A Quick First Look at this Groundbreaking iOS

Shop all Bathroom Signs

Microsoft Seeing AI

by Bill Holton

Source article: AccessWorld Magazine

Scenes: This is a beta feature, destined to improve with time. Snap a photo of your environment and Seeing AI will attempt to describe it. “A street corner with signs,” for instance, or “Two dogs and one person on a lawn.”

By now, you have likely heard about the new Seeing AI app from Microsoft. It’s free, and this groundbreaking app combines the functions of nearly every image and product recognition app that uses your iPhone or iPad’s camera and more.

In future issues of AccessWorld, we plan to take a much deeper dive into this new offering, which Microsoft has rightly dubbed a potential “Swiss Army Knife of tools for visually impaired people.” We’ll describe where Seeing AI came from, and, more importantly, where it’s heading. In the meantime, just in case you have not heard about Seeing AI, here’s what it can do:

Read Text: Seeing AI includes two text OCR modes, Short Text and Document. The Document mode works like many text recognition apps, such as KNFB-Reader. In Document mode simply aim your camera toward a document page and follow the prompts which will alert you as to which edges and corners are not yet visible. Seeing AI will instruct you to “Hold steady,” and then automatically snap a photo and send it to Microsoft servers for recognition. Note: You can also take a picture manually by pressing the “Take Picture” button. The app does an excellent job of maintaining formatting, so you can use VoiceOver’s Heading, list items, and other quick navigation gestures. The recognition is slower than KNFB-Reader because the OCR engine is not onboard your device, but it seems to be just as accurate.

The Short Text mode turns on your camera and searches for printed material continuously. When it finds printed text, it translates it into machine text, and then speaks it aloud. The Short Text mode is not as thorough and accurate as Document mode, but it makes going through recipe cards and the mail a breeze! It can also be quite useful when searching for a conference room number, or when your computer stops talking, as it will identify text on PC monitors. It’s a must-have if you have a messy desk and want to get organized. Using Seeing AI it’s possible to go through a stack of familiar but disorganized papers in only seconds per page. Because Seeing AI is continuously scanning for printed text, the app often interrupts itself with new text. One way to prevent this is to move the device up and away once it begins speaking, so it will not spot new text and move on.

Scan Bar Codes: Seeing AI does perhaps the best job of accessible mobile device bar code scanning around. It can be problematic, locating the bar code on a can or box, but Seeing AI beeps when it spots one, and as you move the camera closer the beeps grow faster until the app snaps the code and identifies that can of corn or box of cake mix. There is also a “More information” button, if the barcode database includes it, with product details such as nutrition labeling, calorie counts and even cooking instructions. It can take a few seconds to locate the bar code and snap an image. It’s by no means as swift and easy to use as a standalone barcode reader, such as the I.D. Mate Galaxy, which AccessWorld reviewed in the August 2016 issue of AccessWorld. But, it does get the job done a lot more accessibly than other mobile apps.

Recognize Faces: You can create a photo library of friends and family, and the app will tell you when someone it recognizes is near. Snap a picture of an individual, and it will estimate the age–sorry, ladies–and offer an emotion tag, such as smiling or frowning. The app will tell you when a face is in focus and offer up the person’s name if they are in your app image database. This might be handy at a conference or party setting to help you find friends or colleagues, but I would advise using a phone case with a lanyard strap and Bluetooth earbuds so you’re not annoying the crowd by pointing your phone hither and yon. Also, check the lighting setting in the app’s Settings menu. Otherwise that phone dangling from your neck may appear as a headlight.

Scenes: This is a beta feature, destined to improve with time. Snap a photo of your environment and Seeing AI will attempt to describe it. “A street corner with signs,” for instance, or “Two dogs and one person on a lawn.”

One extremely useful feature found in Seeing AI is its appearance on the Sharesheet. Tap any image on your camera roll, an image attached to an email or text, or even one of those annoying images that appear on Twitter and other social media sites with no description tag. The “Recognize with Seeing AI” Sharesheet option will send the image directly to the app immediately, and recognize both any text it finds, and offer an image description. If the option to “Recognize with Seeing AI” does not appear, select the “More” option in the Sharesheet and switch the option to “On”.

The app is only available for iOS, and it can be downloaded from the iOS App Store. Microsoft has produced an excellent YouTube introduction demonstrating this powerful new app’s features. Text and video help is also available in the app’s Quick Help tab.