The OrCam MyEye increases the independence of people who are blind and visually impaired. It can read texts, barcodes, recognize faces, identify products, money notes, colors, and can even tell the user the time and date. It does this by conveying visual information audibly.

How AI and machine learning are changing prosthetics

Imagine a prosthetic arm with the sensory capabilities of a human arm, or a robotic ankle that mimics the healthy ankle's response to changing activity.

Hollywood has long popularized imaginative versions of such ideas. While human engineering may not yet be able to produce superhero-enabling devices, prosthetics are getting "smarter" and more adaptive, approaching a reality in which amputees' artificial appendages offer near-normal function.

Bioengineers are increasingly looking to create "human-machine interfaces embodied by a prosthetic limb that really feel like an extension of the body," said Robert Armiger, project manager for amputee research at Johns Hopkins University's Applied Physics Lab, where scientists have developed an arm with human-like reflexes and sensation.

FDA looked to spur development when it released a 'leapfrog' draft guidance in February outlining a vision for invention and testing of brain-implanted devices capable of controlling prosthetic limbs. And the Defense Advanced Research Projects Agency is funneling tens of millions into new products. 

The nonprofit Amputee Coalition estimates about 2 million amputees in the U.S., and that number is expected to nearly double to 3.6 million by 2050. An estimated 185,000 new lower-limb amputations occur each year. 

Among those living with limb loss, below-knee amputations are the most prevalent, with nearly three-fourths related to circulatory problems. In fact, vascular disease, including diabetes and peripheral artery disease, accounts for 54% of all amputations in the U.S. Other major causes are trauma (45%) and cancer (less than 2%). 

With the numbers of amputations rising, activity is the space has been heating up.

Driving demand are aging populations and rising incidence of vascular diseases, as well as developments in artificial intelligence and machine learning. Advanced materials such as silicone and urethanes are also resulting in lighter-weight prosthetics with "memory" to respond to changes in pressure. 

Among those competing in the space are Ottobock, which sells the bebionic hand, and ReWalk, maker of a powered walking assistance system. Icelandic firm Össur markets a mind-bionic prosthetic for lower-limb amputees.  

But despite their potential, smart prosthetics are still a ways from becoming a reality for most people who need them, in large part due to the relatively small population, high costs and lack of reimbursement.

Building a robotic system that incorporates all the movements and sensory components of the missing limb, and doing so in a natural and intuitive way, is also challenging. 

Another slowdown is getting technology transitioned out of the lab and into a company that will pursue FDA approval and take it to market.

"The risks are in working with insurance companies to reimburse for these types of devices that are already very expensive and have a high degree of abandonment," Armiger told MedTech Dive. "Insurance companies say, 'we're spending this much money on this device and people don't wear it, and now you're asking us to spend more money on a more advanced technology.'"

To expand access to these devices, companies will need to demonstrate that value proposition, he adds. 

Functioning on rough terrain

For people with lower-limb prostheses, adaptability is key not only to a more fluid, natural experience but also to safety. Amputees fall at about 200 times the rate of healthy individuals and seek medical attention on a par with the institutionalized elderly, according to Michael Goldfarb, co-director of the Center for Rehabilitation Engineering and Assistive Technology at Vanderbilt University. Risk of falling increases in more active younger amputees who are less daunted by challenging terrain.

While microprocessors in prosthetic knees have been around for some time, allowing them to change behavior based on activity, or the idea of adaptive ankles, is relatively new. 

Goldfarb's lab has developed an ankle that uses sensors and a microprocessor to mimic physiologic function as people move along flat or uneven terrain and go up and down stairs and slope.

"It's kind of a mini version of the human neuromuscular system with its own sensors, its own brain, its own muscles and the ability to change what it does," Goldfarb told MedTech Dive.

The ankle's design also increases the amount of clearance when people swing their prosthetic limb, providing a margin of error slightly better than that of a healthy individual to reduce trips and falls, Goldfarb said, adding that response during tests has proven promising.

Franklin, Tennessee-based Synchro Motion LLC has received funding from the National Science Foundation and NIH's Small Business Innovation Research grants to bring the product to market. 

Despite its high-tech capabilities, the ankle is regulated as a Class I device, requiring only a predicate for FDA clearance. Goldfarb expects it to be commercially available within two years.

Thought-controlled arms

The robotic arm being developed at Johns Hopkins APL has 26 joints and 'load cells' in each fingertip to detect force and the torque applied to each knuckle. Sensors give feedback on temperature and vibration and collect the data to mimic what the human arm is able to detect. It responds to thought much like a normal arm.

"Our bodies natively have control loops at various levels, and that's really where AI comes into play, because there are things that you can build into that arm like reflexes and anticipating an object you might be interacting with and allowing the user to provide the highest level of command," Armiger explained.

The device attaches to the user via an osseointegration implant, a small post that goes into the bone and protrudes through the skin. That orientation allows the device to clamp directly onto the skeletal system, giving the user a more natural understanding of how the limb is moving. By attaching to the bone, the perception of weight is also lower than traditional external attachments.

Users also undergo what's called targeted muscle reintervention, which rewires the nerves in the upper arm to twitch when the user thinks of moving his or her hand. Separate sensors attach to the skin over top of the muscles and correspond with the intended movement.

"It's a great combination of the engineering of the robotic arm, the cutting-edge surgical procedures and these sort of AI algorithms which basically, when those muscles twitch, recognize and can translate that into I want to wiggle my pinkie finger," Armiger said. 

The opportunity lies in maintaining an object in the hand, such as grabbing a coffee cup and holding it steady so the contents don't spill — those types of activities that are sort of ambiguously controlled even within the human body, Armiger added.

The arm is currently in the advanced prototype research stage, and researchers are looking for potential transition partners. Armiger said his team has also been talking with FDA about what type of approval it would need. 

Still, one of the biggest hurdles will be cost.

"Right now what we're trying to do is demonstrate the improvement in control through research studies that would then make a case for a market transition," Armiger said.

Happy National Assistive Technology Awareness Day!

Earlier this month, the United States Senate unanimously adopted a resolution declaring today National Assistive Technology Awareness Day. The resolution noted that "assistive technology devices and services are not luxury items but necessities for millions of people with disabilities and older adults, without which they would be unable to live in their communities, access education, and obtain, retain, and advance gainful, competitive integrated employment,"

ACL is proud to fund Assistive Technology Act programs in every state and territory that help older adults and people with disabilities discover, try, reutilize, and finance assistive technology. In FY18:

• Over 72,000 individuals participated in assistive technology device demonstrations to find the right device for their community living needs.
• Nearly 50,000 AT devices were provided on short-term loan to individuals with disabilities, service providers, and agencies.
• More than 70,000 AT devices were reutilized, allowing consumers to save more than $28 million by obtaining a lightly used or refurbished AT device.
• 96% percent of consumers who received financing loans from an AT program said they would not be able to purchase or obtain the AT without this financing.

Toronto’s Pearson Airport now supports assistive app for people with cognitive special needs

Toronto’s Pearson International Airport has announced a partnership with MagnusCards, a free-to-download app that offers digital how-to guides (known as ‘Card Decks’) for people with autism and other cognitive special needs. Pearson says it’s the first airport in the world to work with MagnusCards to offer the assistive app. Developed by the Waterloo-based MagnusMode, the MagnusCards app offers assistive Card Decks for a variety of venues and brands, including CIBC banks and Maple Leaf Sports and Entertainment. Pearson visitors can download up to ten different Card Decks, each of which offering a personal step-by-step guide on how to navigate different parts of Canada’s largest airport. The Pearson Card Decks include tips on how to check-in to an airline, getting help or asking questions, going through US Customers and Border Protection and more. Card Decks are also offered in English and French and feature pictures, text and audio to help as many different users as possible. 

Read more at MobileSyrup.com: Toronto’s Pearson Airport now supports assistive app for people with cognitive special needs

Apple and Cochlear team up to roll out the first implant made for the iPhone

Apple has teamed up with Australian-based Cochlear to bring iPhone users the first made for iPhone Cochlear implant.

Approved by the U.S. Food and Drug Administration in June, Cochlear’s Nucleus 7 Sound Processor can now stream sound directly from a compatible iPhone, iPad or iPod touch to a patient’s surgically embedded sound processor.

The device also allows those with the implant to control and customize the sound from their iPhone.

There have been other implants and hearing aids that have used iOS apps to control sound and other features and Nucleus’s own app can be downloaded to do the same. However, Cochlear’s newest processor is controlled by the phone itself and does not require an app download.

More than 50 million Americans have experienced some sort of hearing loss due to one reason or another. Apple saw the hearing loss problem and has spent a number of years developing a hearing aid program within the company.

Apple soon developed a protocol the company offered for free for hearing aid and implant manufacturers to use with their devices.

“We wanted to see something that could become ubiquitous out in the world,” Apple’s Sarah Herrlinger, senior manager for global accessibility policy and initiatives told TechCrunch. “We want everybody to use our technology and to say ‘wow my iPhone is the best piece of technology I’ve ever used before’…with every iteration of our operating system our goal is to add in new accessibility features in order to expand the support that we can give to people all over the world.”

Accessing the control settings for your Cochlear implant is relatively easy. Those who get the new Nucleus 7 Sound Processor or other made for iPhone hearing aid simply go to their iPhone settings, click on “General” and then click “Accessibility.” As you move down the screen you’ll see a list of different devices you’ll see “hearing devices.” Tap on that and then the device should show up the way a Bluetooth device would in Bluetooth settings. The implant will then pair with your iPhone.

Just like headphones or another Bluetooth-enabled device, as soon as the implant is paired up it can be controlled using the iPhone’s volume controls. So, for example, when a phone call comes in, you can hear that call at the volume settings within your implant.

The new Nucleus 7 comes with a longer battery life and is also smaller and 24 percent lighter than its predecessor, the Nucleus 6 Sound Processor, making it ideal for small children with hearing loss as well.

“The approval of the Nucleus 7 Sound Processor is a turning point for people with hearing loss, opening the door for them to make phone calls, listen to music in high-quality stereo sound, watch videos and have FaceTime calls streamed directly to their Cochlear implant,” Cochlear CEO Chris Smith said in a statement. “This new sound processor builds on our long-standing commitment to help more people with hearing loss connect with others and live a full life.”

Ava gives the deaf and hard-of-hearing a more present voice in group conversations

For those with hearing issues, simple dinner table group conversations can be pretty painful to stay on top of.

Ava is aiming to bring deaf and hard-of-hearing people back into group conversations with their threaded speech-to-text application that gives people with hearing issues an easy way to stay on top of a conversation.

The app, formerly known as Transcence, starts with each participant in a conversation downloading the app and setting up a profile. This may seem like a bit of work to get a casual chat going but especially for families or groups of friends, it’s a really simple way to bring everyone into a conversation regardless of their specific hearing abilities.

After getting everyone onboard, people just talk normally near their phones microphone and the speech to text translation is organized into a threaded message for everyone in the group text allowing users who are deaf or hard-of-hearing to have a record of the conversation right in front of them to quickly respond to.

The company announced this past week that they’ve closed $1.8M in funding to grow their team and accelerate product development.

I sat down with Ava CEO Thibault Duchemin to chat about tech meeting the needs of the deaf community and what’s next for his app.

The potential for the app expands well beyond dinner table conversations, and fits into pretty much anything where multiple people are chatting in an environment where deaf people are present. The Ava team had a major opportunity to show off their tech last month when Salesforce used the tech to live transcribe the majority of its breakout sessions for audience members at its Dreamforce conference. Duchemin and COO Pieter Doevendans also chatted a lot about the potential for education markets in giving everybody in the classroom an equal voice.

For the most part, speech recognition tech has been slow to wide user adoption because it’s not perfect. Even when systems can handle 90 or 95 percent accuracy, it’s hard to focus on anything other than what the system got wrong. It makes sense because voice assistants are so fragile and even a single misheard word can throw off the usefulness of an answer.

For now, Ava is just as accurate as the 3rd-party solutions that are currently powering it, though Duchemin says that once its system gathers more data on a user’s voice it will grow more capable in distinguishing it from the background noise. But for deaf users that often can only achieve 20 percent accuracy in reading lips during a conversation—making up the rest through body language and context—the ability to gain 80 or 90 percent accuracy through using Ava is quite empowering.

I reached out to TC contributor and accessibility writer Steven Aquino to hear if this is something the deaf community would actually use. “I grew up in the deaf community, as both of my parents were deaf, so I have that connection between both the deaf and hearing worlds. ASL is my first language,” Aquino said. “I know from experience how difficult is it to have deaf or hard-of-hearing folks involved in conversation. I was always acting as an interpreter when my parents were around hearing family or friends.”

Right now, the app allows users to host up to 5 hours of conversation per month. Any user can host a group conversation and participating in another user’s session doesn’t eat up any of your alotted time. For Ava users that move past this 5 hour time frame, they can upgrade to a paid unlimited version of the app for $30 per month.

Duchemin detailed that one of the most critical things it getting people on the app and through the setup process as quickly as possible so that groups can get to chatting. The interface of the app is appropriately very simple with just a few controls while much of the screen real estate is devoted to what’s being said. Users can sign into the account with different options, either as someone who’s hard-of-hearing, deaf or hearing, with the app slightly altering the experience to best serve the user’s abilities. The app is available on both Android and iOS.

Duchemin emphasized that the app definitely isn’t perfect and that there are still a lot of improvements the team is hoping to roll out soon. To them the key was really getting the app out in time for Thanksgiving so that family conversations around the dinner table could be a bit easier and accessible for the 15 million people in the US with hearing issues.

This blind Apple engineer is transforming the tech world at only 22

Apple engineer Jordyn Castor has never been one for limitations.

She was born 15 weeks early, weighing just under two pounds. Her grandfather could hold her in the palm of his hand, and could even slide his wedding ring along her arm and over her shoulder. Doctors said she had a slim chance of survival.

It was Castor's first brush with limited expectations — and also the first time she shattered them.

Castor, now 22, has been blind since birth, a result of her early delivery. But throughout childhood, her parents encouraged her to defy expectations of people with disabilities, motivating her to be adventurous, hands-on and insatiably curious.

It was that spirit that led to her interact with technology, whether it was the desktop computer her family bought when she was in second grade, or the classroom computer teachers encouraged her to use in school.

She says the adults in her life would often hand her a gadget, telling her to figure it out and show them how to use it. And she would.

"I realized then I could code on the computer to have it fulfill the tasks I wanted it to," says Castor, whose current work focuses on enhancing features like VoiceOver for blind Apple users. "I came to realize that with my knowledge of computers and technology, I could help change the world for people with disabilities.

"I could help make technology more accessible for blind users."

Bringing a personal perspective to Apple innovation

There's an often overlooked component of "diversity" in workplace initiatives — the need to include the perspectives of people with disabilities.

Keeping tabs on the needs of the blind and low-vision community is a key component of Apple's innovation in accessibility. Castor is proof of how much that can strengthen a company.

She was a college student at Michigan State University when she was first introduced to Apple at a Minneapolis job fair in 2015. Castor went to the gathering of employers, already knowing the tech giant would be there — and she was nervous.

"You aren't going to know unless you try," she thought. "You aren't going to know unless you talk to them ... so go."

Castor told Apple reps how amazed she was by the iPad she received as a gift for her 17th birthday just a few years earlier. It raised her passion for tech to another level — mainly due to the iPad's immediate accessibility.

"Everything just worked and was accessible just right out of the box," Castor tells Mashable. "That was something I had never experienced before."

Sarah Herrlinger, senior manager for global accessibility policy and initiatives at Apple, says a notable part of the company's steps toward accessibility is its dedication to making inclusivity features standard, not specialized. This allows those features to be dually accessible — both for getting the tech to more users, as well as keeping down costs.

"[These features] show up on your device, regardless of if you are someone who needs them," Herrlinger tells Mashable. "By being built-in, they are also free. Historically, for the blind and visually impaired community, there are additional things you have to buy or things that you have to do to be able to use technology."

At that job fair in 2015, Castor's passion for accessibility and Apple was evident. She was soon hired as an intern focusing on VoiceOver accessibility.

As her internship came to a close, Castor's skills as an engineer and advocate for tech accessibility were too commanding to let go. She was hired full-time as an engineer on the accessibility design and quality team — a group of people Castor describes as "passionate" and "dedicated."

"I'm directly impacting the lives of the blind community," she says of her work. "It's incredible."

Innovation with blind users in mind

Increased accessibility for all users is one of Apple's driving values, under the mantra "inclusion inspires innovation."

Herrlinger says the company loves what it makes, and wants what it makes to be available to everyone. She describes the need to continuously innovate with accessibility in mind as part of Apple's DNA.

"Accessibility is something that is never-ending," Herrlinger says. "It isn't something where you just do it once, check that box and then move on to do other things."

And it's a dedication that isn't going unnoticed by the blind community. On July 4, Apple was the recipient of the American Council of the Blind's Robert S. Bray Award for the company's strides in accessibility and continued dedication to inclusion-based innovation for blind users.

The company, for example, made the first touchscreen device accessible to the blind via VoiceOver. Recent announcements of Siri 

coming to Macthis fall, and of newer innovations, like a magnifying glass feature for low-vision users, have continued the promise of improving the Apple experience for those who are blind and low vision.

"The fact that we take the time to innovate in these ways is something new and different," Herrlinger says. "It was not the expected thing in the tech community."

Often, the success of such innovations depends on the input of the community — and employees like Castor provide irreplaceable first-hand insight into the tech experience for blind individuals.

The most recent example of community-driven innovation can be found on the Apple Watch. During a meeting, Herrlinger explains, a person who sees could easily peer down at their watch to keep an eye on the clock. A person who is blind, however, hasn't had a way to tell time without VoiceOver.

After confronting the conundrum, Apple solved the issue by making a feature that tells time through vibrations. The addition, Herrlinger says, is coming to watchOS 3 this fall.

High-tech meets low-tech

Castor says her own success — and her career — hinges on two things: technology and Braille. That may sound strange to many people, even to some who are blind and visually impaired. Braille and new tech are often depicted as at odds with one another, with Braille literacy rates decreasing as the presence of tech increases.

But many activists argue that Braille literacy is the key to employment and stable livelihood for blind individuals. With more than 70% of blind people lacking employment, the majority of those who are employed — an estimated 80% — have something in common: They read Braille.

For Castor, Braille is crucial to her innovative work at Apple — and she insists tech is complementary to Braille, not a replacement.

"I use a Braille display every time I write a piece of code," she says. "Braille allows me to know what the code feels like."

In coding, she uses a combination of Nemeth Braille — or "math Braille" — and Alphabetic Braille. Castor even says that with the heavy presence of tech in her life, she still prefers to read meeting agendas in Braille.

"I can see grammar. I can see punctuation. I can see how things are spelled and how things are written out," she says.

The technologies that Apple creates support her love of Braille, too — there are various modifications, like Braille displays that can to plug into devices, to help her code and communicate. But Castor also often forgoes Braille displays, solely using VoiceOver to navigate her devices and read screens.

That autonomy of choice in accessibility, Apple says, is intentional. T

he company believes that the ability to choose — to have several tools at a user's disposal, whenever they want them — is key to its accessibility values.

Giving back to the community

Last week, Castor attended a conference hosted by the National Federation of the Blind, where she gave a speech telling her story. She says the impact that Apple has had on the blind community was extremely clear as soon as she stepped into the conference hall — just by listening to what was going on around her.

"When I walk through the convention, I hear VoiceOver everywhere," she says. "Being able to give back through something that so many people use is amazing."

Castor was recently able to use her presence and perspective at Apple to give back to a part of the community she's especially passionate about — the next generation of engineers.

She was a driving force behind accessibility on Apple's soon-to-be released Swift Playgrounds, an intro-to-coding program geared toward children. She's been working to make the program accessible to blind children, who have been waiting a long time for the tool, she says.

"I would constantly get Facebook messages from so many parents of blind children, saying, 'My child wants to code so badly. Do you know of a way that they can do that?'" Castor says. "Now, when it's released, I can say, 'Absolutely, absolutely they can start coding.'"

Castor says working on Swift Playgrounds has been an empowering experience, and her team has deeply valued her perspective on the VoiceOver experience for blind users.

She says the task-based, interactive app would have made a massive impact on her as a child. The program is, after all, a guided way of taking tech and figuring out what makes it tick — a virtual version of the hands-on curiosity adults instilled in her as a child.

"It will allow children to dive into code," she says of the program. "They can use Swift Playgrounds right away out of the box; no modifications. Just turn on VoiceOver and be able to start coding."

As someone who was always encouraged to challenge expectations, Castor says she has one simple message for the next generation of blind coders, like the children who will sit down with Swift Playgrounds in the fall.

"Blindness does not define you," she says. "It's part of who you are as a person, as a characteristic — but it does not define you or what you can do in life."