Smart Connected Sensors | From KNOW-HOW to WOW Podcast

So, first things first. For our listeners that can't see us, I'm going to strap on some sensors here. - Geoff and I we were at CES, at the Consumer Electronics Show in Las Vegas last month where we recorded this. - Oh, and they're marked as well, of course they are. Alright, arm left, yes. Right arm, right. - I'm glad he doesn't have the same problem as I do with left and right because that would have been an issue if I were the one putting on the sensors. But Melena, seeing as you were the

re, what exactly is he doing? - So, he's trying one of the demos on the Bosch Booth. - Okay, so now ankle L. Clip in, ankle R. - It sounds like he's doing some kind of yoga stretch or something. Maybe Geoff sounds a little bit rusty... but okay so anyway. - Wait until the demo actually starts. It's a little game where dear Geoff head to dance. - I definitely can't wait to hear it. This is Shuko and Melena in our home studios. Today we're exploring smart connected sensors, a technology platform

that was presented at CES. [Music] From Know-How to Wow! The Bosch global podcast. Melena, so great to have you back! So, maybe in  a few sentences, how was Las Vegas, how was CES? - Oh yeah, wow that was quite a blast! I have to say, I mean being in Las Vegas for business purposes that's in itself already kind of wild.  I mean, getting out of that Convention Center, late in the afternoon, in the evening and just being surrounded by all these flashing lights, the insane Las Vegas architectur

e, that's really really cool. But I do have to say, it was also a little bit exhausting. - I can really imagine. I've seen also a few clips that came over through social media, you guys were sending me also some pictures and videos, so it did seem that you had enough to do. - Yeah! But it was fun! I can't really complain, really. Because it was even more exhausting for someone like our Bosch colleague Abhi. - So, this was definitely a lot of fun. It was also a bit tiring for all of us because

we were constantly moving. So, yeah, was a fun event. - Abhi is the product manager for our smart connected sensors and he's the guy who ran who dance demo that Geoff tried at CES. So Abhi was very, very busy, he had to dance a lot as well. - The demo was actually based on full body motion tracking, with gesture recognition, and also giving you an active feedback while you're doing those motions. And of course, we wanted to add some perfectly cool element to it, and that's why we dedicated th

is demo for dancing. So we were doing a lot of dancing. And knowing how bad or how good we were dancing. - So without any bias Melena who was a better dancer, Abhi or Geoff? I guess these sensors could give us an objective answer based on accurate measurements, right? - Was that a trick question? Yes, they could. But first, after strapping them onto your arms, legs and ankles they need a quick calibration. Very easy to do for anyone who wanted to try them at the Bosch Booth at CES. - So, what

I need you to do is, a T pose now So the T pose works like this: you follow the robo on the right. - So, what Geoff sees is a little animated robot on the screen of a tablet and he has to imitate its movements. - Okay, so yeah I am seeing this right now.  He's spreading his arms and standing still. - Stay stable. Don't move for three seconds. And let go, let go. And you're calibrated. - Very cool!  - Yeah. Move your arm, the arm is a bit, laggy. You need to come a bit ahead. It'll react. And

now try to move your leg. - Oh, wow. So, for our dear listeners, so I started in a T pose and my little robot friend here on the tablet was also in a T pose and that, that did the calibration. That's cool. And now as  I'm just kind of shuffling my legs around here, you know, everybody knows my phenomenal dancing skills. You can see it directly on the robot. It's very cool. It's almost like there's a camera on me. But there's certainly not. Yeah, excellent point. The camera on the tablet is als

o covered just for the record.  But yeah, it's super responsive. That's neat. - So now these sensors on his limbs capture any movement that Geoff does. Lifting his arms and legs, twisting, turning, doing a little shimmy. - You got it! Exactly they capture much more than just the immediate movements of the limbs they're attached to. - And it's so, sensitive that even as I'm just holding, my hand out and rotating the palm of my hand up and down, my robot buddy is also doing the same thing. It'

s quite remarkable! - So that's pretty nice that movements get detected accurately but where it gets really interesting is when you get feedback on your movements. That's what the dance demo was actually about. So the robot on the screen wasn't mimicking your movements you were trying to mimic its movements. And to clarify by you I mean Geoff. - Cuz I've been looking at this the whole time, there's a "start game button" in the corner. I've been kind of nervous. - You wanna try? - Yeah definit

ely! Of course. - Let's go. Is everything connected still? - Everything's connected. I'm ready! - Go! Go slow. - Slow. Oh, and now there's colors on my robot buddy, telling me... I assume green is good. Telling me which sensors are getting close to the one I'm trying to mimic. - Yeah. And now you get the score. So you're five. It tells you how good you did. And you can also see each individual node was given a proper metering. So, when you were moving your right leg you probably didn't move it

in the best possible way. And also the thigh. - Perfect. Very cool! - I like how Abhi tells Geoff, he didn't move his thigh properly and then Geoff goes "perfect, haha". - But come on, in Geoff's defense the system perfectly showed all of his flaws in  the movement and that's kind of what it's about, right? - Yeah and ideally you would then of course try to get better at it and actually achieve perfection. Here's Abhi's very diplomatic summary after he did this exercise with countless people at

the CES booth. [Music starts] - Let's say there was some bad movements, there was some good movements. And people would then know, what's the average score for the number of repetitions done. You would also get a highest score of the best repetition you had. Individual limbs where the sensor was connected on, also displayed sort of a level and a score, which was, which was given to them. So they would exactly know which movement they did properly and also realize which limbs were not moving

or which part of the bodies were not moving properly. - Got it. But you know if we're looking beyond dancing, there must be a host of applications for sensors like this, right? - Yeah, totally! The dancing was just a fun little demo. [Music stops] But you could do so much more with these smart connected  sensors. If we move more towards virtual environments [Music starts] we can naturally move through them [FX] using a very easy and also lightweight setup. A system like this could be used in

a gaming setup. [FX] Imagine using your full body to control a character on the screen discovering new worlds. Suddenly monsters appear in front of you, [FX] you jump to the side getting your sword out, [FX] trying to slash the ugly beasts . [FX] [Music stops] - So, virtual reality and augmented reality are going to be huge application domains for these motion tracking sensors. And I definitely can't wait to get rid of bulky controllers and replace them by something much more comfortable. No

t just for VR, also gaming, as you mentioned. - Also games in the real world, like good old Sports. [FX] - You can also imagine this for football shoes, you can imagine for all kind of sports applications, you can imagine this put on a baseball glove as well as a baseball bat and then see how good the swing is. [FX] - A completely different application domain for  these sensors is Warehouses. [FX] People doing lifting movements can use them to make sure that  they're executing the movements co

rrectly. And in a more abstract way these sensors know  where they are in relation to each other. So you could also use them to track objects  or people and their relation to each other. - It's crazy what kind of applications we  have for for things. Extremely interesting! - So all of these use cases suddenly become possible, which is a big boost to the new applications to come. - Back to tracking body movements and giving feedback. This might remind you of an episode of From Know-How to Wow t

hat is very far back in our podcast feed. I looked it up it was almost two years ago exactly that we featured a fitness tracking censor that could analyze your movements and provide feedback as well. At that time it was my turn to try it, with Geoff watching. [FX] - So what that means is that the same exercise will look somewhat different from the tracker's point of view. - Oh yeah, and we experienced that when I first tried the tracker. [Music starts] What should I do? What should we both

do? - Jumping Jacks. - Jumping Jacks. Okay, we try Jumping Jacks then. - Yeah put your phone down, weirdo. - No exercise detected yet. I did it wrong. - I've never seen someone do a jumping jack wrong before. [Music stops] - Melena, I think I need to apologize. That wasn't very nice, I'm sorry. That was a little mean. - It's okay. [FX] - "It's okay." It's not okay! Oh God, turns out I'm less talented at doing Jumping Jacks then Geoff is at dancing.  Listeners if you want the whole story, go ba

ck and listen to that episode it's pretty fun. But the technology has evolved and what Abhi presented at CES is the latest iteration of it. - Correct me if I'm wrong Melena, but two years ago you had only one sensor strap to your wrist? - Yeah, that's right. - Now Abhi's system uses multiple sensors. And I kind of lost count but how many did Geoff strap on? - You're absolutely right. That was one of the major innovations here. This is not just one sensor, it's a network of up to eight sensors

. All working together. [Music starts] - When it's a network of connected sensors they deliver a much more precise input. They give you a much more detailed understanding of how things are connected together, how two movements are connected to each other. - Which makes absolutely sense. The movements  are connected and the sensors are connected. - Right. Shall we dive a little bit into that?  First how are these connections organized? Who's talking to whom? Using what protocol? - Choosing the

right network topology, choosing the right way of communication be it Bluetooth or WiFi or any impossible wireless connectivity. All of these were really challenging decisions. - Okay, but if we stay there, with regards to network topology, how did they figure out which one was the best to use? - Abhi would you please take us to the stars? - We are right now gone with the star network topology. So, star network topology allows us to have one central node and up to seven leaf nodes. This is, al

lowing us also to be much more flexible. That means we can choose from going from one central node to one leaf node or to two leaf nodes, or three leaf nodes, and up to seven leaf nodes. - But here's the thing: you can't see which one is the central node. They're all the same. Of the sensors that Geoff was wearing, maybe the one on his right ankle was the central node, maybe the one in his left wrist. They all have the same sensing and compute capabilities. The central node just happens to

collect and process all of the data and organize the network. - That sounds like that makes things really versatile for the network. So, depending on which device these sensors end up in, perhaps one earbud, could become the  central node or in a different setup, a SmartWatch with one of these smart connected  sensors inside could act as the central node. - And the central node then communicates with the  app that you're using and relays the information how well you executed the movements. [Mus

ic stops]  The second challenge that Abhi mentioned was finding the right way to wirelessly communicate between the nodes. And if I'd ask you to guess how they are connected, what would you say, Shuko? What would be your first guess, what technology did they use for connectivity? - I mean, this is a simple one because I use my phone almost on an hourly basis. I probably say they use Bluetooth or I don't know Bluetooth Low Energy to be really specific. - Bingo. Yay! It's kind of a no-brainer, r

ight? You want to have a small, energy-efficient device that transmits data? Of course you use Bluetooth LE! But Abhi says it's easier said than done. - Getting Bluetooth to work in a robust manner, is something which is very difficult. You can imagine now, let's say a SmartWatch and a mobile, and let's say a fitness band. All of these devices have their own way of working. All of the devices have their own clock signals. All of them are tuned to a different clock. To get this working toget

her and to get synchronous data to synchronize all of these movement data together with the perfect timestamp is one of the most difficult things which we had to come across. And to make it even work in a very robust manner, is even more difficult. - I'm always so surprised to hear what our Bosch engineers get up to and what they are actually concerned with. I mean you take it a little bit for granted. I work for Bosch  or I work for the Bosch group but I'm also a consumer and you know these g

adgets just work and wirelessly talk to each other. But then again, I can't say I haven't had issues with  getting Bluetooth devices to connect. - Just pairing two devices can be a headache already. But as Abhi said, building a robust and reliable network of eight devices that was a big challenge! A significant  amount of research went into figuring this out. - We have a huge research center, here within Bosch Sensortec, where Bluetooth experts, I think probably locked themselves inside the r

oom, did a lot of brainstorming sessions. And we basically did a lot of research and a lot of development behind it to really, get a very innovative solution. We also worked with our corporate research team in the US. And this basically led to the time synchronization,  which we have as of now. [Music starts] - It makes sense. When you have sensors measuring so precisely every little tiny movement that position has to be met by the synchronization across the network.  Otherwise you compromise

the quality of the data. So it's actually important to get this right, so you can track much more complex movements and get higher precision. - Which then allows for applications beyond exercising. Like we tried in that old episode where we also discussed the technology with a fitness instructor. Now it's possible to move into realms where it's even more crucial that a movement is executed in the right way. Think rehabilitation but without having to go to a physical therapy practice every wee

k. - I can imagine, people doing rehabilitation,  let's say at your grandmother's house or your parents' house, to feel much more comfortable, rather than just always going to a doctor and just going to a proper physiotherapy. And it's both ways, right? I mean, it's helpful for people who are actually using it, but on the other hand, it's also helpful for, let's say, professionals who really understand what the person is doing at home and help him even recover faster or help him even achieve

his goals of fitness very early. - I think there's just really a wonderful social aspect to these applications. There really, you live the "Invented for Life"-part of what we do. Or what the Bosch engineers do. But so yes, an app or a device can provide feedback on my execution. But also a therapist  would get exact data on my performance. - Given the respective solutions being developed by the industry. Mind you, Bosch Sensortec have developed the sensor - how it will be used is up to manufact

urers of consumer electronics, for instance. - But what about those devices that Geoff used at CES? Were they just made for CES for show or...? - Yes and no. Actually, yes they are just demo devices. But no not just for CES. Manufacturers who want to use the sensors can actually use this example implementation as well. - So let's say I want to develop a device and a software platform for remote physical therapy. I could simply modify the dance demo from the CES and sell it? I'm oversimplifyi

ng of course  but generally that's how it could work, right? - You're welcome to use what's already there. - You basically get the entire technology given directly to you. So we can give you already a rudimentary, let's say, stick figure or even these robot avatar, which we showcase in the booth, already given out to you where you can wear directly the sensors on your body and you are already starting,  let's say doing body movements, mirroring. - The idea is that developers of new devices and

applications don't have to start from scratch. They can immediately use the sensors and tinker with the hardware and software. - And you can basically then create a new backend, you can create a new front end on how do you visualize this data? How do I connect my physio therapist to this data? And all of these things then sort of become easily available to our developers. So it's basically, we take care of the technology, you take care of the rest. - Great, so I can just take the already ex

isting  hardware design and the implementation and focus on my rehabilitation app. Honestly, I think that's  really good news for developers of such solutions. But then, how do I get the AI on the sensors to  learn the movements that my patients need to do? - Another part of the technology platform is the software, that allows you to register new gestures on the devices. [Music starts] - We will be also providing a tool, for gesture addition as well as gesture recognition. That means we give t

he full autonomy on the customer and the developer side to really sit down and add their own gestures. Adding much more privacy to our, platform with their solution. As well as not only that, we also allow our customers to enable the threshold. Now, what does this threshold do is basically, this really sets the limit how precise the movements have to be done. - So this gives you full control in the end.  You don't have to talk to Bosch engineers to have them train the AI for you. You can do it

yourself, using the provided tool. - Okay so by having control over thethreshold I suppose I could always gauge what my patient needs. So, at the beginning of their rehabilitation process I can allow more wiggle room and other patients who are maybe further in their rehabilitation process,  I'll set the threshold a little more tight. - You sound like a very, very nice therapist! - Well, okay I'm opening a new practice and I'm taking new patients, so if you're interested...  No, I'm just jokin

g! That would be a terrible idea. - By the way, Abhi's team will also pre-train some  gestures for you. So if you tell them, what you want to use the sensors for, they'll happily deliver them to you with some gestures ready to use. You can imagine, Abhi's team is probably  the most fit and healthy team within all of Bosch. - Well, I hope so. They have been  doing a lot of movement, just not, sitting down on the computers, but doing a lot of  movement and they're also making me move a lot. - Abh

i says, they'll often go to the cafeteria or a meeting room and just record and train movements. Casual Monday in the life of Abhi! - Team motivation at its best! Let's talk a little more about the hardware that we mentioned. What actually are those sensors sensing? I mean, what are they measuring? What's in these wearables that Geoff put on at CES? - That is a great question. Geoff accidently  pried one open when he was taking it off. - Accidently... - Wink, wink. We were just done with the de

mo but our recording was still running... - Oh, see. All right. Now let me, let's talk about this. Let's, let's talk about this. All right?  Yeah, yeah. So interesting. My, my clumsy hands. So I, I actually popped the cover off of this,  which I appreciate Abhi having some good humor and only laughing at me and not telling me I'm fired.  But it's really, the, the device inside is rather small. So there's a USB, a mini USB port... Oh, it is USB-C, of course it is. A very small board and then just

the power supply. Please tell me a little bit about this chip architecture. It's about the size of my thumbnail. - That's I will tell you all the sensors if I can find them. - They're so small! It's like someone shrunk a computer board down. - This is the magnetometer.  - Okay, so which one's the accelerometer? - This is the one. - There it is, okay. - Super small. - It's so small, it's unbelievable! - It's smaller than the microcontroller. - That's unbelievable, absolutely fascinating. -

It's really nice to hear Geoff being so amazed by some micro electronics. Because usually he's like the software and these kind of things and it's just really, really refreshing! - Mister "I've seen it all". Yeah, he was quite blown away. For a good reason! What Abhi's team packed into this thumbnail-sized board is, well, Geoff said accelerometer, but actually it’s an IMU, an Inertial Measurement Unit. We've mentioned this type of sensor a few times before on the  show, so most listeners mig

ht be familiar with it. Then, as we also heard, there's a magnetometer.  That basically provides information about angles and rotations. And what wasn't mentioned  in this short clip is a pressure sensor. - We're talking for atmospheric pressure. So you can measure elevation, right? For instance when I climb a flight of stairs as part of my daily workout. - It doesn't even have to be that much elevation. - When it comes to pull-ups, you can understand, okay, if you, did you really pull the body

above the limit, or did you just do, let's say, half a pull-up? - No more cheating!  - It's amazing how sensitive that is! Big wow! [FX Wow] - So these smart connected sensors, they are  actually a modular sensor system. Depending on your use case you can work with just the Inertial  Measurement Unit, or IMU, or you can attach a magnetometer or a pressure sensor to it. Or both. - I find it fascinating because it makes it even more flexible. I think it's really interesting  that this is not a

sensor with a dedicated purpose. It's not like, this is a sensor for  a hearable and this is a sensor that measures the rotation of a SmartWatch. Instead, it can  go into so many different kinds of devices. - And perhaps even devices that don't exist yet. This technology platform, these smart connected sensors sure have the potential to  enable entirely new gadgets. [Music starts] - And who knows, maybe Geoff found an application for dancing. Maybe some dancing shoes that analyze how elegant yo

ur Tango steps needs to be. Why not? But it will definitely improve whatever was happening at CES. - Tango, oh my God! I'd score more points trying Jumping Jacks again, I think. [Music stops] But yeah, I can't wait to see in which devices we'll find smart connected sensors in the future. Maybe some of them will have their debut at CES next year. Wouldn't be surprised - Well, I'm hoping they will be as fun and fascinating as the dance simulator because then I definitely cannot wait to try them

. But anyway, thanks Melena, for reporting from Las Vegas for us and explaining some of the newest technology that Bosch put out there. - Thanks Shuko, and thank you so much our dear listeners! If you enjoyed the show, tell your friends about it! And of course, subscribe and watch out for our Deep Dive  with more insights on smart connected sensors. [Music] From Know-How to Wow!  The Bosch global podcast. Did you know that the code base of the smart connected sensors is tiny too? Kilobytes,

not Megabytes or Gigabytes.  More on that in the upcoming deep dive episode. With me, Geoff's voice avatar.