Blog 6 - The Lock In

 Heartbeat in the Cloud: Our IoT Lock-In Project

At the end of our IoT module this semester, we took part in a a two-day lock in, where we worked in teams to build a full project in a short space of time. the project idea was to create a heart monitoring system using a micro:bit and heart senser to collect heartbeat data and send it to a cloud where we it could then be streamed and visualised in real time either on a website or Unity application.

We spit the whole thing into 4 sprints, each team focusing on different areas of the project, but we all worked together to connect the pieces. In the beginning we worked together to build a timeline and structed plan on how to attack this project splitting each of us into teams where everyone honed there skills in order to execute the project. Both me and Brendan felt our game dev experience could be used.

Teams

Unity App Team: Me and Brendan — building the visual Unity application to show the heartbeat.

RTC & Audio Team: Daniel, Shane, and Andrew — working on WebRTC and FFMPEG to handle the audio streaming.

Micro:bit & Sensor Team: Arthur — Micro:bit data streaming

Cloud Infrastructure Team: Alex Jay and Dean — building the cloud infrastructure that tied everything together.

Sprint 1:

Me and Brendan started by setting up our unity project. The main idea was to create an app where the heart would beat visually in sync with the heartbeat data from the cloud.

We started exploring libraries that could handle real-time communication (RTC) which Andrews team had come up with this way of data streaming the heartbeats so that we could get as real-time as possible. RTC is commonly used in video calls, live streams and in our case audio streaming of the heart data.

we managed to find a library called WebRTC which could stream audio between two devices, we started trying to test this between both laptops. 

We setup our project using git pushing and pulling changes as we went.

Sprint 2

In this sprint we kept looking into getting the RTC library up and running. Unfortunately we started to run into issues were the WebRTC version and had unsupported features. The online documentation was mostly old documentation and didn't match up with the newer versions of WebRTC. 

We also discovered another option: Unity Render Streaming, which might have worked better. but with time ticking we kept pushing with what we had.

While Brendan continued trying to get the WebRTC connection working and integrated with the cloud (set up by Alex dean and jay), I stared working on the UI, designing how the heart would appear and beat inside the app.

 By the end of this sprint i had a Unity app showing a heart image that could beat based on a simulated audio input.

Sprint 3 

In this sprint I spent my time refining the app.

I set up a audio listener in Unity that could read a heartbeat-like file and detect heartbeats in Realtime. This involved sampling chunks of the audio and applying thresholds where only beats would trigger the heartbeat on the UI heart.

This could be then tweaked with added controls for the heartbeat Scale, Speed, and threshold from the incoming audio.

It wasn't heartbeat data yet, but the system was ready and working.

Sprint 4    

In this sprint we attempted to connect up my work and Brendan's but Brendan was having difficulty's with actually establishing, Brendan worked hard at trying to fix this issue while i helped testing and improving the unity app. The issue turned out to be a firewall issue, in the end we didn't have enough time to get it working between the cloud and the app (but we were confident that with more time this was possible). While working on the unity app the other teams were making progress but also had similar issue with time.

Final Thoughts

This project was one of the most collaborative things I've worked on, Every team had their own focus, but we were all contributing to every goal to build  a working IoT system that could monitor and visualise. someone's heartbeat in real-time.

It was exciting to be part of such a large group all working together towards the project.

I learned a lot about WebRTC and how it works including how to manage and work together with a large group.