Lock-In

What was the lock-in?

 For our remaining time in the module we dedicated two days to completing our final iteration for the HRV sensor prototypes where we as a class joined together and collaborated on one project. We had from 11am - 4pm over two days to plan and execute our ideas and have a working prototype by the end.

 

The Idea:

The main core of this iteration was to have as close to real-time streaming of the heartbeat data which was read in from a heartbeat sensor from one device to another. Andrew had a genius idea of converting the data to an audio wave and streaming that live through WebRTC (Web Real-Time Communication). WebRTC is a set of APIs and protocols that allow browsers and mobile apps to create peer-to-peer (P2P) connections for real-time audio, video, and data sharing. This is often used in many over the web streaming services such as live streaming sites and video call platforms such as Zoom.

 

Project Architecture:

 As shown in the diagram above, our initial plan was to have one Micro:Bit reading in the data and relaying it through Serial to a Raspberry Pi. The data is then converted to an audio file and streamed to other clients through WebRTC.

 

Project Management:

For the project management we chose the Kanban method. The Kanban method is a visual way to manage work and improve workflow efficiency in project management. It helps teams see what needs to be done, what’s in progress, and what’s completed. This allowed us to break the project into chunks and divide up into teams respectively. The great part about this project is there was some part of it that played to everyones strengths, for example setting up the AWS servers was a strong point for the cloud students. Everyone had a task they were confident in doing. Our planning and meetings as well as the overall workflow was quite smooth and reflected real practices done in many tech companies around the world.

I was placed on a team with Mark and tasked with creating the Unity Client for mobile devices since Unity is Mark and I's forte as we are Game Dev students. It was also convenient as the Unity Engine has built in packages for WebRTC and streaming data.

 

 

Unity Team:

As mentioned previously, Mark and I were tasked with creating the Unity App. We decided to split our workload amongst both of us so at the very least should the streaming data not go as planned, we would still have a prototype for what the client could have looked like given it was working. Mark decided to work on the App itself whilst I worked on the code and streaming the data through Unity WebRTC. Immediately however we had an important decision to make, as the Unity WebRTC package was deprecated, meaning it was no longer supported in newer versions of the engine. I had discovered a newer package that is fully supported called Unity Render Streaming, which essentially does the same thing as WebRTC and even utilizes WebRTC in some of its protocols. However, swapping to this would not only mean researching the package itself but also researching compatibility with the already functioning signal server that had been created, so we decided to use an older version of the Engine with WebRTC.

 

Roadblocks:

The major setback in utilizing the WebRTC package in Unity was the documentation, or lack there of. Since it was no longer supported, after the major issues were fixed with the package the documentation was not updated to support those fixes. Meaning a lot of information about newer components were missing as well as older components that were mentioned in the documentation not existing anymore.Thankfully, Daniel had already made an iteration of peer to peer streaming with WebRTC in Python and allowed me to reference his code for connecting to the signal server. Over many different iterations however and triple checking I was attempting to connect to the correct server, I was not able to send or receive any data from the AWS EC2 instance. I decided to first try sending small packets of data from my Laptop to the server through the ping command but to no avail. Eventually I had discovered that my Laptop's firewall was blocking all outbound and most inbound traffic to other devices on the college's WiFi and after disabling the firewall, I was able to send data to the EC2 instance. Since I spent a lot of time debugging this problem, we were already nearing the end of our development time. I unfortunately did not get the WebRTC to connect correctly.

What we had to show for it:

Although I was unsuccessful in the streaming part of the prototype, we did manage to have a prototype for the UI and app itself. We chose to create a heartbeat image that would grow in size and "beat" based on the sample data of an audio wave. Essentially turning the wave into a value that would shrink or grow the heart. This data was also used to calculate the BPM of the wave and in theory would display a semi accurate heart rate and visualize this. Accuracy is lost in the compression of the audio file as well as packet loss during streaming over the web.

 Conclusion:

Overall it was very interesting to research and experiment with the WebRTC protocol within Unity. Although it was challenging, I feel I learned a lot and was great to put my skills into practice for something other than Game Development. All of us managed to do a lot given the short time frame and I'm confident if we had more time, we would have a fully working prototype with everyone's pieces of the puzzle integrated together seamlessly. It was very insightful to work with everyone and see how their perspective can shift based on the skills they have. The ideas everyone came up with were very impressive and this project wouldn't have been possible without everyone's input. In the end, we did have a working stream of data from one device to another and is shown in the video below.

 

 

As a final note, I would like to thank everyone in the class for the collaboration and the willingness to help each other. I hope we can work together again in the future. I'd also like to thank Jason for being our biggest supporter throughout this whole module, keeping the morale high at all times and being open to our ideas. IOT was the best module I have taken and it always felt like a place to focus on learning and trying something new rather than stressing over other modules and assignments. Thank you Jason for giving us the perfect environment to utilize our skills and learn from each other, wishing you all the best.