Traveling Across the Country for TreeHacks!

Justin, Natalee, Pranav, and me! Day 3, after we submitted our project.

Coding for 36 hours, sleeping on a concrete floor, and a red-eye flight back home the next night might not sound like fun, but somehow, TreeHacks 2020 will still be one of the highlights of my senior year in college. On February 14th, I flew out to Stanford to participate in their annual hackathon, which draws 1,100 participants from all across the country and the world. In this blog post (my first ever one!) I'll be recapping my experience, putting my project on show, and discussing what I learned from the whole process.

Of course, I have to start out with a shoutout to my wonderful teammates, Justin, Natalee, and Pranav. We all attend different schools, and met through the team-building TreeHacks Slack channel. Justin attends UIUC, Natalee attends RPI, and Pranav attends CMU. We decided our skills complemented each other and formed a team. Right from the start, we were vibing well.

First Impressions

Walking into the Stanford engineering building with the CEO of Nvidia's name on it was a whole experience. The place was radiating modern, techie, bougee-ness. The TreeHacks organizing team went all out, going far enough to put down pink and green footstep stickers leading into the building as well as a giant searchlight. Big name donors like Paul Allen were casually on plaques around the building. And, as I found out on a tour of Stanford, students also get all paid-for ski trips and boat tours of the Bay Area. No wonder this is the top school in the nation.

The hacking process didn't start the night that I got there. In fact, it started the week before, when my team first got together on Slack. One of the most important parts of the hackathon process is coming up with a good idea, so whenever I had free time, I tried to think of some problem areas and hackathon ideas, drawing on Devpost projects for inspiration. A good idea finally hit me when I was browsing the sponsor workshops, one of which was the Microsoft HoloLens 2 workshop. The HoloLens is a new augmented reality technology that was just released in the fall, and I knew developing something with it would make for a stunning demo. Since my team was focusing on the medical access field, my mind went to Alzheimer's patients who need help remembering where they put something, faces of their friends and family, and the sequence of basic tasks. It seemed like the HoloLens, a wearable and programmable headset, could provide visual aids to help them. After more discussion with my team on Slack and at the hackathon in person, we settled on the idea, with the main functionality being video journaling and facial and object recognition.

After we listened to a talk with Vlad Tenev, co-founder of Robinhood, at the opening ceremony, we got straight to playing with the HoloLens. I highly recommend trying this out at Microsoft Stores. It's hard not to say "This is the future" like all of us did when trying it out. On startup, I was greeted with a tiled display like Windows 10, but the cool thing was, it was floating in space in front of me. To press the buttons, I reached my hand out (which looked like E.T. if you were the one watching) and tapped the floating icons. I was also able to take some videos on the HoloLens, so it seemed like it could work for our project.

Hacking Begins!

At the workshop, we heard from a HoloLens developer from Microsoft who created a C# package that allowed developers to create interactive HoloLens apps in a few lines of code. Sounds too good to be true, right? Well we would try for ourselves. First, we had to install some massive programs like Visual Studio and Microsoft developer extensions. Oh, and not to mention, development could only be done on a Windows PC, which only Justin had. Nevertheless, we forged on, tempted by the novelty of the HoloLens. Three hours later, the software finally finished installing. It was getting late though, and all of us were feeling exhausted having travelled from the east coast, so we headed to bed.

Bed is an overstatement. I brought a blanket and a pillow to sleep on, but boy I was not expecting the sleeping room to be in the auditorium. I set up camp between the seats on the slanted concrete floor, and tried to fall asleep. 2 a.m., 3 a.m., 4 a.m. passed before I finally felt exhaustion overcome me. The next morning, I was up at 9 a.m. ready to code again.

At least the food was solid. Breakfast was acai bowls, an extremely healthy option compared to pizza and chips at other hackathons. Other food over the weekend included Indian food, subs, pasta, hummus, and burritos, all very satisfying.

It was time to start setting up the HoloLens for development. This was my task, and I traded laptops with Justin to do this, since he was going to start working on a machine learning model to recognize objects and faces. There was so little documentation out there to help me, so I went on Slack asking for help from the HoloLens developer who gave the talk. He hit me back with "I'm back in San Francisco for the weekend, but I'll try to help debug!" After going back and forth for a few minutes, he says "the HoloLens is difficult to set up initially for development." Well yes, I definitely believe that now, thanks for not telling me last night at the presentation.

The Pivot

We burned probably 5 hours on trying to set up the HoloLens. I guess that's a lesson to not try to be overly ambitious with new technology. Our team came together to discuss the new game plan. We decided we were going to make a prototype demonstration with our computer webcam instead of an AR headset. Here was our final idea: An always-on camera that records an Alzheimers patient's day, sending the video to a server to be processed. The output would be individual clips of important events and items that the person saw throughout the day, like where they last put their keys, when they saw their son, and when they last took medication. These clips would be stored in a video journal for the patient to review, hopefully strengthening their memories.

Finally Coding!

I started creating a Python Flask server as a middle-end, React server for the front-end, and a Google Compute Engine VM for the machine learning back-end. I was most comfortable with Python Flask, as I had done a lot of web development work with it in past projects, but I was less comfortable with React and cloud computing, having only done a few small projects with them. While I was doing all of this, Natalee was building the website and drawing some graphics, Pranav was preparing the pitch, and Justin was putting together the ML model.

By the afternoon, I was feeling a bit overwhelmed. There were so many moving pieces to the project, and naturally, many issues big and small popped up. Flask requests, Google APIs, CORS, and video recording all didn't work correctly on the first try. There was a lot left to be done on all fronts, and I knew the day was going to be long. I even missed out on the Citadel Poker tournament (shoutout to my company for being a huge sponsor, giving out water bottles, sweatshirts, and t-shirts) since I was in the middle of figuring out a bug.

The hours passed by, and before I knew it, it was midnight. Justin's ML model was ready to be tested on a GPU, so we started setting up a Google Cloud VM. Again, there were many hoops to jump through, such as getting GPU instances approved for the hackathon cloud credits we were using. We started with a dual-core 4 GB memory instance with a Nvidia Tesla T4 GPU attached. Everything was going ok, except pip installing packages was a bit slow and we had to resolve some package versioning issues. But, if you didn't expect it by now, there was an issue. The default OS that is selected for Google Cloud VMs is Debian (ugh, why is that a thing!), and there are no good drivers for it to recognize the Nvidia GPU. I tried a hacky solution I found on Google to no success. At this point, we burned through nearly two hours getting everything setup, only to find out we had to create a new VM with Ubuntu.

This time we beefed it up with 4 cores and 8 GB of memory. As a result, the pip installations went much faster. Nvidia GPU drivers installed fine on Ubuntu, and we were finally able to test the ML model with a GPU. I still don't really understand what Justin implemented, but it was a combination of 3 separate models into one pipeline, one for object recognition, one for vectorization, and one for facial recognition.

The ML model didn't work immediately on the GPU, so Justin had to go back and debug, then re-run the code a few times. But finally, at 4 a.m. the whole machine learning pipeline ran. A 5 second video took about 10 seconds to process with the Tesla T4 GPU. Not ideal for our real-time video analysis application, but still workable. It was exciting to see a video of me and Pranav get automatically processed into three parts where Pranav was seen, where I was seen, and the part where the keys and smartphone were last seen. We accomplished our goal of video journaling for Alzheimer's patients!

The last thing to do was to serve the model from a Flask server. Everything else, like recording video and displaying the video, worked at this point, but a good demo would need to have the video segmenting ML feature. When we called the model from a Flask endpoint though, it would hang and not finish running the model. It gave a weird error saying the processed video could not be written back to a file, even though it could when it wasn't run through the Flask server. I was super puzzled. It could have been related to the numpy-array-to-mp4 package that we used (which, by the way, is surprisingly not easy to do using plain numpy functions), since the error was something like ffmpeg stream could not be converted to mp4. But it was getting way too late in the night to figure something like this out. I was running on five hours of sleep, and hoping to get another five hours in before project demos the next morning.

Demo Day

This was my favorite day of the hackathon. We finally got to show off what we had built in the last 36 hours, and also got to see some other very cool projects. Without further ado, here is our project, remembAR! (The logo and graphics were hand-drawn by Natalee.) Here's also a link to the site, which might stay up depending if the Google Cloud credits run out.

Judging ran from 11 a.m. - 1 p.m, and during this time we saw 8 judges. Pranav pitched the idea very well, as expected. We had to use pre-recorded video clips instead of a live demo, but nonetheless, I thought we had a good shot at a prize going into the award ceremony. Everyone I shared the website link with also thought it was a great idea.

After judging, we cleaned up our mounds of energy drinks and snack wrappers from the table we were working at, returned the HoloLens, and headed over to the auditorium for the award ceremony. There were 8 finalist projects that presented to everyone. I could see why they were finalists - they all had very interactive and visual demos. Some of my favorites include Neutral, a really well-made carbon emission Chrome extension that ended up winning the moonshot prize, Reading Glasses, glasses that could read to you, SafeCup, a cup that could detect if your drink is drugged, and Lucid Drums, a VR drumming game. If the idea is the most important part of a hackathon project, then the demo is near second, and all these teams executed really well.

As we filed out of the auditorium, we looked at each other with a bit of disappointment that we didn't win a prize, but also a sense of accomplishment in the project that we created. It didn't take a prize for us to know that we had worked hard and built a useful app, and also made friendships with each other along the way. I'm glad that we all meshed so well for a team that didn't know each other 3 days ago. I'm also happy with what I learned during this hackathon. Since I was working across the full stack, I got to learn a lot of the tools and skills like Google Cloud and React that I had only briefly used before. I also got a better sense of how to shape a project around an idea and demo, since I learned those are the two most important parts of the hackathon. Going forward, I've got one hackathon left in my college career at UVA's HooHacks, and I hope to use these learnings to take home a prize. I'll keep you all updated!

Hoover tower