Projects category

Upwork Image Recognition

Like all other technologists, I share the dream of building something or finding a side hustle towards the goal of financial freedom. The common adage is to find problems, solve them, and the money follows. So in search of these problems, I created an Upwork account to find job listings.

Most listings were uninspiring. 90% of jobs wanted web scrapers that rotated IPs to get around banning, and others wanted to pay $5 an hour for a 10x, business revolutionizing engineer. I wanted to find a well-scoped and tractable problem to tackle and stumbled upon a request for an image recognition task.

Upwork Job

My previous experience with CV models have been primarily for classification purposes, so I thought this would be a fun opportunity to learn something new. So the provided training data consisted of single icon images like this: Emperor Image And then composite images that almost looked like a slot machine. Emperor Image

Given my rather uninitiated knowledge of object recognition, I googled the best frameworks for these problems. I landed on https://yolov8.com/ which seemed promising given its popularity in searches, and it appeared to outperform other methods while still being actively developed.

The main problem is that the sample job listing provided such few images for training. We ultimately wanted objection recognition on the screen images and then a print out of the objects list.

I was only provided 4 screen images and a handful of images with singular icons. I wanted to keep 2 images for training, 1 for validation (parameter fine-tuning), and 1 for holdout testing. So I carefully selected the 2 screen images for training that had each possible icon and took screenshots of each icon within the images. I did the same for the image in the validation set. I then used labelimg to label the images within the separate screenshots, and then also to label each icon in the holistic screen images. We initiated the model from the base yolov8s model and fine-tuned the model on the available image set. The result on the holdout test image was promising. I'm not sure of the image post-processing that made it appear more like a negative, but it worked reasonably well. Batch Validation

This was an interesting exercise in building an obviously overfit model for this specific task given the really limited training data. I initially only trained the model on the few single icons for too few epochs, and it returned nothing. However, it improved once I added the additional screenshotted icons from the screen composites. I also managed to improve performance by reducing the transformations. Typically, we want to transform the objects via rotation, reflection, etc. to prevent overwriting. But I knew additional constraints of the task such as the fact that we would never pass an upside or reflected "princess" image. It was a quick and nifty project that I can see actually being useful to know in the future.

Curly Icenberg

I updated my DigitalOcean droplet and consequently, this website with intention of working on a new project related to the Housing Works NYC non-profit. The organization supports the fight against AIDs and homelessness. And one of their funding sources is via an auction. The auction often has incredible items, including unique artwork, but unfortunately, there's no means of staying updated about the items unless you visit the page every day. Consequently, I wrote a scraper that emails me all the items available in the next auction. It's not a bidding bot, just an alert system. But it's fun to see what random assortment of items is available. And our empty white apartment walls could use some personality.

So, 8 months later, I've finally bid in my first auction and actually won! It is a print signed and dated from 1968 by the artist John Ulbricht. For those who know, the artwork is painfully on brand. I've named it Curly Icenberg, who is a fictional head of lettuce that I created in a deck to explain the Blend Ratio Solver project I worked on at Bowery. It's an impressive work of art, much larger and vibrant than the images led me to believe. And the little dopamine hit of winning an auction was icing on the cake.

Curly Icenberg

Krispy Kreme is Never Hot

Desserts have never been a calling for me, but I've always had a soft spot for donuts (along with strawberry ice cream, carrot cake, and cheesecake). NYC has some great donuts (shout-out to the tres leche donut at Doughnut Plant), but I'm a sucker for the familiarity of a classic, warm glazed Krispy Kreme donut. When I heard they were giving away free donuts every time their iconic Hot Now signs turned on, I was ready to load up. I had recently moved nearby the Flatiron location.

The donut lover in me was inspired to create my own alert that would check to see when the Hot Now sign would turn on at the Flatiron location. Krispy Kreme's store locator website has an indicator that shows when the Hot Now sign is turned on. So I created a service that would run every 8 mins, check the Hot Now status, and send me a text message saying "HOT NOW!". To my dismay, I had received no salacious text messages for a Hot encounter -- it turns out that the Flatiron location never turned on their light even though they claim it happens twice a day.

Hot Now Claim

I would know because I logged every attempt to check (except when the VPS went offline for a month when it ran out of memory oops)

Hot Now Chart

So it turns out there are many locations in Manhattan that never turn on except for Times Square which, upon further review of the data, never turns off. Every single run says Times Square is Hot Now. I like donuts, but I'm not taking myself to Times Square. I also thought it might've been them purposely avoiding the promotional period (06/08 - 09/05), but this pattern persists well after 09/05. Perhaps there's a strong asterisk on participating locations, but this felt downright deceptive!