Posts
-
Feature Creep!
I know I made a post earlier about the biggest pitfall in Project Management being Feature Creep. But here I am, dealing with the problem myself. I started out on my new project with a simple app in mind, that had some slightly complicated math behind it. The idea was simple… Let people shoot videos as ink sketch animations! In case you haven’t seen what it looks like on my twitter feed yet, here’s a pic.
I got that to run at a smooth 60fps on my iPad3 at a 1080p Square Resolution, and around 27fps on my 3 year old iPhone 4 at a 480p square :) I’m sure I can get it to run at 1080p^2 on the newer 4S and 5 devices.
Tasks involved so far:
- Algorithm to convert Video Frames into sketches really fast.
- Learn GPU programming. Either through OpenGL, or using libraries such as OpenCV and GPUImage.
- Design UI w/ Record/Stop Button that saves to the Photo Library.
Then I thought, may be I should have different kinds of animation effects. That led me to designing an interface to select different effects. Sounds pretty simple. Why not?
- Single View App to Multiple Views
- Build multiple Effects, and create a whole class hierarchy for abstracting common functionality among all effects.
- Build an effect selector, that shows a list of effect icons, that you tap to get to the recording interface with that effect applied. A simple Table View… May be a side-scrolling “Belt” like the one found in Fotoyaki.
But, the effect selector was boring. Being a UI Snob, and due to my recent obsession with minimalism – a total 180 from the Skeuomorphic clusterf**** found in Fotoyaki – and “zero-appiness” like the design found in Flipboard, I went about getting creative with the UI for selecting effects. I kinda showed a preview of it in my ugly prototype from the last post.
Well, if splitting the video feed and applying different effects to different regions was a small speed bump to get over… making those effects scroll on top of the video feed “as smooth as butter” is like climbing K2. That’s where I’m at now. Not on top of that mountain, but barely at basecamp.
- Split the Video feed and apply different effects to different regions.
- Create a custom Scrolling View that hooks into the UIScrollView’s CoreAnimation layers to track its current position and adjust the video splits, and choose the appropriate effects to be applied.
And as if that wasn’t enough, I want the app to allow you to shoot videos in scenes. So, instead of Record/Stop. It’s Record->Pause->(Select different effect & Continue Recording)->Stop. It’s a really simple way to make movies with multiple scenes in them. (ala Vine, only longer).
- Figure out how to manage Scenes, and append Live Video to the end of previous recording. (And implement it of course)
And finally, the thing that made me realize all the feature creep that went into the project thus far. I want to build a whole backend component to the app for 1. Simple Sharing, 2. Really cool collaborative video making… Alright STOP!!! Save it for V2!
Anyway, I still think I can get all this done within the month, and ready for testing. I originally planned on working on the first three task bullet points over a month from June 1st - July 1st. I’ve been working on this part-time since May 8th – My full-time hours these days are spent on finding my next paycheck ;) So, In about 2 more weeks… June 8th? If you want in on the Test phase, let me know, and I can send you a TestFlight invite to try out the app first hand.
But, more importantly, if you are looking for a Software Engineer w/ mad skillz. Check out my resume at http://vijayarunkumar.com/resume.pdf and Hire me!
-
The Good, the Bad, and the Ugly
The Good:
I found out about this awesome GPU accelerated image processing framework, GPUImage. Thanks to @bradlarson for writing it, and to @roderic for tweeting me the link. It’s an extremely well-designed library, and it took me about 15 minutes to comment out all the openCV code, and integrate GPUImage into my test app. Of course, my test app didn’t do much, other than run Canny Edge detection on frames from my video camera. But anyway, the results were pleasing to say the least. 640x480 on my iPhone4 runs at a smooth 20+ fps. And, although I haven’t tried it yet, I’m sure my iPad3 can do much better than that, and could probably handle 720p pretty well.
The Bad:
I’m no longer with Bre.ad. I wish that affair had lasted longer, but it wasn’t meant to be. So, If you, or someone you know, is looking to hire a mobile/backend engineer with mad skillz (hopefully inside the 7x7), check out my resume and pass it along. Okay, that wasn’t so bad. Onwards to…
The Ugly:
So, I have this super slick UI in my mind for the app, that shows live effect previews overlaid on top of a live video stream. It involves applying different effects to different parts of the video, and making the live previews scrollable as well (and smoothly at that), to make it appear as if you are looking at the world through these magical panes of effect-glass.
Anyway, I had to first make sure this was possible, and hacked out a simple prototype with 4 live effects (Canny, Canny+Inverted Colors, Sepia, and a Posterize/Saturation filter). And, following in the age old tradition of making ugly prototypes, it’s a pretty ugly implementation. But, it’s the beginning of something amazing. Check it out!
-
This Lunch Ain't Free Either
So, I got all too excited a bit prematurely yesterday. You see, that line drawing of my parents’ backyard - It was performing good enough because I was limiting video capture to a pretty low resolution of 322x240 pixels. Even old school VGA (640x480) has 4 times as many pixels as that, and my iPhone 4 capturing video at 720p (1280x720) produces videos with 12 times as many.
I knew this wasn’t gonna be an easy task. Just a few lines of code using a readily available open source library, and that is it? I was a bit surprised, and was feeling a bit let down, by how quickly it all seemed to have worked itself out yesterday. But, today when I increased the resolution.. my iPhone chugged along barely at 4fps for VGA, and both my iPhone and my iPad 3 Retina came to a screeching halt at higher resolutions.
Does this mean the app is not feasible? Of course not! The simplest solution would be for me to limit the output of my app to generate only low-res videos. But, where’s the fun in that? Also, being a usability snob, I would cringe at the idea of limiting the user to an inferior video quality, after they’ve gotten used to a retina display.
So, this lunch ain’t free. Okay, let’s skip to the dessert menu and get to the good news then.
OpenCV on iPhone is not OpenCL based, and thus, not GPU accelerated.
Under usual circumstances, this would’ve been considered to be bad news. But here, it also means it’s not the end of the road. Using CPU for image manipulation (large byte arrays of pixel values) is just a bad idea. I mean, look at Fotoyaki. I used my own simple algorithms that run on the CPU, and to deal with the delay involved, I had to come up with snarky comments to distract and keep the user happy while their phone chugged along. Compare that to the real-time effects of an App like Instagram.
I just have to write my own OpenGL ES code to get all that lovely math to run fast on the GPU… and GPUs are very good at their math skills. Now, if only I knew how to do that :)
-
Feasibility Test: Passed!
That idea that I posted about earlier… Yeah, the one that I can’t talk about just yet. Yes, that one! All that math, and the algorithms which worked out fine in my mind take a lot of processing power. And, if I’m going to build a “Video” app, ensuring real-time performance is paramount. But of course, you all knew that already. Why else would I be reading up on OpenCV and GPU accelerated programming?
So, today, I hacked out a little app to make sure that what I want to build can be done with today’s hardware. i.e, it should be able to run on my 3 year old iPhone 4.
Okay, I will be a little less cryptic about it, and let you in on just this one little secret. For what I want to achieve in the final app, I need to be able to detect edges in real-time. That means, I should be able to apply a gaussian blur function on the video frame (to smooth out unwanted edges), and then run the Canny86 algorithm on top of it to detect the edges. And do all that while achieving real-time or near real-time performance on my iPhone. I need at least 12 FPS for my app. (In fact, I prefer 12fps for the effect I desire, over either 24 or 30fps)
Coding it in OpenCV couldn’t have been any simpler. Talk about readily available functions with names like cv::GaussianBlur, and cv::Canny. It’s as if they wrote the library with my app in mind :) Anyway, skipping over to the good news: It works! I wrote the app not knowing what kind of performance to expect, and when I ran it for the first time… I smiled.
Here’s a little preview of the app showing my parents’ backyard:
-
Starting on a New Project
So, things have been a bit uncertain at work, as we’re trying out different things to gain traction for our product. There are many reasons why we haven’t been able to find as big of a user base as we wish… starting with, not having an open sign up :) But, I’ll save those reasons for a different post. At the end of the day, I work at a startup. And along with the thrill & benefits of working in a small team, comes the risks of the idea/product not making it.
So, we’re kinda at the stage where we go big or go home, and I really hope we “Go Big”, and not go home :) Anyway, these uncertain times are usually when I get most creative with my side projects and start-up ideas. My mind started going through all the big ideas that have been brewing - An open ad server on the cloud; A location based gaming framework with local ads; A project management tool on the cloud for software companies that simplifies scrum; An app that vastly improves on the UX for music consumption on iOS devices; A game based on broken-picture-telephone… And, after a bit of pondering on my BART ride back to my parents, my mind settled on something totally out of the blue.
An iPhone/iPad video app. I can’t say much about it, but I promise to keep this blog updated on its development. Here’s what I can say…
- The app is math intensive (which all works out just fine in my mind. At least, it all did on the BART ride)
- I just started reading up on OpenCV and GPU accelerated programming.
- It will blow your mind ;)
I am gonna be busy at least until the end of May to finish up our current project. If that is the case, then I expect to have the app on the marketplace by July. A month seems like a good amount of time to get it done, if I get to work on it full-time evenings.
subscribe via RSS