Real-Time Applications: First Dive

After working primarily with just ExpressJS & MongoDB, I started to wonder if I would need to take a whole new approach if I wanted my application to be a real-time application. Routing is simpler to understand, but each request for a new route results in a refreshed browser that returns things to their initial state, which, in the end, makes everything just feel really really slow with a broken user experience.

I already knew I wanted to make the app real-time, but I was putting it off thinking I’d come back and just do it after I get a functional prototype built using just ExpressJS & MongoDB. Limiting what I’m using in my stack will reduce complications and make it easier to build. However, I started to really feel like I needed to consider architecting everything with real-time in mind. Continuing down the path I was going would result in a theoretically simpler application in the short-run, but I think I’d be sacrificing a lot in the long-run. I decided to stop what I was doing, and restart with the notion of building my app to be a real-time app.

Socket.IO was my starting point for learning how to build a real-time app. I spent the past 3 days just trying to figure out how Socket.IO works. The available documentation wasn’t really helpful, so I searched around for examples and tutorials in order to see how it was being used and to find things I could get my hands dirty with. I spent a long time trying to figure out how to integrate sockets into my existing routes. I wanted to do something like:

app.get('/:user', function(req, res) {
   io.sockets.on('connection', function() {
      //do something
   }
}

The intent was to have a real-time event occur for this route, and to give that socket access to the parameters provided in the route. However, it took me 3 days to really figure out that how I was thinking about implementing Socket.IO was entirely wrong. Yes, I do want certain real-time actions to occur with certain routes, but instead of trying to access the socket through routes, I need to accurately map what events the sockets should be listening for. I forget where I read it, but someone else was trying to do what I was doing and asked for help on stackoverflow. The response that really helped me begin to understand everything was regarding Socket.IO as the communication layer that communicates all the actions between the server and the client. Yeah, I know that’s kind of how Socket.IO is described. It’s like realizing the sky is blue…

Anyways, during the time I spent with Socket.IO, I came across another handy thing – Mongoose ODM. It was frustrating because I had noted it as something to use a while back, but completely forgot about it. I realized that Mongoose basically replaces a lot of the boilerplate I had to build out for MongoDB + Express, and does a better job at it.

Long story short, the real-time app is slowly coming along. In the end, my stack comes out to be ExpressJS, MongoDB, Socket.IO, Mongoose, Crypto (for encrypting), Connect, and Redis (eventually).

Mongoose ODM

Love that moment when you spend days, if not weeks, building your app only to realize that there’s something already out there you missed that would make your life a whole lot easier. Mongoose ODM – you’re going to be my friend.

Book: The Founder’s Dilemmas

Just bought The Founder’s Dilemmas after Brad Feld encouraged entrepreneurs to read it. Read Brad’s post this morning, then went straight to the nearby bookstore. Luckily, they had one in stock and I bought it full price. Would have been cheaper through Amazon, but with how busy (and spontaneous) things are with work I figured the extra $10 is worth it to gain those few extra days to read the book.

I read the introduction as my break from coding, and I think I’m going to enjoy this book because this is related to the things I wanted to do as a PhD student.

Node.js: Not just hipster?

Node.js is often considered the next hype in terms of technology, and even comes across as hipster tech to some people. I ended up using Node because I was looking into what I could use with MongoDB. ExpressJS is what I ended up coming across, and I started to really enjoy using it after working with it a little. Since it was fun, I decided to use it to build the app I’m working on.

One of my friends, whom I usually trust a lot when it comes to discussing technology merits, recently decided to go against Node. I remember a few years back he was really into using Node, but for some reason he decided to go against it. It might be because it is the next hype and that’s not just something he wants to get into anymore. Then again, I also remember when he was really into Ruby on Rails, CakePHP, etc. and now he’s against those too. During our last discussion, he mentioned his own personal philosophy he developed with languages and that’s what drives his opinion now. I don’t have as strong of a personal philosophy when it comes to languages, but his decision against using Node made me reconsider building my app in Node. I was thinking maybe I should look into Python and Django instead of Javascript and ExpressJS to build my app because Django has been around longer and should be more stable.

As I was reading my morning news, I came across an article discussing how the new LinkedIn iPad app is actually built 95% HTML5, and relies heavily on Node. They thought it wouldn’t be stable so they kept nginx, but it turns out Node is stable enough and can scale. After reading this, I thought it really is more dependent on the programmers capability in using the technology than the technology itself that determines how stable, and scalable, an application can be. All technologies inherently have their own strengths and weaknesses, and that’s why we will always have new technology and languages being born. C++ and Java are considered the industry standard, but they too were born in response to the technology and languages of their time.

Node has its uses. It’s up to me to understand it’s strengths and limitations. I may need to switch entirely to Python/Django (or another language/framework), use a hybrid, stick with Node/Express, or something else entirely. For now, I’m going to stick with what I’m using since I’m really enjoying it. Issues will arise as the app is used, and if it turns out Node cannot address those issues effectively (and efficiently), then I’ll have to look for what can. It will be worthwhile looking into other technologies that are used to deal with scaling.

Reflection: Life & Entrepreneurship

I have been fortunate enough to celebrate another birthday. As another year is added to my life, I wonder if I’m living my life to the fullest. Steve Jobs talked about death being the destination we all share, and the biggest regret people have before they pass is not pursuing their dreams and doing what they love. So with another year passing I want to reflect on my own life.

Childhood isn’t something I enjoy talking about so I will talk about high school leading to college. Before graduating high school, I wanted to be a programmer. I really liked being able to build things, and enjoyed logic subjects like physics, math, and programming. I took AP Computer Science during my junior and senior year, but failed to pass the AP exams. I still remember my teacher talking to me after I failed the first exam by one point. He said, “Since the grade you got now is what you need to pass the next course’s exam, you should pass. Or else I’m going to hunt you down at UW… By the way, I’m a guest lecturer there so I’ll find you.” It still makes me laugh thinking about it. FYI, I failed the exam by one point again, and no, he didn’t hunt me down… To the best of my knowledge at least.

Although I was taking computer science in high school, my parents didn’t approve of me studying that in college. The dot com crash was still fresh on my dad’s mind, and as I was growing up I remember him constantly telling me that the computer science field didn’t lead to a secure career. Even though I enjoyed it, I wasn’t allowed to use a computer freely (or TV… or much else for that matter), and what my dad told me left quite an impression (the harsh punishment for breaking the rules was a strong deterrent too).

The paths “encouraged” by my parents were typical of Asian families: 1) Doctor; 2) Lawyer. Saving lives appealed to me more so I said I’d become a doctor. My first quarter course load was physics, math, and chemistry. Next quarter I passed on math and physics since I was already ahead in the series due to AP credit, and took Introduction to Psychology and Japanese. Learning Japanese was always a passion of mine, so I was glad to finally be able to take the course. Despite switching up my courses, I ended the first year with an average GPA of ~2.7 out of 4.0.

I realized that I really didn’t like what I was doing. I didn’t want to become a doctor. I skipped almost all of my classes because I just didn’t enjoy it at the time. Even though I really wanted to learn Japanese, I still ended up skipping a lot of the classes. I remember arguing a lot with my parents over the summer once I told them my grade.

As I was waiting for the first class of the new academic year to begin, I started to really resent what I was doing. The class I was in was a chemistry class I was retaking in order to meet the grade requirement for the next course. As the professor walked in and started to set up for class, I decided this wasn’t what I wanted to do, and got up and left the class. At the time, I really wanted to learn Japanese and study abroad. I had another lengthy argument with my parents, but this time they said I can do whatever I want as long as I do my best. I went straight to an advisor and changed my major to Japanese linguistics.

I look back on that and am reminded of how I finally took control and started living my own life. Studying abroad is the best thing I did in college. It was delayed a little though because of my low GPA. I wanted to join a program that followed our academic year, but because of my GPA the study abroad advisor said she couldn’t recommend my application. However, if I showed progress over the next 6 months and raised my major GPA, I could try for the year long program that followed Japan’s academic year and started in Spring. I worked hard and impressed the advisor so much that she even recommended me for the JASSO scholarship without my knowledge. Only 2 people receive that scholarship for our school, and she told me I was the first one she thought of. During one morning commute to school in Japan, I came to the realization that a Japanese linguistics degree would not help me find a job that could support a family. I went straight to the library and looked into available graduate degrees. Being in Japan, a country known for technology, I started to think that what my dad had told me growing up wasn’t all inclusive. So I decided to look into graduate programs that I thought were more technical. I called my parents that day and told them my plan. They were a bit surprised that I had decided to pursue a graduate degree on my own, but said I could pursue it.

I decided to join the Masters in Information Management program at UW because I thought it would be a good blend of technical and business. There were two things that left a lasting impact. The first was the user-centered design course we took our first quarter. The professor really changed the way I look at things. We were pushed to envision what mobile devices would be like in the year 2020, and had to focus on product design and presentation because we presented our solutions to a panel of senior employees at mobile companies like Intel. The change management class left the biggest impact. For the first time, I felt like a professor actually cared about helping us succeed in our lives. The course was designed to encourage us to pursue what we enjoyed, and to help us realize that we’re capable of achieving anything we set our minds to. I still work with and consider Kevin Desouza my professor, mentor, and friend.

The change management course really made me evaluate myself again. Coming into the masters program, I had originally thought I would graduate and look for a management position at some large technology company. I thought that was all I could achieve. However, I was beginning to think I wanted to try and achieve more. During the final quarter of our program, Kevin told me about a jam session he was putting together where companies would share their real information problems, and we would be tasked with finding a viable solution. I got a team together, and the problem we were presented with was a knowledge management problem. We were runner-up, but the experience was invaluable. This was when I really decided that I wanted to try and venture out on my own.

I had several ideas, but at the time I thought the natural route was to start a consulting company. I wanted to help foster knowledge exchange. Unfortunately, the global economic crisis had just happened a few months earlier so no one was willing to spend money on some new grad. I decided to drop that and pursue another idea I had developed during an internship I did in South Korea. I realized people needed information, but in many cases that information was locked away in another system. I decided to try and pursue that idea to create an integrated knowledge system. Two friends, who are both old enough to be my dad, took a liking to my idea and decided to join me in the venture. I thought it was great because one had extensive executive experience and the other had engineering/architect experience. They both also shared the same enthusiasm I had for solving problems. I thought we had a great team. However, for various reasons and a fundamental difference in how we viewed startups, the venture never took off. We had opportunities, but personal differences made it difficult to take those chances.

Around this time, Kevin made a Facebook post wondering if there were any PhD students interested in his work. I responded and said I wasn’t a PhD student, but I was interested in his work. He responded saying we’ll talk, and before I knew it I was putting together my application to join the PhD program. I thought it was good because it gave my venture a deadline to start, and in the event it failed (which it did) I had something else that would allow me to pursue my interests. I was deeply interested in entrepreneurship and innovation, and so was Kevin.

After joining the PhD program, I really started to wonder if this was what I wanted to do. I was studying innovation and entrepreneurship because I was really interested in it, but is studying it all I wanted to do? I realized that I really wanted to jump in to try and be a part of it instead of sitting on the sidelines researching and documenting it. Towards the end of my first year, my professor asked me how I felt about the program so far, and I told him my honest opinion. I was afraid at first that he would be upset after he’d helped me so much, but he wasn’t. In fact, he was really encouraging and told me that I should pursue it. Go forward without looking back. It was perfect timing too because he was pursuing something else as well. With my professor’s blessing, I quit the PhD program. Several people thought I was crazy. I also talked with the two friends I was trying to start a company with, and told them that our differences made it impossible for us to do a startup together. Nothing had happened in the year we were working together, so I was going to go and do this on my own with one of my best friends. I still think they’re really talented, but a startup isn’t something they can commit to because of the high risk it has.

The goal with my friend was to look into joining an incubator. However, despite having known each other for years, we hadn’t really “worked” together. We played badminton together so we knew how to fight for each other, but we hadn’t worked on something together. We decided to try and create a badminton tournament management app as a way of working together and giving back to the sport we loved. We gave ourselves a few months, so I thought we were good. Unfortunately, things weren’t good. Both of us had some important personal matters come up that slowed things down, and in the end my friend said he couldn’t consider doing a startup at the time.

A bit devastated, I took some time to think about things and committed to finding a job. I believed in my idea, and was able to refine it and scale it down to a starting point that wasn’t a “boil the ocean” approach. Once firms grow larger, it is (extremely) difficult for leaders, particularly the CEO, to know what’s going on in their company. What projects are going on, who’s working on what, what’s the value proposition of the projects, etc. These are all things the CEO, as well as managers, need to know, but have difficulty figuring out without asking someone who ends up needing to ask others. The thing that struck me was when I read an article about Larry Page emailing managers telling them to pitch him on what they were working on in 60 words or less. There is no way this is a one-time event. Senior managers, especially CEOs, need a way to know what’s going on in their own company. Surely technology can serve as a tool to better support all levels of the organization. This was when I really felt like I figured out the starting point – project management. The thought of the integrated knowledge system really came together as a result of the jam session. As I contemplated the issue we were presented with, I started thinking about how best practices and lessons learned repositories are severely underutilized. That’s because they’re repositories. The very definition of repository is to store something. It isn’t designed to make things easy to retrieve, or leverage what’s stored inside. I was discussing this with the guy who had extensive executive experience, and he shared with me how a lot of waste was done at the large aerospace company he worked at because best practices and lessons learned would rarely be used primarily because it was so vast that no one knew how to find what they needed when putting a project together. I thought a system could be designed to make better use of the vast knowledge in companies. My starting point for this was around project management, and I ended up being more convinced that project management, particularly the design of the software, was critical in solving this issue. Most of the people I shared my idea with gave me their honest opinion that while it was ambitious, if I were able to pull it off it would be something people needed. The more vague idea I had shared with the executive guy is what got him really interested in getting more involved because he said he would’ve wanted what I was setting out to do. However, I had failed so I needed to go and get a job. I put my ambition to rest and started looking for a job.

As fate would have it, someone else I knew was doing a startup. He was a part of the cohort after me in the masters program, and we were meeting up for coffee to talk about his startup. He invited his brother and co-founder over as well since he was in the area, and we talked about their startup and entrepreneurship in general. This really started getting me interested in trying again. At one point a while back I had shared that I was doing a startup and they asked me what happened with it, so I shared what happened. I also shared that despite the issues with my “co-founders” (we’re the founders of nothing), I also had to really reflect on me being an entrepreneur – or a “wantrepreneur.” The both of them looked at me for a bit, said they understood what I said, but also really encouraged me to reconsider. While it is tough, being an entrepreneur is the most fulfilling thing they’ve done.

Shortly after, my best friend and I were talking and he was more open to trying to do a startup with me again. We put the badminton tournament management app aside and started working on my idea. A few weeks into it, my friend realized he wanted to pursue something else, and this didn’t seem to align with his pursuit. He is one of the most talented developers and designers I know, especially because he has an uncanny desire to continuously learn, but his interest was more in the intersection of art and expression through technology. While I was pretty disappointed that he bailed on me a second time, I want him to pursue what he loves. I had a feeling this would happen too, so I wasn’t too surprised. I spent a few days really thinking about things again. What is it that I’m failing at? Am I really just not cut out to be an entrepreneur? Then it hit me: the problem is me.

Reflecting on everything, I realized that the reason why things weren’t succeeding was because of me. The entire time I was relying on other people to get things done. Mark Suster has the phrase, “JFDI,” which is an altered Nike phrase so you can figure out what it means. I was thinking I was doing something and things would magically get done when in reality I was sitting on the side doing nothing. I needed to stop thinking so much about the idea, waiting around for others to do the work, and just do it myself. It hit me that this was my last chance to pursue this and really get things going. I won’t get another opportunity. Although I’ve had other (paying) work to do that has slowed things down a little, the past 2 weeks have been the most productive and motivated weeks since deciding to change my major to Japanese. I have spent my time picking up ExpressJS and MongoDB so that I can build a functional prototype.

This time I really am jumping in. What little I have is being spent towards this. I will give it my all and see where it goes. Live life without regrets.

Python: CSV to JSON

Over 2 years ago, a close friend of mine encouraged me to get back into coding. The language he wanted me to try and learn was Python. In order to learn through application, he challenged me to create a simple program that would take a csv file and convert it to JSON (he helped me when I was stuck).

To my (pleasant) surprise, I am finding a need for that program. A (soon to be official) consulting job I am working on had me create a large spreadsheet filled with publicly accessible data, and now I’m tasked with interacting with that data to help write a report. I’m not a master of using Excel to its fullest extent with things like pivot tables, but even if I was I’m pretty sure I would dislike working with Excel. I know it’s a very robust tool that has withstood the test of time, but it’s still cumbersome to use and definitely not the most enjoyable user experience.

While I was working on the report and scrolling through the data, I thought to myself, “Wouldn’t it be nice if this was in JSON so I can import it into Mongo and build a basic querying app around it?” Then I realized I do have an app that can convert it into JSON! Unfortunately, my gripe with Excel was instantiated when I realized that Excel will have entries with various character encodings and display an alert that you have compatibility issues with the data, but it won’t let you convert everything into a standard encoding like UTF-8. The problem probably lies more with the creators of the data, but if Excel can recognize the error one would hope it can help fix it as well. Luckily, a quick Google search helped me find a solution, which was to have Google help with the encoding conversion by uploading it to Google Docs then downloading it again as a csv file.

Now I have a csv file using the correct character encodings. I ran it through my Python program and I got the JSON file I wanted. The next steps are to import it into my local MongoDB and start building a basic app that will query the data. We’ll see how far I go in terms of data visualization. I have another app that takes priority because I need to get it done by the end of this month.

ExpressJS & MongoDB: DBRef

Google search didn’t get me anywhere in trying to figure out how to use DBRef with ExpressJS & MongoDB. The only thing I really found was the MongoDB docs that showed how to use DBRef in the Javascript Mongo shell, which isn’t something you can use for ExpressJS. After failing to find a solution online, I went into the source files to see if I could find the answer. Luckily, I did.

DBRef is something I’m trying to use because I think it’s something important in making data relational. Why not use a relational db then like SQL? Simple answer: I just don’t want to. It’s a matter of personal preference. SQL is still pretty solid and widely used, but it just feels old and rigid to me. NoSQL like Mongo and Couch are fun to use, and right now I think that’s a really important thing. Unless there’s some critical, impossible to argue against reason to not use what I’m using then I want to continue using it because it’s fun.

Anyways, rant aside, the idea is that you have data in different collections that need to access each other. For example, you might have an app that manages employees across different store locations. Store information will be stored in one collection, and employee information will be stored in another. So what do you do when you want stores to contain information about employees working there, but don’t want to store the entire employee information? You make a reference to the employee doc. So within the store doc, you will have an object reference to the employee.

Continuing with our example, let’s say your store doc looks something like this:

{
  location: storeLocation,
  employees:
  {
    '$ref': 'employees',
    '$id': someObjectID1
  }
}

For simplicity, I’m keeping it at one employee. If you have more, then you just loop through the array. ‘$ref’ is the collection the referenced doc is stored in, and ‘$id’ is the object id of the specific doc you’re looking for. Next is the employee doc:

{
  _id: someObjectID1,
  name: 'Bob'
}

To get the referenced object, you do this:

dbRef = store.employees;
db.dereference(dbRef, function(error, result)) {
}

That will produce the referenced object (result). Hope this helps anyone trying to use ExpressJS and DBRef.

App Development: Layers

This post is a documentation and reflection of the development process I’ve been going through in order to build a web app. I’ll most likely be making more posts as well as I continue. This post will probably be a little erratic because I’m tired and the coffee shop I’m at is about to close.

The past few days have been spent doing a refresher on MongoDB, ExpressJS, and Node.js. I am far from being an expert in any of these, but I continue to push myself to learn more. ExpressJS is a framework for Node.js so I feel like I’m learning the least about Node.js itself. By no means am I someone who is formally educated in programming, but I have always been interested in how to build things. I took computer science for 2 years during high school, and spent the little free time I had with computers (more on that some other time) learning about web programming, so I have some basic knowledge of programming. I should probably spend time really learning everything there is about Node.js since I’m using ExpressJS, but my current focus right now is to build a functional prototype, so “coding correctly” isn’t a big concern of mine at the moment (at least, I’m trying not to make it a concern).

Why am I trying to learn MongoDB, ExpressJS, and Node.js? Building a web app is a goal I have set for myself, and learning about MongoDB has been fun. A friend of mine got me learning a bit about CouchDB, but the reason why I’m spending time on MongoDB is because the notion of “querying” makes more sense to me because of SQL. I’m not going to bother thinking about what technology is better. The important thing right now for me is to work with something that makes more sense to me – and have fun while doing it.

The past few days have also been spent trying to figure out how to go about building the app. I have been thinking about building this app for over a year so I had an understanding of what I wanted it to do, but I needed to scale it down to something more manageable and hash out the details. Luckily, the same friend who got me learning CouchDB also helped me scale my project down into something more manageable by asking me to think of it in terms of what can be done within a one month timeframe.

To accomplish this, I created a basic storyboard that outlined the core features that were needed. Once that was done, I created a data model based on the storyboard. I also created a basic system architecture in order to have a clearer understanding of what components I needed to build the app.

The process is iterative. The storyboard acts as a basic foundation, but as I worked on other pieces I came to realize that I needed to reconsider things like: Should I separate the data into their own collection? What are the components? What actions should this component be responsible for? I could spend weeks planning every little detail, but no plan is bulletproof so the important thing for me to remember was that I need to be sure I’m not missing anything critical at this point and then just start building it.

Then came another problem. I found myself going in circles wondering what I needed to start working on. After realizing I wasn’t being as productive as I should be, I stopped working and spent a little time thinking about why I was having trouble with getting started. I realized that I was trying to bite off more than I can chew. Specifically, I would start working on a component, come to some realization that I needed another piece to continue, work on that, come to another realization, and just continue down that rabbit hole.

To address this, I stopped myself and took the time to actually think about how to piece everything together. Looking at the data model, I realized that each component was embedded in another component. I started viewing each component as a layer that encapsulated another layer, and that helped me see what the base layer would be (i.e. where to start).

So now I know where to start building. It took me the entire day to realize this, but at least I have a clearer understanding of how to start building the app. On the bright side, not all my effort was wasted since I do have bits and pieces of each component built.

{localization:globalization}: Step 1 – Content

A friend of mine at Microsoft got me interested in localization. Through her recommendation (which also came from a localization PM at MS) I ended up getting a book on localization called, “A Practical Guide to Localization.” While this book is a little outdated, it provides insight into the localization process. Reading this got me thinking about localizing (or for websites, globalizing) my website.

One of the critical things with globalizing a website is separating content from presentation (i.e. code). It’s surprising how many sites you can come across that still have the content mixed in with presentation. To be honest, in order to get my own website up and running quickly I had considered just putting my content in with the presentation. Luckily, I decided against it and put my content in JSON. One of the biggest reasons why you should always separate content from presentation is so that people can edit or translate the content without risk of accidentally producing an error with your code.

With my content separated from the code, the next step is to start translating my content and prepping my site to handle other languages. I have decided to create a Japanese language version of my site since my background is in Japanese. I can think of two different approaches at the moment for organizing my JSON.

The first approach is adding a sub-dictionary to take into account the context. For example, since I have something like:

{“About”:”the content“}

I can adjust it to be something like:

{“About”:{“ENG”:”the content“, “JPN”:”the content“}}

This would give me more contextual awareness of the content. This is the approach that I am currently favoring just because I favor being more contextually aware.

The second approach would be to have the content organized more according to the language. So instead of the contextually aware approach, it would be more like:

{“ENG”:{“About”:”the content“}, “JPN”:{“About”:”the content“}}

While this approach loses the benefit of context, you eliminate redundancy with this approach. With the contextually aware approach, each section in my JSON that has translated content would need the sub-dictionary {“JPN”:”…”}. This is repeated throughout and could lead to scaling issues.

I think the first approach would be easier if you’re doing things on the go. Since I’ll be translating this myself, it might be easier for me to just translate each section into Japanese and add it in as I go along. The second approach would probably be easier if you’re taking a more systematic approach and have translators working on the content because they’ll end up just creating a translated copy of your current content, which you’ll probably end up receiving in chunks. Since you can’t really go live with incomplete translations, I think the second approach might be best.

The next thing for me to do is to go ahead and translate my content into Japanese.

Update: First I need to add in content for past work that I’ve done.