The RefLog – July 2012

In this edition of The RefLog, we talk with Bryan Helmkamp. A Rubyist, GitHub User #19, and creator of Code Climate. Matthew McCullough sat down with him to talk about what Code Climate is and what its future may hold, especially for open source projects hosted on GitHub.

Matthew McCullough: When did you start formulating the ideas that became Code Climate? What are some of its major milestone dates in addition to the recent commercial launch?

Bryan Helmkamp: I had been interested in static analysis tools like Flog and Flay for years, but could never find a great way to work them into my flow. At the same time, I’ve worked on a number of large, long-running Rails projects and observed that even excellent developers with great intentions continued to struggle with keeping code quality high over the long term. These were high functioning, Agile teams that used techniques like pair programming, and so I started thinking if static analysis could help.

Looking at my Git log, the first commit was on March 6th, 2011, so it’s been about a year and half in the making. The first program I built just counted the number of lines of code in different layers of your app (like rake stats) and charted it over time. It was cool and pretty but not that useful. This past March, about a year after I started building the product, I left my full time job to focus on Code Climate. The company is 100% bootstrapped, so that was a big milestone.

MM: Was there any inspiration from products like Sonar for Java?

BH: Most of the inspiration was from metric_fu. It did some things really well, and I thought, “What if this data was available on a hosted website, and automatically kept up to date?”. The vision today is much larger than that, of course, but that was the start. Eventually I found Sonar in my research of the space. I’ve never run it myself, but it seems to be very detailed and comprehensive. So there’s good inspiration there, but Code Climate is focused on being as actionable and easy to use as possible. That focus has led to more simplicity in the user experience than what you see in Sonar.

MM: What made you realize that there would be a market for this? What would make someone want to pay for this?

BH: What I started to do, when I felt like there could be something useful to me there, was putting together some wireframes for a “first cut” at what I thought an experience might be like. At that time, the idea that I was playing around with, just as something super simple, was just email based with no web frontend at all. It was like, what if you got an email that looked like this, every week and it had some information about how the code base had changed. I put that together in Balsamiq mockups and started asking people who I knew in the Ruby community for a few minutes of their time when I ran into them at things like conferences and whatnot. I would sit down with them and get their take on it and ask them questions about what they were currently doing to manage quality.

I also did a survey that I sent out over twitter asking people if they used static analysis tools and about their experience with that kind of thing. It was a kind of qualitative and quantitative research process I went through to get the confidence that there was an initial foothold. It was also comforting that the product is completely bootstrapped and that I don’t need there to be a huge market for this business to payback VC investors or anything like that. I was really just trying to do something that I thought would be interesting and useful to me and people who had similar experiences. So, yeah, I did a lot of that stuff before really building too much of the product.

MM: Do you feel the feedback helped to shape things dramatically by correcting assumptions you had about what you were building or was it more of a confirmation that you were on the right path?

BH: I’d say somewhere in between. I think the core is still very similar to what it was in those initial wireframes using emails though much of that has migrated onto the web, at this point. There are still emails, but they are slimmed down a bit, partially because it’s easier to implement things in a website than a HTML email.

MM: (Sarcastically) Wait a minute, you mean supporting different email clients is worse than the browser wars?

BH: (Laughing) Yeah, much worse. Fortunately, I don’t have to support very many browsers and email clients.

So that part was fortunate and a lot of the feedback lead me to really focus on clarifying the user experience. One of the biggest improvements to Code Climate, based on the feedback I’ve heard since launch, was the introduction of the rating system. That’s something that is unique, as far as Ruby static analysis, to the tool. Code Climate gives every class and module a rating from “A” to “F”. Those ratings are an aggregation of different types of static analysis information and they’re really aimed at making this sort of digestible grade, just like you got in school, for each part of the code base. That really unlocked a lot of interesting things you can do, cause now you can look at this high level of which grades changed. What are the things that are currently failing verses what are the things that are in a passing grade. So, that was the biggest change that the product has undergone since launch. When it launched, it didn’t have that and it was all just numbers. So, it would say, for example, “complexity is 37 and your duplication is 46”. It will still tell you those things but the grades are the focus now.

MM:The slogan is “Code Climate is hosted quality metrics for Ruby apps.” Are there any plans to go beyond Ruby?

BH: I’m focusing on Ruby first because I’m a Rubyist at heart and there’s a great set of open source static analysis tools available to build from. Code Climate stands on the shoulders of those tools. It will support other languages someday in the future, but right now I’m focused on making the experience excellent for Ruby. To that end, there are more Ruby/Rails-specific code insights coming soon.

MM: What’s the category of user or project that should be running Code Climate? Newcomer to Ruby? Veteran? Small project? Large project?

BH: I know people and projects that meet all of those descriptions that have found benefit in Code Climate, but the sweet spot is definitely above-average teams maintaining Rails applications over the long term. Beyond a certain code base size and team size, the natural direction is for an application to become less maintainable over time. Code Climate is designed to help teams change that.

The idea with Code Climate is that it should be approachable by anybody on a team, at varying skill levels, to help the whole team produce better output. Generally with teams, you’re going to have a mix of your senior engineers and also more junior engineers. Often what I see is a more senior engineer or hands-on developer-manager role is the person who sets this up, but the data and information is being made available and communicated to the whole team and you want everyone to be able to act on it.

MM: Would I be accurately summarizing in saying, if someone had limited time, Code Climate grade letter score and color coding is meant to allow them to make some improvement even though they are unlikely to bring everything up to an “A”?

BH: I think that’s a good paraphrase. Except that I would add that I would never recommend that anyone ever try to get all “A’s” in Code Climate. The guidance that I like to give around Code Climate is that it’s not a comprehensive to-do list of things that you need to fix. Everything that Code Climate identifies isn’t necessarily something that needs to be worked on. So, what I would really focus on is not having classes that are failing. So, no “D’s” or “F’s”. That’s the way that I like to use the tool. The way I do that is two fold. One, don’t let classes start failing. If things are in decent shape, keep them in good shape. If it’s a “C”, that’s fine, just don’t let it cross that line. Then, if you’re working on classes which are failing right now, try to follow the “Boy Scout Rule” and try to leave things a little bit better than you found them. Eventually, you’ll end up with everything getting passing grades.

MM: Are you actively trying to find other channels to promote, such as at conferences, to get more open source into Code Climate. Do you feel this would sort of “lift all boats”, so to speak, in the open source community if there was significant number of projects using Code Climate?

BH: Open source projects have been in a sort of private beta, so far. However, with this interview I’d like to get as many open source projects on it as are interested in using it. Conferences are also going to be a big part of that, for sure. I’m interested in any channels that are good ways to reach people.

MM: Are there any very interesting discoveries that you’ve made by running Code Climate on an OSS project? Any particular ones that have had a positive reaction to it?

BH: While the OSS support is pretty new, the reaction to it from the OSS community has been very positive. It’s great to see that so many project maintainers are looking for ways to ensure they ship high quality RubyGems. This is also apparent from the rapid uptake of Travis CI.

The team building Data Mapper 2 was one of the earliest OSS adopters and they’ve been using it to refactor as they go. I don’t recommend anyone strive to get all “A”s in their Code Climate ratings, but Virtus, a DM2 component, does.

MM: Do you feel like the infrastructure is well prepared for an influx if say, a thousand new projects in the next week signed on? Is it ready to take that on?

BH: Yeah, I think it will be fine. The nice thing is that the web requests don’t actually do any processing. So, its just a big Resque farm that records things into Mongo and then pulling out pages is pretty straight forward. It is pretty resilient to the number of projects that get added, so I’m not concerned.

MM: What’s the next area of enhancement focus for Code Climate?

BH: I try not to comment too much publicly about functionality that’s not ready yet, but I can give you a sense of what I’m thinking about. I’m going to keep working to make it as easy as possible to integrate Code Climate into your day-to-day development process. It’s not enough to have metrics and activity feeds on the site and send weekly summary emails. So, it’s going to do more to guide teams through the process of getting their code base in shape over time and keeping it there.

We hope you enjoyed this edition of The RefLog! If you have any suggestions for future projects or communities that you’d like to see highlighted in The GitHub RefLog, tweet us @githubtraining.

The RefLog Team,
Jared, Kami, and Matthew