Anthony Scampavia interviews Ivaylo Lenkov, CEO and Software Architect at SiteKreator. Lenkov explains his team’s technique for doing daily releases.
Anthony Scampavia: Will you provide a brief overview of your company SiteKreator?
Ivaylo Lenkov: We started SiteKreator in 2002 for business owners with no knowledge of programming or web design to be able to build and modify their own websites. Recently we introduced the first SaaS platform for creating custom web designs without coding. We work with many designers who use a private-labeled version of SiteKreator to deliver complete web solutions to their clients. Our technology powers more than 100,000 business sites and we operate from 10 global data centers with hundreds of servers.
Scampavia: What made you re-evaluate your release cycle and abandon traditional software development release cycles?
Lenkov: When we started SiteKreator, our release cycle was almost a year. We were spending more than half of that time in stabilizing and polishing the release, because there have been hundreds of new features. During that time our customers weren’t able to see anything new and by the time we finally released the new version, many of the cool new features were no longer cool and some were not even needed. The web is a very dynamic space and the annual release cycle was limiting our ability to be ahead of the curve and to deliver innovative solutions to our customers. We started looking for ways to shorten our release cycle, by reducing the features going into each release. We first tried a three-month release cycle, then one-month, then one-week and we ended up with daily releases.
Scampavia: So how you accommodate long-term project planning while doing daily releases?
Lenkov: Well, that’s not easy. When you are focused on a very small feature-set, you sometimes lose the big picture. You just can’t see the forest from the trees. We had to change our planning process. We still maintain a product roadmap with milestones and everything but we chop each milestone into many small releases. This way every day we have a working product. We also review our priorities every couple of weeks and adjust them as needed.
Scampavia: How you actually do these daily releases?
Lenkov: Our team is distributed between two locations: one in California and another one in Bulgaria. This has many cons and few pros, one of them being that we can work in two shifts using the so called “follow the sun” model. Our engineers in Bulgaria finish the coding for the daily release around 11am PST and deploy on several staging servers. Then our product managers, marketers and usability experts based in California review the release and file their comments in our defect tracking system. After 11pm PST our engineers in Bulgaria come to work, fix the reported problems, if any, and push the release to the production servers worldwide. Sometimes we may skip a release if we need a little more time to fix something, but this does not happen very often. When you make small changes, it’s hard to break something big.
Scampavia: How do you ensure the stability of the release?
Lenkov: The daily deploy model would not be possible without a large-scale automated testing. We run a big farm of screen-capturing and comparing servers. We basically compare thousands of sites before and after the deploy on our staging servers. This takes few hours as we capture all possible browsers, both on Mac and PC, and in many screen resolutions.
Scampavia: Visually. So, you are doing just a Pitman comparison really.
Lenkov: Yes, but on a very large scale. The rule of thumb is that by doubling the number of sites we compare, the probability of catching new problems increases by one percent. But every increase in the number of sites also increases the number of false positives that has to be manually reviewed. So we tweaked our comparison algorithms to be more tolerant towards small differences in the screenshots. We still do human review of some screenshots that do not match, but at least it’s manageable.
Scampavia: What happens when you find a problem after the software has been deployed on the official servers?
Lenkov: This does not happen that often, but if needed, we have a single button roll-back to the last stable version. It takes about 2-3 min and it rollbacks all datacenters. But we use this for only really critical problems. For non-critical or cosmetic issue, we just fix the problem with the next deploy. That’s the advantage of having a daily release cycle.
Scampavia: How do you handle larger features that cannot be implemented in a day?
Lenkov: We implement the larger new features as modules, which are initially in stealth mode. We enable these modules only for specific users, so they can be tested. Of course we start by eating our own dog food. Then we enable the new features for users who have signed up as beta testers, as well as for our reseller partners. Once we feel confident with the stability of the new features, we begin bucket testing with real users. We start with a very small bucket (usually top 100 most active users), while keeping a finger on the Off button. Then we expand the bucket few times and at the end we enable all users. After a few weeks of running everyone on the new modules, we decommission the old modules.
There are two potential complications from this approach:
- If there are differences in the interfaces between the old and the new module, we need to create some throw-away glue code that “talks” to both interfaces.
- If there are backward incompatible changes in the database schema, we usually need to add a wrapper on the database level.
Of course this approach could not be possible without having tens of thousands of users using the software on a daily basis. Even if we miss a defect or a use-case, it always pops up during the bucket testing.
Scampavia: What do you use for version control?
Lenkov: Subversion in combination with Trac.
Scampavia: You have been doing one-day release cycles for two or three years now. Has it been worth it?
Lenkov: Initially, I thought it would be very inefficient but it turned out that shortening the release cycle resulted in reducing the time we spend on bug fixes, stabilizing the release from 50% to about 20%. Also, now we can introduce new features in a matter of days, while before it was taking months even a year for the smallest new feature to become publicly available. Our users very much appreciate that. In the SaaS world, the ability to show something new every day helps a lot in building a loyal customer base.
Scampavia: I want to thank you for your time and your thoughts. This is valuable to the SaaS world for people to realize that there is information they should at least think about for their methodology. You have put in much thought in this through the years. I will say that not all SaaS companies have done that. My experience is many companies are too busy to get to the market and do not consider the method. If they do not have the stability and methodology in place, things start to tear apart.
Lenkov: Thank you for having me. I am happy to share our experience and hopefully it will be useful to other SaaS vendors.
Scampavia: Lenkov, thanks for your time.