Q&A: RIM co-founder Mike Lazaridis

IT PRO spoke to one of the founders of RIM about the success of BlackBerry, the next level of integration with enterprise systems, rivalry and co-operation with Microsoft - and the truth about the recent outage.

By Mary Branscombe, 12 Jun 2007 at 16:47

gallery

When he was 12, Mike Lazaridis won a prize from the local library for reading every science book on the shelves. He dropped out of university before he finished his engineering degree to co-found RIM but he's stayed an engineer at heart; he's donated millions of (Canadian) dollars to found institutes in theoretical physics and quantum computing and he hates the idea of letting marketing dumb down innovation to make products simple. We caught up with him at RIM's annual Wireless Enterprise Symposium to ask if the system failure had damaged RIM's reputation and what the next big thing after mobile email will be.

What can you say about the BlackBerry system failing in North America in April?

It happened. First of all, we hadn't had an outage in two years, and it was going through hypergrowth, so it wasn't like the system had any issue from growth or scale. There was no outside influence that caused it.

Our first priority was to get people up, and once we had people up and running we had to make sure the system was healthy so it wouldn't happen again - because once you open the floodgates and let everybody reconnect there's a lot of email to suddenly deal with. Remember, we didn't lose any data. The next thing was to work with the carriers so their networks didn't get flooded by the build-up of connections. Once we got there, we went to try and figure out the root cause. We'd kept a continuous log of what was going on, so we could do a post mortem later. When we figured out what root cause was we posted it.

It was maintenance that was done on the cache system for the file servers for the database, to improve the efficiency of the cache, and it should not have been done on both at the same time.

Did you make any changes to internal procedures as a result of the outage?

Well yes. The biggest of them being "Don't optimise both the system and the backup at the same time." I should say, we don't normally do that. We ran our tests, it looked innocuous. All the tests we ran and the vendor ran made it look very routine. It turns out that there is a rare occurrence and under the kind of load we run our systems at that it can rear its ugly head.

Unfortunately, we were all confused over what root cause was so we were going down the wrong way to recover the system. No one was expecting that this was the issue. It wasn't a hardware failure, it wasn't a software failure. It was an intentional interaction of the cache based on the maintenance.

How negative were the reactions you saw from customers and potential customers? Do they have concerns about reliability? Or did the outage show them how much their business relies on email?

I don't want to put this too lightly. We were down. We went down at night. We were up and running before the morning load increase that we expect but we had a backlog that had to filter through that took many hours. The whole of North America went down and the systems are designed to bring back customers at a rate that doesn't cause excess load on the network. However, one of the things people learned, they looked back and said "when was the last one?" and we said "well it was two years ago". Then they started realising that is almost five nines - that's telecoms grade. It is just that people have become so dependent on their BlackBerry; it's become so mission critical - it's become a way of life.