About the company
Fidelty Investments is one of the world's largest financial institutions, with total managed assets of over $1 trillion. Although best known as a mutual fund company, it has 80 other business units including retirement funds and investment management, with around 21 million customers holding more than 60 million accounts. Many of these are Internet-based, and 94% of the company's retail trades are executed over the web. Almost a third of Fidelity's operating costs are spent on technology, putting its annual IT budget at $2.2 billion.
While the web has enabled Fidelity to be very competitive, such dependence on the network is not cheap. The infrastructure supporting its 7,000 call centres and 3,000 connections to other institutions drove network spend to $200 million in 2004.
Historically, the cost of expanding and maintaining that network put strain on relations between IT and the business. End users were upset that their bills were going up, but the IT department could not persuade business units help them to find out why demands on the network were balloning. Fidelity decided that charging business units for network usage would be the best way to rein in costs but Bobby Lie, enterprise architecture chief at Fidelity, found that "to co-operate and be governed" was counter to the company's DNA.
Using network probe technology from NetScout to capture usage data and a package from Evident Software to generate the billing information, Fidelity has cut telecoms costs by $100 million – and improved business-IT relations.
Information Age (IA): As IT is increasingly delivered as a service to different business units, IT departments have tried different charge-back models. Fidelity has moved to a usage-based model, but how were business units charged for network capabilities before you introduced usage-based billing?
Bobby Lie (BL): I liked to call the previous model a buffet charging system : For $9.99 users could eat all the lobsters and drink all the champagne they wanted, no limit. Those business users were just gluttons and consumed the network like crazy. As a result I had to upgrade the network across the board – LAN, WAN, bandwidth. At the time the billing system was based on headcount – it didn’t matter how much you used, we billed you based on headcount. Allocating costs equally across the board is really the wrong model as it encourages people to be irresponsible.
Shifting to [usage-based billing] was revolutionary. In such a change, there will be winners and losers. The winners are going to laugh all the way to the bank and the losers are not going to be happy. In Fidelity, we have lots of very different businesses, and in implementing a market-based charging system we could potentially put some of those out of business as their usage costs shift [upwards].
IA: Given such concerns, how did you sell the idea to the business units?
BL: We had guidance from a lot of people. But ultimately [the move to usage-based billing] was made by the CFOs and the business unit CIOs. They all wanted it – even before they knew what the outcome was going to be.
We did a lot of internal socialising; that was key. We wanted to make sure they all knew what they were getting into. All of a sudden we had a visibility we never had before of who is using what, how much they’re really spending and why and what applications are consuming resources. People ought to have known that and have an opportunity to change behaviour. So as far back as 1997 we started to send out pseudo bills which read: “I won’t bill you for these services but if we were to bill you, this is how much you would pay.”
|
|||||||||||||
We set out a regular report, called ‘Top Talkers’, on the 100 biggest users in the business. We can show who is incurring costs by application, by workstation and by server. In almost all cases there were surprises – to the business units and to us. The list doesn’t tend to stay the same; it is different month to month.
We created a website for people to view their monthly report and to take action to reduce their traffic. We are now seen as the solution; not the problem any longer. The perception was that we used to send them bigger and bigger bills, now we are trying to reduce their costs.
IA: Rollouts always end up different to the original vision. What changed and how did you cope with it?
BL: Actually the initial implementation was at a relatively high level – at the business unit level. And they immediately said, “We can’t use this thing, we need to get to the departmental level, so we can really understand who is running up the network costs.” So all of a sudden we had to deal with 7,000 [of our] call centres. We had to do a lot of work to get there but the key thing here is the Top Talker reports. We can concentrate on that top 10% and worry about the stuff that’s really consuming resources. The biggest problem we had was we didn’t know what the heck we had out there. How do you associate the servers and desktops to the right call centres? How do you associate all the IP addresses to the right person?
There is no easy way to do it. Ownership is always a big issue. We over-billed the users a little bit so we had a bit of a reserve fund to help the ones that were really in trouble. Initially we gave people some rebate so they could fix things – if people have problems with their service we don’t want to charge them for it.
IA: What was the impact on the consumption of resources and the way IT can cope with that?
BL: Even right after we implemented the usage-based billing [in 1999], traffic didn’t drop right away. But from 2000 onwards, the network traffic stabilised. The good thing is the users are [still] doing what they want, but the usage of the network is much more efficient. We immediately found out all the problems we had due to errant applications. An application would go into a loop or send stuff out several times and generate a lot of traffic…we captured it all.
Over time, the rise in wide area network traffic slowed down but it still went up by 115% and the number of Internet external connections went up 400%. The number of devices on the network went up 27%. But the overall cost of telecoms to the company went down 31%. When you look at the growth of the use of network resources, this is a very significant reduction. In telecoms, we spent close to $300 million in 2000, but in 2004 that spend was closer to $200 million – a $100 million reduction even as utilisation went up across the board. The return on investment in the first year was 500% and the WAN costs stabilised over time. The savings here are recurrent year after year – it’s not a one time thing.
IA: Did the shift in billing model help to bridge that divide between business and IT?
BL: The key message is that this is not just about billing people for what they use – it’s about working as partners with users to optimise the use of the network so that everybody benefits. Normally that’s an adversarial relationship – one of vendor and buyer. For example, previously nobody paid any attention to server topology. We did a wholesale move of a lot of servers, even for the email system. People had put all the email servers in one location – but did they know how much it was going to cost employees in India using mail servers in Dallas? Before nobody cared. Now even the business units demand that application developers predict what the network [performance will be] before the introduction of an application. How wonderful is that? Some applications cost less to develop than to run, so it’s important to know that. Our problem now is we have too much business – people want to engage us. That’s a good problem to have.
IA: Could the fact that the IT department’s budget is so directly tied to business usage cause any problems if that usage was to fall dramatically?
BL: Clearly, I am not keen on usage dropping because we are billing our customers real money even though it’s internal. It’s never happened but if it did, I don’t have any problem sitting down with business units and saying with a straight face, “Hey, guys we did a great job cutting usage by 25% but your bill is going to go up 25%.” In fact, what I would do is to reduce the bill that I pay to the vendors so there is a reduction in cost anyway but people would still see a bigger bill.
IA: What other areas of IT can this charging model be applied to?
BL: It’s been so successful we’ve extended this to Internet usage and mail storage – another area where people just want to retain everything. Again in such cases we tell people to go ahead and use the facility but now they are made aware they are going to be billed for it.
Now we are extending the model to server usage as well, focusing on server utilisation. It’s very similar to a network charging model – a typical server runs at 10% to 15% utilisation [over any given month] and there is no motivation to do anything differently. So now we want to come up with a hosting model for applications that will allow them to run in a shared environment. In this way we aim to reduce users’ costs significantly.
