If you think you have a lot of data, processes and service delivery to deal with, think of the poor souls at internet giants such as Google, Facebook and Amazon.

Actually, don’t. Their engineers, most of whom sit on the sunny West Coast of the United States, are light-years ahead of the rest of the world when it comes to scaling IT to ensure capacity and performance. It’s something analysts now call web-scale IT.

These types of scalability problems haven’t been a problem for most organisations, and for the most part they still aren’t now – but that is all set to change.

With the rise of the Internet of Things – which will see conventional objects become smart devices through the attachment of internet-connected sensors – the digitisation of the enterprise, and mobility in the workplace, the need for web-scale IT will soon be a reality for the majority of large organisations.

>See also: Next big thing: Preparing for the Internet of Things in the enterprise

Gartner predicts that the availability of capacity and performance management skills for horizontally scaled architectures will be a major constraint or risk to growth for 80% of major businesses by 2016. As such, it anticipates that web-scale IT methods will be found in 50% of global enterprises by 2017.

‘Traditional IT technologies were not designed to manage the scale and cadence that operating at web-scale demands,’ says Roger Levy, head of products at SkySQL. ‘Moreover, the vast amount of data being generated drives the need for new ways to store, manage and analyse information.

‘With open-source communities, innovation can happen much faster, with the sharing of knowledge leading to the rapid development of services in response to customer demand’

Open-source technology is indeed one of the driving forces behind web-scale IT, with many important technologies coming from open-source communities and driven by companies like Google and Facebook.

Both of those firms, along with LinkedIn and Twitter, announced in March that they are working together to develop WebScaleSQL, an open source, web-scale branch on top of Oracle‘s database management system (DBMS), MySQL.

The collaborators have built a community structure to share and review WebScaleSQL contributions with the view of developing a more unified and collaborative branch of MySQL.

MySQL is the most popular open-source DBMS in the world, and competes vigorously with Microsoft and IBM in a relational database management system (RDBMS) market that has faced little challengers for many years.

That was until a surge of NoSQL upstarts, such as MongoDB and Couchbase, emerged with promises to tackle the same issue with cloud-based purposed-built databases. They will feel threatened by WebScaleSQL, just as much as Oracle’s RDBSM competitors.

Left behind

Many businesses now spend more on IT than R&D, which damages the design and manufacture of innovation.

Built on the cloud, web-scale IT now enables this to be corrected. However, those that get left behind on this will get behind very quickly.

‘On-premise software can make a lot of demands, though it does then yield idiosyncratic capabilities that may be the basis of competitive advantage,’ says Phil Lewis, solution consulting director, EMEA channel, Infor.

‘Cloud enables the industry-specific capability to be out of the box and may even them remove the customer idiosyncrasies.’

As well as allowing them to scale and grow rapidly, web-scale IT helps enterprises deploy new projects, apps and platforms at faster speeds.

This is a win-win for the customer and the business because it helps businesses to keep up with the fast-changing demands of the modern consumer.

Companies clearly benefit from cost savings, but more significant advantages come from flexibility and reduced time to market, as well as expanded revenue-generating opportunities. 

‘Organisations that do not invest in web-scale technology will find themselves at a noticeable disadvantage over their competitors, with slower operations, more downtime and reduced ability to respond to changes in their market,’ Levy adds.

State of play

Whilst this is largely a scene that will develop over the next couple of years, many CIOs in the UK have been keeping an eye on its progress.

In many ways, an early adopter market phase has already surfaced in the form of cloud strategies across the enterprise sphere.

‘Web-scale IT is really a way of describing what we call “true cloud”, vendors with the scale to deliver services to everyone, everywhere,’ says James Doggart, CEO, Cloud Technology Solutions. ‘CIOs are starting to see the benefits and economies of scale of web-scale IT, and the small number of vendors which can really deliver it.’

>See also: Taking mobility by the reins: the rise and fall of BYOD

According to Postcode Anywhere CTO Jamie Turner, adoption is growing in the UK with tech-heavy sectors like finance and pharmaceuticals already dabbling, but very few are making wholesale commitments to it.

For many, web-scale IT is an alien and drastic approach being met with fear and resistance. So the first question for any organisation should be, is it for them?

‘It’s not for everyone but if you’re starting out it’s well worth it,’ Turner says. ‘Initially though, it’s all about research and choice. There are so many products and potential environments to consider that it can be a minefield.

‘Fundamentally though, CIOs need to understand what they are trying to achieve then look for the software or service to deliver it.’

Ultimately, web-scale IT demands new thinking and a mind shift that CIOs must lead amongst their department.

Mostly, they need to move beyond the idea of ‘ownership’ – i.e. that the modifications and customisations on the systems are what truly make the system theirs or, indeed, effective. 

‘There is a huge amount to be gained by moving to off-the-shelf software that is truly industry specific,’ Lewis says. Around 80% of modifications are industry specific so by having those available out of the box, CIOs can free up teams of technical staff. 

‘It also then means that the CIO can be more prescriptive than reactive and actually partner the business, not just serve it,’ he adds.

Web-scale IT also demands new thinking about security. The web-scale efforts of criminals and hackers now costs the global economy £256 billion, McAfee predicts, at a time when security giant Symantec has admitted that anti-virus software only prevents 45% of file-based attacks at best. 

According to Glasswall Solutions CEO Greg Sim, the web-scale business world requires an approach to that is not based on keeping up with the bad guys and the constant updating of signature based products – but on what CIOs themselves decide is ‘known good.

‘Figures show that attacks such as Gameover Zeus, as with 94% of APTs, are mainly triggered by users clicking on malicious email attachments,’ Sim says. ‘And it is very difficult to remove this element of human curiosity or error, particularly in a web-scale world.’

Most importantly, CIOs must consider the ramifications on the network.

The advent of web-scale IT, which by its very definition is designed to deliver massively scalable real-time applications and services, puts far more emphasis on the performance capabilities of an enterprise’s existing web communications infrastructure.

Suddenly, they will be shifting very large amounts of data between nodes, so they need to work hard to make it work.

‘They also need to recognise that all the redundancy they have in the nodes is nothing if they all run off a single switch,’ Turner adds.

To fully realise the agility and scalability advantages of a web-scale IT architecture, it is critical to maximise bandwidth utilisation and reduce latency and CPU utilisation.

Together, these factors will impact hugely on overall performance when it comes to speed of applications and data delivery, as well as the degree of actual scalability.

‘Industry-standard HTML5 Websocket technology can help address these underlying network infrastructure and scalability challenges,’ says Kaazing CEO Vikram Mehta, ‘while also further enhancing overall user experience through, for the first time, the enablement of bi-directional, real-time web communication.’

Shaping the market

While web-scale IT has not taken the traditional IT vendors by surprise, and some have made creditable efforts in this area, the industry as a whole has still not quite managed to replicate all the capabilities of on-premise data management technology in the cloud.

For those vendors, the ‘cloudification’ of any services will be enough to drive a lot of defection or exodus, particularly as perpetual licences give way to per-use subscriptions.

‘The issue is now how to deliver value to all concerned within such a framework,’ Lewis says.

>See also: Remember the titans: who will win the in-memory battle?

The R&D resources of the large legacy technology companies should also put them in a good position to develop such technology, but the capabilities of web-scale IT are inherently suited to fast-growth and disruptive businesses born in the age of the web.

‘The incumbent IT giants will have to make an effort to adopt a little of that mindset if they are to succeed,’ says Levy.

Indeed, much of the web-scale approach is a response to the limitations of traditional technology, but as hardware gets faster and cheaper will this even be a problem in five years?

‘Assuming that the data explosion only increases then it’ll easily outpace Moore’s law,’ Turney says. ‘That means the importance of this approach is only going to increase.’