Tens of millions of data workers face productivity woes as complexity grows in data science and analytics, according to new research from IDC and Alteryx.

According to the report, The State of Data Science and Analytics, approximately 54m data workers around the world face common challenges associated with the complexity, diversity and scale of their organisations’ data. In an increasingly data-driven world, the term ‘data worker’ spans beyond the 54m identified in this study, but the findings are indicative of the challenges specific to those engaging in significant data work in their day-to-day jobs.

Random Forests and what do they mean for you?

What are Random Forests? What have they got to do with AI and machine learning? What have they got to do with you? And what do you need to know about random forests?

Key findings from the report include:

• Data workers spend more than 40% of their time merely searching for and preparing data instead of gleaning insights;
• On average, data workers leverage more than six data sources, 40m rows of data and seven different outputs along their analytic journey;
• On average, data workers waste 44% of their time each week.

“Data is at the core of digital transformation, but until organisation leaders address these inefficiencies to improve effectiveness, their digital transformation initiatives can only get so far,” said Stewart Bond, director of data integration and integrity software research at IDC. “Consolidating platforms and looking for tools that address the needs of any data worker, whether a trained data scientist or an analyst in the line of business, can help reduce the friction that many organisations experience on their path to becoming data-driven.”

DataOps: getting data right for DevOps

DataOps is an emerging methodology that combines DevOps with data scientists and data engineers to support development

A lack of collaboration, knowledge gaps and resistance to change were commonly quoted issues in the study around data science and analytics. Respondents also reported the lack of creative and analytic thinking, analytic and statistical skills, and data preparation skills as the highest-ranked skills gaps responsible for productivity issues, indicative of the pervasive talent gap that exists between data scientists and data workers in the line of business.

To overcome these data science and analytics issues, many enterprises are now hiring chief data and/or analytics officers to streamline their analytic processes, build a culture of data science and analytics; and encourage data literacy across the enterprise as part of their broader digital transformation strategy.

Data engineer: The ‘real’ sexiest job of the 21st century

As the hype around data scientists fails to meet expectations, perhaps it’s time data engineer took the title as the “sexiest job of the 21st century”

“Collecting data alone won’t digitally transform a business and the answer is not as easy as hiring a leader, a few data scientists or over-investing in disparate technologies. The key is to empower all users, many of whom are currently stuck in spreadsheets, to analyse data effectively to drive real, business-changing results,” said Alan Jacobson, chief data and analytics officer (CDAO) of Alteryx. “As the data landscape becomes more complex, this survey exposes the tip of the iceberg when it comes to the sheer volume of workers needing to conduct analysis on a daily basis and the untapped potential for them to drive meaningful business impact.”

The report is based on a comprehensive survey of more than 800 individuals performing data functions across geographies, industries, company sizes and departments. The full report can be found here.