Digital transformation has accelerated significantly due to the COVID-19 pandemic, but the extra demands on data scientists have revealed significant barriers to effective working and high levels of job dissatisfaction in some areas. For example, around four in 10 are dissatisfied with their company’s use of analytics and model deployment, while more than 20 barriers to effective working emerged, according to a survey of data scientists commissioned by analytics leader SAS.
However, the work of data scientists has grown in importance with many organizations accelerating digital transformation projects by using technology to improve business operations. More than 90% of respondents indicated the importance of their work was the same or greater compared to before the pandemic.
To delve deeper into the state of data science, the report assesses the impact of the pandemic, challenges faced, overall satisfaction with the analytics environment and more. The research showed the pandemic upended standard business practices, shifting the assumptions and variables in models and predictive algorithms and causing a ripple effect of adaptations in processes, practices and operating parameters.
More than two-thirds of respondents were satisfied with the outcomes from analytical projects. However, 42% of data scientists were dissatisfied with their company’s use of analytics and model deployment, suggesting a problem with how analytical insights are used by organizations to inform decision making. This was backed up by 42% saying data science results were not used by business decision makers, making it one of the main barriers faced.
The survey also highlighted some specific skills gaps. Less than a third of the respondents reported having advanced or expert proficiency in program-heavy skills, such as cloud management and database administration. This is an issue given that use of cloud services is up significantly, with 94% saying they experienced the same or greater use of cloud since COVID-19 struck.
“There have clearly been more demands placed on data scientists and type of the demand started to show a lot of variations which in the end require different technical and business skills from them” said Celal Kavuklu, Customer Advisory Director for Middle East and Africa at SAS.
“A major source of frustration for organizations is finding a way to have sustainable and agile AI processes with enabling technologies. Analytical insights are big catalysts for digital transformation initiatives and having a sustainable and agile analytics center brings a clear edge in the competitive market conditions.”
“Linked to this, we found concerns around retaining the data science talent and a lack of skills in teams for the emerging critical roles like Data Engineering, ModelOps and DevOps, which has been an issue for some time with demand outstripping supply. SAS is continuously helping organizations with its advanced cloud-native technology, bringing more than 40 years of experience and by providing fully managed services. Also, we are enabling talent building programs to make sure there are the right skills in place to create a generation whose insights, efficiency and agility will be the driving force behind the business transformation” Celal added.
The research also identified gaps in consistent organizational emphasis on AI ethics, with 43% of respondents indicating that their organization does not conduct specific reviews of its analytical processes with respect to bias and discrimination and only 26% of respondents reporting that unfair bias is used as a measure of model success in their organization.
The research revealed positive outcomes from the global disruption of the pandemic. Nearly three-quarters (73%) said they are just as productive or more productive since the pandemic, while a similar proportion (77%) revealed they had the same or greater collaboration with colleagues. This suggests many of the challenges highlighted were in existence, possibly to a greater degree, before the pandemic.
Other challenges experienced were the amount of time spent on data preparation versus model creation. Respondents are spending more of their time (58%) than they would prefer gathering, exploring, managing and cleaning data.
Overall, the data scientist has ample reason to feel empowered and optimistic about how the pandemic has shone a spotlight on the importance of their role within their organization and how it might evolve over time. This holds especially true if data scientists can leverage the whole spectrum of available tools to manage the analytics life cycle, pursue data science training and skill development opportunities, and embrace data prep as the first step in modeling.
