19 March 2017
The data science profession is expanding and diversifying. Maybe this evolution hasn’t been so much about the actual work, as much as the appreciation for data centric skill sets. Data science is “sexy” nowadays, and that helps, too. People with job titles like data engineers and data scientists are in very high demand. For quite some time now, there has been a shortage of talent. Our profession is evolving. Data sources are proliferating. New platforms like Hadoop and other NoSQL sources have come into play. Analytics professionals need to be more versatile, and stay abreast of a bewildering array of innovations in platforms, data science workbooks, visualization products, and more.
The reality in the trenches is that in many commercial settings, getting your data in a shape and structure that makes them useful for advanced analytics constitutes the lion’s share of work. Very few data scientists will have the luxury of receiving cut and dried data sets, which are ready to be used “as is.” Not only do data scientists need to know about the sampling frame that led to the creation of the data set they are working with, they are often expected to play a significant part in generating (extracting) those data sets, too. Volumes of data are exploding, both structured and unstructured data, and they reside on a bewildering array of systems. Considerable technical computer science expertise is required to integrate all of these sources.
Because of the expanded skill set that practitioners need to bring, and the pace of technology change, you can anticipate that talent demand will continue to outstrip supply. At least for some time to come. Training and coaching of junior data scientists will be more important than ever. In software development a concept of “pair programming” was introduced (as part of the Agile methodology Extreme Programming) to facilitate knowledge sharing, and safeguard quality. Maybe similar practices could serve data scientists as well.
Universities all over the world are now beginning to offer data centric curriculums, even up to the Master degree level. As a new generation of professionals trickles into the workforce, hopefully they will drive further innovation. Given the rather diverse skill set that is required to excel in this field, it is anybody’s guess how our profession will evolve. One thing is clear to me: data is the great “equalizer” across the corporate hierarchy. Those who can extract meaning and value from it, will be in a place of power, will be able to influence tactics and strategy. Irrespective of job title or place in the corporate hierarchy.
Data scientists are at the frontier between IT and business lines. They may interact with research agencies and campaign managers to set up carefully controlled experiments. Data scientists can show where the profits are originating today, and where they are likely to come from in the future. Such findings can and should have a material impact on goal setting and corporate strategy. To be effective requires written and verbal communication skills, but also statistical, methodological, and technical knowledge to help design field tests. Increasingly, those experiments happen exclusively on the web. To find all these skills in one person is obviously a rare find.
My colleague Roelant Vos refers to the search for a five-legged sheep. Growing and leveraging this kind of rare talent is obviously a high-value activity. My experience as a manager has been that these folks often have idiosyncratic motivation. Typically less attracted to status and remuneration, but more by intellectual challenges, and the possibility of influencing strategy and impacting customer lifecycle management.
Systems and business knowledge are always important. Given that such acumen takes a little time to develop, it is not just about finding, but also about growing and retaining this talent. Nurturing a career path like this requires that you offer data scientists the right challenges, at the right time. It also requires considerable investment in education: typically, a multiple of “average” training budget is needed just to stay current. Even more if you want to grow. But the ultimate prize of breakthrough capabilities and innovative insights, must surely be worth it!