Every company worth its salt wants to put together a Big Data strategy these days. Of course, to do so will require a capable team of both technical and functional experts, which can be challenging considering the fact that Big Data is relatively new in comparison to the more mature platforms out there. That being said, finding and building the right team is certainly one of the more challenging tasks involved.
Now let’s take a look at key considerations to build a successful big data team.
Have a Specific Business Goal, and Start Small.
This concept may seem obvious, but it is often overlooked. It’s very important question to answer; how your company should use Big Data? This is not to say you should cross any potential benefit off the list, but you do need to prioritize anticipated results. For example, search engine companies like Google are most interested in search behavior and user trends. Social media companies such as Facebook primarily track trends and user interaction.
That’s not to say Google doesn’t care about consumer interaction and Facebook doesn’t gather data on search behavior, only that both brands have primary concerns. And your company should, too. Developing the right team with the wrong focus will handicap your success. And having the right focus with the wrong team could be even worse.
Whether you are a data-driven company whose entire business model revolves around make optimum use of big data, or a manufacturer looking to optimize your operational efficiency, you need to be very clear about the challenge you are trying to tackle with big data.
Starting small to get into big data can be really useful, because you can get lost in a variety of technical issues such as integrations with Hadoop, Hive, or MapReduce. Prepare a team which is dedicated towards the goals of your company and achieving your predefined results. Offloading some of the smaller tasks to get your feet wet in the Big Data pool can help you prepare for larger projects down the road once you’re ready to scale up.
It’s essential to prove the value of your solution as soon as possible, which is to really pick the right problem and demonstrate very fast that you can find solutions to that problem. If you have found that something that will give you a competitive advantage and the technology is applied right, then the payoff can be great.
Hiring your Big Data Team
Big Data talent is in serious shortage. By 2018, the USA alone could face a shortage of 140,000 to 190,000 skilled people with deep analytical skills. While you may already have some personnel on hand that can adapt to your Big Data initiatives, it’s highly unlikely that you can properly finish your project without bringing in some fresh recruits.
While in some cases, a big data implementation can be done with one person or a very small team, generally having a dedicated, cross functional team will help ensure success. This is critical to make sure that business needs are understood and data is successfully prepared and accessible to meet the defined the business goals.
So what types of roles are often deemed “essential” for big data projects? While different companies will have different talent needs, here are some important roles to consider hiring:
1) Business Analyst
Your business analyst will be prioritizing your business requirements and translating them in to high level technical requirements. This doesn’t necessarily have to be a person who has “big data” written all over their resume, but they should have worked on some large data-driven initiatives (ETL will do just fine!).
2) Subject Matter Expert
Subject Matter Experts plays very important role in non-technical industries where the gap between a data developer and the business user can be very large. This role should be someone who knows your business and or industry very well, and can help bridge the technical and functional gap.
4) Data Hygienists
A Data Hygienist does exactly what the name implies: they ensure that your data remains accurate as it processes through your analytical systems. The data retrieved from a Big Data platform is only useful if it is clean and able to product multi-faceted analytics. Your Data Hygienist should have the attention to detail of a Quality Assurance professional, while still understanding business intelligence best practices.
5) Data Explorers
Data exploration to help realize and define which data has analytical value is invaluable in the Big Data world. These modern-day explorers analyze your data to find additional opportunities to capitalize on or help streamline your data gathering processes.
6) Business Solution Architects
This person will shape the data to ensure it can be usefully queried in appropriate time frames by every user. This role will define your Big Data architecture and roadmap, and is essential throughout the process since data streams and architecture are constantly changing.
7) Data Scientists
Tasked with creating analytics models for your data, Data Scientists are an important asset for predicting consumer behavior and build or update your data models accordingly. Usually this is an ongoing process that can even continue throughout the entire implementation and onwards into maintenance.
8) Campaign Experts
Data harvested from Big Data platforms utilized for marketing initiatives is quite common, and even if marketing may not be a primary focus, it would be wise to have a Marketing Campaign Expert involved with the process. Someone with an expertise in marketing will likely find other uses for the data being processed, and can help create models to prioritize channels.
9) Data Engineer/Software Engineer
Typically you data/software engineers are back-end/server side developers with experience implementing ETL solutions at a large scale, preferably with Big Data platforms. They should also have experience developing distributed or multi-threaded applications. It’s an added bonus if they have a background in statistics or machine learning as well.
10) Data journalist
Data journalism is not as commonly heard as many of the other titles on this list, but essentially they are tasked with analyzing the data and breaking it down into a “story”. Data that is mapped into a story helps other staff and consumers understand and conceptualize the data.
It is important to map the movement of data across the Big Data team and ensure that all data hand offs between humans and machines have clear owners. This mapping makes sure that each person in a given role is held accountable for total delivery, not just for completing his or her individual tasks.
Don’t Segregate your Technical & Functional Staff – Encourage Collaboration & Innovation
We just covered some of the important ingredients to a successful Big Data team, but remember that there’s no proven format to having a successful Big Data Implementation.
Creative thinking and innovative technologies offer an abundance of opportunities for mining additional analytical value from your data. Some of the emerging technologies available help automate, streamline, or reduce the technical barrier of Big Data solutions.
Look Towards the Future
Big Data technologies and strategies are ever-changing, so keep a keen eye on what’s going on in the Big Data realm. The next big thing could help save your company time, resources, and money.
Here are a few considerations that may help your cause:
It seems to be a trend that companies are running all or part of their big data environment within the cloud. As cloud computing on Big Data platforms and becomes more mature and secure, it may be a viable option in the near future or even in the present if it suits your business needs.
Regardless of if the end user is a customer or an internal user, you will need to keep track of insights created from your big data environment available for consumption both inside and outside your firewalls. With self-service analytics, business users will have readily available personalized reports without having to make additional requests to the IT department.
Whether it be a new integration to your Big Data stack or some fine-tuning to your techniques or processes, being forward-thinking can bring some great benefits to your environment.
Hopefully this article has provided some valuable insights to building your Big Data team. If you need assistance with finding the right assistance with hiring additional resources or are looking to outsource your Big Data project, please don’t hesitate to contact Aptude Consulting for help.