Benefits and Challenges of Using Microservices with Big Data Applications

  • microservices_big_data.jpg

While implementation of DevOps practices has been on the rise lately, a slightly lesser-known concept has also been gaining popularity – microservices. The idea behind microservice architecture is to build your application as many independent services rather than one large code base (commonly referred to as a monolith). Rather than accessing the majority of your data using large databases, communication is often handled with API calls between the services, with each service having its own lightweight database.

Microservice architecture doesn’t necessarily mandate any more than the core concepts mentioned above, but there are many other best practices that help facilitate better integration across many facets of software development lifecycles. The primary practices associated with microservices include containerization, continuous integration, DevOps, automated integration testing, and – in many cases- the Agile development methodology.

When implemented properly in combination with these best practices, implementation of microservices can deliver many benefits to your project’s application, including:

  • Easier deployment process
    When deploying updates to a particular feature, the entire application does not need to be re-deployed – just the service that you want updated.
  • Use of varied programming languages and technology stacks
    Microservices give you the ability to use different technology stacks across your microservices – want to use Java and Scala? Go for it! Additionally, this makes it much easier to migrate to different technology stacks than with a monolith application.
  • Better failure detection
    With microservices, it is easier to monitor and detect when one of your services has an issue. Of course, this requires that the right software be in place to monitor said services.
  • Enhanced continuous integration and deployment
    Spreading your application across many different code repositories makes it easier for developers to constantly push changes to their microservices, and also makes for easier automation testing.

There are several different technologies and platforms that help enable microservice architecture to succeed. Here’s a few of the important ones:

  • Containerization Platforms
    Moving between developing features on many different microservices locally would be a nightmare to setup, but thankfully platforms like Docker and VMware vSphere allow you to deploy images on your computer locally, regardless of what operating system you use.
  • Deployment Automation Tools
    Automation software like Jenkins, Puppet, and Chef pair well with microservice architecture allowing you to achieve true continuous integration and deployment.
  • Automated Integration Testing Platforms
    There are way too many integration testing suites available to list here, although Selenium is probably one of the most popular choices. These platforms play a very important role in Continuous Integration and microservice architecture (and for any application, really).

Say Hi. Don’t Be Shy.

Looking for a Big Data or Business Intelligence Solutions?

Aptude Consulting, a Big Data Systems Integrator, brings over fourteen years of Middleware and Business Intelligence experience to the world of Big Data.

As a Cloudera, MapR, Hortonworks and Oracle Gold Partner, Aptude synergizes rich, practical data warehouse experience with our partner offerings to bring a modern solution via a trusted, experienced Big Data Systems Implementer and Integrator.

Contact Us

There are also some technologies related to Big Data systems that integrate well with microservice architecture:

  • Messaging Systems
    Lightweight messaging is required for microservices to succeed – especially with the significant amount of API calls being made. Commonly used platforms include Apache Kafka, Amazon Kinesis, and Rabbit MQ. While not necessarily a Big Data technology, these systems are commonly used with Big Data platforms.
  • Orchestration Systems
    Platforms such as Kubernetes and Apache’s Mesos distribute workloads across distributed computed platforms (such as Apache’s Hadoop).
  • Cloud computing with Big Data
    Services like Amazon’s AWS and Microsoft’s Azure have really been leading the way in cloud computing solutions geared towards Big Data solutions, allowing for ease of deployment and scaling of resources – which is essential for microservices to realize their full potential.

Benefits of using microservices with Big Data systems

Aside from issues with data visualization and the functional challenges of planning and implementing Big Data solutions, microservices can help address – to some extent – some of the hurdles companies often face. Here are a few of the most important issues related to Big Data initiatives, and how microservice architecture can help out.

(continued after break)

Keep Moving Forward with Aptude

Aptude is your own personal IT professional services firm. We provide our clients with first class resources in a continuous, cost-containment fashion.

Our support services will free up your senior IT staff from the overwhelming burden of day-to-day maintenance issues. Now they’ll have time to launch those new projects and applications you’ve been waiting for. Simply put, we can free up your resources and contain your costs. Let’s have a quick chat to discuss our exclusive services.

CONTACT US

(continued)

Application Scalability

One of the biggest benefits of using microservice architecture for Big Data applications is the scalability it provides.  While not necessarily synonymous with cloud computing, it is relatively common that the two are utilized together. A traditional monolith application doesn’t have the flexibility of applications built on microservices. With each service running independently, the servers they reside on can scale up and down in resources as needed. This is especially important for Big Data systems, which tend to be resource hogs as they handle data at a high volume and velocity.

Data Quality

As data flows through the ingestion process, there are many points of failure that can occur, and data quality is one of the biggest challenges associated with Big Data initiatives. While data quality is a multi-faceted issue, including many functional aspects, microservices can also help in this regard.  Software development teams that are creating the microservices have a much more focused task at hand. Since each service has a focused task, it makes it easier to create, maintain, and test services as they flow through your applications ingestion and transformation process.

Talent Allocation and Acquisition

As mentioned previously in this article, microservices allow for using various technology stacks across your services. This also allows you to staff your software engineering teams accordingly.

There are many different approaches to architecture in regards to how work is distributed across teams, but having cross-functional teams is more common in applications with microservice architecture. Since the services are not as complex as a monolith application, they are much easier to understand, making it easier for developers not familiar with the code – or, in some instances, the programming language – to add features or modifications.

With the ability to use a variety of programming languages and technology stacks, this also makes the talent pool of available hires significantly larger. Of course, you wouldn’t want your technology platforms to become too fragmented – usually companies will limit their choices between 2 or 3 main programming languages and associated technologies.

Challenges with implementing microservice architecture

Of course, all the benefits of microservices do come with some caveats.

Additional Staffing Needs

Although microservices have the benefit of being able to reach into a more diverse talent pool of developers, there are additional roles that need to be filled. The added layer of complexity of utilizing microservices in coordination with Continuous Integration requires some serious DevOps talent, which can be hard to find – which is compounded by the fact that it’s already hard enough to find top Big Data resources. Additionally, we’ve been building monolith applications so long that not everyone is familiar with the architecture, which complicates the situation even more if trying to implement microservices in an existing monolith environment.

Added project complexity

There is also a great deal of cross-team coordination required to ensure that redundant services are not being built, while also making sure that data formats are consistently maintained while communicated across the services. This issue is particularly challenging to address, since teams working on isolated code repositories often will have limited knowledge of the other services being developed, especially with larger applications. One way to overcome this challenge is to divide the workload into small vertical slices, and have teams gain exposure to different pieces of the application over time. This approach does require programmers that can develop the full stack – both front-end and backend.

Project costs

Another consideration is the additional overhead of the architecture – costs are likely going to be much higher with running many microservice instances, including the cost of servers, database licenses (if applicable), and finding the right talent to build the services.

Is the microservice architecture right for your environment?

One of the reasons microservices could be potentially more compelling for a Big Data initiative is that many of the projects are ground floor opportunities – Big Data has been around for a while now, but many projects are new initiatives. Implementing microservices into an existing project would be complicated and expensive, and it may be hard to justify the cost of the undertaking.

Like many relatively new concepts, microservices are not universally accepted as the right architecture for their technology environment. However, the architecture has been successfully implemented by many big tech-savvy organizations, including Uber, Amazon, Netflix, and Ebay – to name a few.

It’s common knowledge that Big Data scalability is a major concern – you can only throw so much hardware at a problem before poorly planned architecture makes it become unmanageable. As with many technology implementations, over time the systems gain added functionality and especially in the case of Big Data, higher velocity, variety, and volume. Adapting a microservice architecture could help offset these challenges in the future if implemented properly.

Though implementation of microservices is challenging, many would agree that the benefits outweigh the additional cost and application complexity. At the very least, if you’re starting a new Big Data initiative it would be worth your time to consider analyzing it as a viable option.

Ready For What’s Next?

Looking for a Big Data or Business Intelligence Solution? Looking for consultation on to handle your Hadoop and/or MondoDB projects? Contact Aptude’s team directly.

Guy DeRosa

Senior Vice President & Director of Operations
EMAIL GUY
p: 630.692.6700 ext. 406

Srinath Parepally

Vice President of International Development
EMAIL SRINATH
p: 630.692.6700 ext. 402

Uday Mehta

Vice President of Development
EMAIL UDAY
p: 630.692.6700 ext. 403

Ravi Das

Sr. Business Development Manager
EMAIL RAVI
p: 630.692.6700 ext. 409

Gain Time, Increase Currency, Contact Us

It’s amazing how one quick email can change your life. Give us a shout! We’ll get back to you right away with the right person for what you’re looking to accomplish.

You don't have to go into a lot of detail now, we'll dial it in later.
=
What our clients are saying…

schneider-national

Aptude provides onsite and offshore Oracle DBA support, which includes troubleshooting, back-up, recovery, migration, upgrades, and daily maintenance of Oracle database servers. Aptude has been working with our team for the past four years and we continue to use them and are satisfied with their work

Warren E., Schneider National

build

Aptude provided Build.com a Java, MySQL, Webservices and other UI based solution in the business domain of analyzing and reporting on user activities for our ecommerce website. Utilizing Omniture’s APIs to download, parse, and regenerate and upload back so that we could be more effective in our marketing. I was satisfied with their project work and delivery and would consider utilizing them for future projects.” Build.com

Dan D., build.com

ugahead

Aptude provided us with Oracle DBA migration support, including an upgrade from Oracle 11.1 to Oracle 11.2, and the project was completed on time and to specifications. The project manager and project consultants were responsive and proactive, resulting in a successful conclusion to the work. I would definitely contract with them again, and have recommended them to other technical offices at the University of Georgia.

David C., University of Georgia

wolters-kluwer

Thank you for the hard work your team has put forth to staff the contract positions at Wolters Kluwer. Aptude has consistently scored high in our supplier carding and even more important you are a vendor we can always trust. I am especially impressed with your ability to tackle our positions that other vendors have not been able to fill.

Andrea G., Wolters Kluwer
By |2018-11-21T17:26:18+00:00November 1st, 2016|

About the Author:

It's What We Do