In our last blog on BI trends to watch we touched on the exciting prospect of Natural Language Processing (NLP) becoming a tool for building queries in business intelligence. Advances in NLP are giving rise to potentially groundbreaking technology, and the subject deserves more than a cursory glance. In this blog we’ll explore some of the pitfalls, challenges and barriers that have kept NLP out of BI, as well as some of the exciting possibilities of its application as BI developers begin to integrate NLP into BI.
If search engines like Google have been using NLP technology for the past couple of years, why is BI only beginning to make use of this incredible tool? Clearly it’s not for lack of interest, demand, or potential applications. Bridging the chasm between human language and machine language is far more complicated than one might imagine. Computer scientists have been racking their brains over this technology since the middle of the last century and with every new innovation a new challenge was encountered. But with advances in deep neural networks, deep learning, and various machine learning models, NLP has improved enough to support commercial applications, including improvements in BI functionality and usability.
Common Language Analytics: One application of NLP already in practice in BI, as mentioned in the previous blog, is the translation of analytical results into common language to make the information more accessible to a broader audience. Narrative Science has bridged their Quill platform to work on Microsoft’s Power BI through their API, Narratives for Power BI. The Natural Language Generator (NLG) translates visual representations of analytical output into descriptive text. The generated text can be customized to meet the needs and preferences of the user. Adjustments can be made to verbosity—to make the text more simplified or illustrative—and the user can choose to view the output as a bulleted list or a continuous paragraph.
This may seem like a superfluous function to data nerds, like me, who have been trained to interpret charts and graphs, but it’s quite valuable. While not everyone has that background training, everyone in an organization brings an important perspective to the table. This feature allows access to information to individuals across a much broader spectrum of educational and training level, and learning style. It even has the potential to assist individuals facing disabilities—like visual impairments and visual processing deficits—to have a novel way to interact with information. With voice-to-text software, an individual with no eyesight can now have an audio representation of a pictorial analytical output. NLG provides access to data that can improve decision making to individuals at all levels, across an enterprise.
Common Language Queries: A second application briefly mentioned in the previous blog was the use of NLP as a means to translate common sentences into usable queries. In a piece for wired.com, Stephen F. DeAngelis stated,
“Most analysts appear to agree that the next big thing in IT is going to involve semantic search. It’s going to be a big thing because it will allow non-subject matter experts to obtain answers to their questions using only natural language to pose their queries. The magic will be contained in the analysis that goes into the search that leads to answers that are both relevant and insightful.”
We all do this regularly when we talk to the virtual assistants in our mobile devices. We don’t need a special language or structure when interacting with Alexa, Cortana, or Siri. We speak, they listen, and more often than not the program works correctly and delivers the requested information. The simplicity makes the feature available to any user the instant they interact with the device, without any special training.
That accessibility is precisely what makes natural language query builders so valuable. An individual no longer needs to know the right language to communicate with the analytics software. End users will have ever more access to information, and can offer novel insights on data analysis.
Wizdee is one company offering Natural Language Understanding (NLU) to translate text or voice prompts and questions into machine language. The company asserts that with their software “business intelligence becomes a part of every team member’s skillset.”
Unstructured Data: Another exciting use for NLP in BI is to make use of the vast amounts of unstructured data that have been made available by the ever increasing rise in social media use, on-line reviews and news, and IoT enabled devices. Allisa Lorentz, VP of Creative, Marketing and Design at Augify, gave the following explanation.
“We’ve become extremely proficient at collecting data – be it from enterprise systems, documents, social interactions, or e-mail and collaboration services. The expanding smorgasbord of data collection points are turning increasingly portable and personal, including mobile phones and wearable sensors, resulting in a data mining gold rush that will soon have companies and organizations accruing Yottabytes (10^24) of data.”
Until recently we had no way of searching, sorting and analyzing all this text data for content and correlation. Lorentz offers that the information locked in unstructured data provides context to our structured data vastly increasing its value. NLP is opening up this wealth of information for analysis by the powerful BI tools already available.
In December 2016, IBM announced its Watson Discovery Service, which allows users to find, standardize, and analyze unstructured data. IBM’s Luke Palamara described the importance of this innovation as follows.
“The analysis of structured content – numbers, dates, organized groupings of words, which tell us WHAT is happening – has been largely conquered with traditional analytics systems; however, the analysis of unstructured content presents continuing challenges. But it’s precisely unstructured content, like product reviews, social media and images, that tells us WHY things are happening.”
And that analysis of unstructured data—finding the ‘why’—is precisely what Watson Discovery Service, and others who follow, can provide to BI users.