There are many vendors and solutions for data visualisation for Business Intelligence (BI). How do you choose? This article focuses on 6 typical scenarios and discusses some of the possible solutions. For instance, SMEs should look for 3rd party BI solutions which can be customised and extended depending on their particular needs.
Intro
Through many years of experience, I have identified six typical situations where data visualisations for BI are needed. The right tools for the job are determined by how the data are collected/stored, who the end users are and, how they use the data afterwards. A strong driver for picking one solution over another is what other technologies or platforms are already used by the organisation. Data viz for BI is the final destination of a long chain of physical and business processes and hence, it has to fit around them often.
Example: DataBuildTool (dbt) is a popular tool for building data pipelines. Airbyte is a platform for scheduling the data pipelines in the cloud, which supports dbt as well. Lightdash is an open-source BI tool built on dbt and hence integrates really well with it. Any organisation heavily invested in dbt and Airbyte would consider Lightdash because of all the synergies between the three tools.
Typical Scenarios and Solutions
Scenario 1: The business wants to share business intelligence with internal stakeholders
In this case, typically there is an existing team of analysts who cleanse and manage the data and produce periodic reports. Hence, the main question is how to share the team’s findings with the wider organisation. A quick and dirty solution is creating static reports in Microsoft Excel and sharing the files with the relevant departments. This approach is simple and easy to implement. After all, MS Excel is well established and many people know how to use it. The main disadvantage is that it is difficult to scale up in terms of the number of reports and when the data volumes increase.MS Power BI is a BI tool which offers Microsoft customers easier access to databases and also better data visualisation capabilities compared to Excel. QlikView and Tableau are also well established BI / data analytics solutions on the market. Both have solid offerings ranging from a desktop version to full-fledged cloud solutions.For SMEs, these solutions can be costly given the relatively small amount of data they generate. Also like any feature-full solutions they have steep learning curves and require significant investment to embed in terms of skills and training. This is true for almost any software solution of their calibre. The exception to this rule is the MS Office products because they have been around since the 90s in one shape or form.
Scenario 2: Online (web) dashboards with access controls
This kind of scenario is common for SMEs who need BI for operational purposes. In this case the target audience for the BI data can be internal and/or external stakeholders. These companies often have some technical skills in-house but do not want to support a large IT department (unless it is their main domain of expertise). For example sharing Excel spreadsheets within a large organisation has its drawbacks like managing their lifecycle and stopping multiple copies of the same document from circulating around. Hence, companies opt for a web-only BI solution when they want a simpler, more streamlined approach.Typically, a BI team curates the data and maintains the dashboards centrally. Three good solutions in this case are Google DataStudio, Looker (owned by Google) and ThoughtSpot. All three offer an online platform which is capable of doing some data analytics, e.g. counts, sums, etc., and can nicely visualise the results. The final dashboards can be static or with minimal interactivity, because the data behind them is not accessible to the end user. This is an advantage because simplicity enables clarity which saves time and money; on the other hand some users may find this approach rigid especially when they are used to highly interactive tools.
Scenario 3: Custom data visualisations embedded in a product / web application
Custom data visualisations are an approach often taken by IT start-ups and companies. They have the knowledge and engineering skills to create their data visualisations through programming. One downside of this is the long-term commitment of these organisations to maintain the code. Hence, I would argue that doing everything in-house for BI is an overkill nowadays. For example, Highcharts is a licensed JavaScript library for creating sophisticated graphs and charts with little coding. There are also free alternatives out there, like D3.js and Echarts.An interesting offering comes from Bold BI whose platform is similar to Looker in terms of features including the option to embed charts or dashboards within an app or website. The twist is that they have a pricing tier for self-hosting their software (in the cloud). This approach is suitable for businesses who want to take advantage of 3rd party solutions but want to retain control over the application, architecture and data storage. Cumulo.io, GoodData and Metabase also offer very similar features to BoldBI. They have extensive free or trial versions for teams who want to experiment with the platforms. Only Cumulo.io has no option for self-hosting at this stage.Remaining in control of the data processing and storage is crucial especially for complying with data privacy laws. A typical problem arises when the data has to remain within the EU under GDPR but the servers of the BI platform are based in the US. Hence, being able to manage how the BI tools are deployed and hosted can be very advantageous.
Scenario 4: What about big data visualisations?
There are mostly proprietary solutions and the reason for this is that such solutions are relatively specialised for a given type of data or industry vertical. Netflix developed internally a platform called Lumen to facilitate BI reporting because of their specific requirements and scale – a large organisation like them can afford such investments! (In 2018 they published a blog post about this.)One particular challenge with big data in the context of BI is the danger of trying to display millions of data points on the same graph. One way to do that is using the WebGL technology, which platforms like Zegami employ to display data from large AI datasets.An end-to-end solution in this area, for instance, is Azure Synapse Analytics combined with Azure Data Explorer. It is a big data high-throughput analytics engine with some BI capabilities.
Scenario 5: We love open-source tools
For tech-savvy organisations, who have the time and people, they can opt for some open-source technologies:
-
- Lightdash: BI tool for dbt
- Plotly Dash: Python based library for making web-based visualisations which works well with Python servers;
- Kibana: visualisation tool for the NoSQL database Elasticsearch;
- Superset: visualisation tool for data held in SQL databases.
- Python Bokeh is a visualisation library which can be used to create dashboards with coding; it can be very useful for ecosystems relying heavily on Python.
The advantage of open-source is that generally the software is free and the underlying code is accessible making it easier for developers to find solutions to most problems. Large open-source projects are used by many people across the globe meaning that any defects are quickly reported and (hopefully) fixed. The downside is of course the lack of dedicated “customer support”; instead users can ask for community support in online forums. If your organisation runs into some weird problems with the software you can ask the project maintainers for a comment and support but there is no guarantee that the fix will be implemented soon. From my perspective as a solution architect,open-source is easier to work with because you can still reason about the problems and try fixing them; with proprietary solutions you are at the mercy of the vendor and they are not always forthcoming either.
Scenario 6: My company works with Microsoft/Google/IBM/SAP/and so on
If your company is already procuring services from some of the big vendors like Microsoft and IBM, they have end-to-end solutions for almost any use case. In this case, the synergies between the different services often determine the choice of BI tools. Conversely, the ease of integrating different services from the same vendor often saves a lot of time and money allowing customers to take advantage of as many services as possible. This approach though is suited mostly to large organisations who can invest in the adoption of these technologies, which includes a lot of hours for training and upskilling. NB! This is not an exhaustive review of all available BI tools and platforms. There are many solutions out there depending on the use case. The aim of this post is to provide an overview of what is possible. Often different providers offer similar solutions. The price models and extra services determine what you want. For example, good customer support is invaluable.As a side note, Microsoft has been the market leader for 14 consecutive years in analytics and business intelligence platforms, according to Gartner.
Conclusion
The above gives a brief outline on the types of BI / data visualisation tools that are most appropriate in six common scenarios. However, in reality, the choice of which tools to use is rarely driven by the tools themselves. For, any BI tool can display bar charts, scatter plots, pie charts and the like. What is as important as the choice of tool is the construction of the data pipeline that feeds the tool.So beyond the visualisation capabilities of the tools, the main drivers when choosing the right tool for an organisation are in fact questions like:
- Which tools connect seamlessly to our database?
- How can we enforce access controls without asking users to create yet another log-in profile?
- Which tools can query and visualise comfortably our type/volume of data?
- Would it be easy enough to use or we need to send half of the company on a training course?
- Can we provide the right access rights to users based on our business rules?
- Can we guarantee the data staying within certain geographical regions?
Hence, at Jumpdata, we implement the best solution for your organisation based on this information.
