How I Built a BI Platform from Scratch without committing suicide !
PART 1 - ENGLISH
Who am I first?
First of all, Hi Astronaut!
I'm Orbit Turner (Mohamed GUEYE of my real name) I'm a young passionate of technology, car and design.
I started to touch my first lines of code when I was 11 or 12 years old and since then I followed a lot of Roadmap in self-taught but also through my various trainings before ending up becoming a Fullstack developer in late 2020 then DevOps currently.
BACKGROUND
It all started in early 2021 when I was contacted by a French company based in Senegal that was looking for a Fullstack developer who could meet their internal needs.
After a long and usual process as well as some weird tests I was able to join the ship which has more than 17 years of existence and nearly 600 employees as well as a huge and complex IT infrastructure.
My initial mission was to propose a tool capable of tracking and managing the workflow of the department in charge of one of the company's clients.
It is important to understand that the company is an outbound call center, i.e. a B2B customer contact center for telesales, surveys and appointment setting. This means that the customers are often large French telecom companies or others who require metrics (Call statistics (DMC, number of contacts argued, positive contacts, rate of completion, sales / cards delivered, penetration rate, average waiting time, first call resolution, ...): in real time, by hour, day, week, month and by telephonist / team, partial Tris, monitoring of KPIs and quotas during the field, Editing and sending of the field follow-up - Daily automated reporting) and a very specific and intensive workflow.
From the moment I became aware of the requirements and once my integration was complete, the impostor syndrome set in and I said to myself: "well, brother, you're up to your neck in shit!
THE PROBLEMS
Ah well there were many 😨!
The company for most of its tools, is on Python 2.
The company has a plethora of Database Servers with about 50 databases in each server as well as millions of equally sensitive records.
The company wants, in addition to automating some manual processing, to have total visibility in real time of what is going on. Sales, call times, badged hours, profitability, forecasts, etc. This means extracting, transforming and visualizing very clean reports from millions of lines of information scattered around in less than a second.
The company calls it a "Tool". 😭😫
TO TOP IT OFF, I WAS THE ONLY DEV 👨🏾🦯
FACE THE UNIVERSE
I had one week to come up with a plan of action and the "Thinking' of a solution.
My impostor syndrome told me to do like all those who came before me, that is : GET THE HELL OUT OF HERE!
But the sadistic part of me, the one I call "Shadow", showed up with a narcissistic smile and told me: "You know very well that you like this kind of thing. You only live once! So don't even think about it. Just go for it and bust your ass with pride". 😫
Yes, like an idiot, I jumped in head first!
The first two days like everyone else I was already in my office thinking about entities, how to manage multiple DBs without blowing everything up, Tech Stack, deployment, Docker, microservices, Big O Notation etc.
But, like every time I face a new big challenge, I took the time to step back and think outside the box.
THINKING OUT OF THE BOX!
I have always been someone who likes to go fast by taking shortcuts or using what already exists. I always refused to do something twice because I automated everything that could be automated.
It's simple isn't it? NO ! It's not that simple because over time, projects and people you meet, you end up detecting quickly those who are made to solve problems and those who are made to code. For my part, here is how I was able to deal with my problem.
REDEFINE THE NEEDS!
A project is always the definition of someone's need and when someone tells you what he wants it is obvious that he will say whatever comes to his mind.
The difference between a dev project for a company and a dev project for a customer is that often the company wants, as they call it, "an all-purpose tool" that allows them to work and manage their various operations but the difference is that this tool often has no delivery date or final version. As long as the company lives the tool will grow.
So the crucial step was to say to our CIO: "*Okay cool! I see what you want from me. Now, where do we start?".
It sounds trivial, but redefining requirements is first of all breaking down the requirement into smaller blocks and then organizing them by priority. Yes, it has a name, it's being : AGILE!
Then the next step was to review, with great hindsight, the evolution over the long term and to deduce the nature of what will be achieved.
It is necessary to know that according to the first requests I believed that it was a ERP. So even if I knew what an ERP was, I had to redefine it clearly and compare it to the needs.
According to Oracle an ERP is:
A type of software that companies use to manage their day-to-day operations such as accounting, purchasing, project management, risk management and compliance, and supply chain operations. A complete ERP suite also includes performance management (EPM) software that helps plan, budget, forecast and report on a company's financial results.
However, even though many of the modules in the project were similar, it wasn't as much of a full ERP. The analysis, metrics and monitoring part was much more important. Knowing also that the company already had an ERP managing all the departments and that its data will also be integrated in the new platform.
So, by making the technical file that studied in depth certain aspects of the needs and departments for which the tool was going to be implemented, we were getting closer and closer to a platform, and not just any platform! A Business Intelligence Platform!😫
But what is BI really?
So, According to Oracle again 😅:
Business Intelligence (BI) is a technological process of analyzing data and presenting information to help executives, managers, and other business end users make informed business decisions. Business Intelligence encompasses a wide variety of tools, applications and methodologies that enable organizations to collect data from internal systems and external sources. This data is then prepared for analysis to create reports, dashboards and other Data Viz tools to make analytical results available to decision makers and operations.
I GOT IT 🔥 That's what they need!
Yes it's true Today, companies rely on business intelligence software to identify and extract valuable information from the large volumes of data they store. These tools can extract information such as competitive intelligence and market trends, as well as internal information such as finding the reasons for lost opportunities, etc.
This step is what allowed us to understand the direction to follow and the mindset to have.
Although I've been in development for several years now, I've never had to do this kind of thing or had this kind of responsibility to be the one to lay the foundation for such a big project. But I felt like I could do it basically so I grabbed my balls with both hands and started the work.
THE LEGO GAME
There is a law that I particularly like. The one of Gall.
Gall's law explains why prototyping and iteration are methods of creating value that work well. Instead of creating a complex system from scratch, it is much easier to build a prototype.
The main reason for this is that a complex system consists of a large amount of parameters and dependencies. This is the logic of the Lean-Startup movement with the creation and improvement of a MVP.
I didn't tell myself from the start that I was going to build a BI platform. I didn't tell myself that I was going to build a high performance system with load balancing, caching, and optimizations on everything etc. I started by building a BI platform.
I started by building a rigid skeleton that would allow me to move forward and place my blocks without weakening the structure. The decisions at the beginning were the keys to the rest.
THE ARCHITECTURE
At the beginning, I was thinking about all the possible architectures of the BFF world, Microservices, Micro-Frontend, etc.
But you will understand that Rome was not made in a day. Because of our passion we are often under the "hype" of wanting to use or test everything that comes out. But these solutions apply to specific problems. If the FAANG or GAFAM are always releasing new architectures, frameworks, etc.; it is not by will but by necessity. The names often come much later. During this project I had to make some choices, some tweaks and solve some problems before I realized that there was a name for it.
I started the project with a distributed architecture to, as I progressed and as I needed to, move towards the extreme 'Segregation of Principle' by doing MICRO MICRO 😂
FRONT-END FIRST
"Is this a Joke 😨?!" No! The first and biggest challenge of course was on the backend. 'How do I connect to all of his databases and gather all of this data?'
This was a question I didn't have an answer to yet but I'm someone who likes to, as I told you earlier, move fast. Since all eyes were on me I had to live up to the hype and what better way than to bluff 😏.
Whether it's in the corporate world or outside what users and applicants see is the front. So I couldn't spend another week saying I was doing research. Since I'm very comfortable with design, I started directly integrating what I had done as a mockup and proto with ANGULAR so my CIO would at least have something to work with in the meantime.
I had to deploy a version of the application in one of our internal servers from the start and set up a CI/CD pipeline so that every 02 or 03 days they could follow the evolutions with pride.😂
Yes I chose Angular on the front end and the reasons were many. In my opinion when it comes to enterprise project, platform or management tools it is the best choice in JS technology. Its scalability oriented architecture, maturity, structuring and stability make it easy to standardize and evolve the built solution. I used to say that Angular is to JS technologies what JAVA is to other languages. It is strict and powerful and very scalability oriented.
However it is not because of Angular that the platform became what it is. Angular, React, Svelte, Vue, Laravel are all tools! Like a pencil in the hand of an architect. It's up to that hand to make excellent use of them and create a jewel.
Within a week I already had the login, the homepage and several UI components that made the project very easy to assemble.
From the beginning I wanted to find very efficient and scalable shortcuts. After redefining the needs I knew that we will have a lot of 'Data Visualisation' on the platform so I turned to the Suites or libraries of UI components such as Syncfusion, KendoUI, PrimeNG*, etc.
Usually one is enough. Once the choice was made, it was all about UI crafting. I spent several days creating reusable components allowing to create an interface, a page or a module just by calling them and setting parameters.
FRONT ARCHITECTURE & CODING CONVENTION
When I started the project I knew that sooner or later I would need to take more people with me to go faster. But it was important that these people, when they arrived, understood easily and quickly how the project was structured and how to contribute to it.
So I made a document defining the coding conventions in relation to Angular at the box level. It's a document of about 15 pages available to all IT people and new devs such as the incredible Ababacar Diene Ndiaye who took over successfully.🔥
Here is a small overview.
FOLDER TREE
We apply the basic tree structure of an Angular project. However the src folder which contains all the features of the application we have organized it as follows:
As you can see we have created some folders to make the project easier to read.
THE COMPONENTS FOLDER
The component folder will contain all the views and UI components of the project. Our applications will mostly use a [Module Separation] system (angular.io/guide/feature-modules) (See here) so this folder will be organized as follows:
Ui-elements
We will put all the unitary components in this folder that can be used everywhere independently of the module. For example: DataGrid, Chart, Chart Generator, Exporters, Button, Card, Modal, etc.
Layout
We will put all the components that make up our "Theme Styling " and the different Modules of the application. For example: Admin Module and its theme, Navbar, General Module, etc.
View
In this folder we find all the pages/views of the site organized in subfolders according to the modules they belong to. These views will be attached to the ChildRoutes provided so that they pass through their respective modules when called.
THE CONFIG FOLDER
In this folder you will find all the global configuration files of the current application. For example, you can put in this folder: the links to the backend, the default language, the state of the application in debug or in production, etc.
THE HELPER FOLDER
In addition to Angular's Component-oriented architecture I apply what I call a 'Helper-oriented microarchitecture'. Helpers can be considered as providers that allow several modules to share or use a processing engine in order to avoid code duplication.
In order not to overload the "components.ts" file of a component when it needs to do some processing, we create a Helper for it which allows even in the future if another module needs this processing or this data it will only call the helper.
THE MODELS FOLDER
As its name indicates it will contain all the interfaces (types) of our system if need be.
THE MODULES FOLDER
This folder contains all the complex components forming an isolated mini-system. Example: the notification module, the system logger, the encoder (to compress or encrypt some payloads sometimes), etc.
Uh...😑 as soon as I don't have a name for a package or a utility I create, I prefix it with 'Orbit' it's not out of narcissism 😂😂😂.
THE ERROR FILE
In this folder you can find the error pages and the error management module. The latter allows us to intercept, study and manage all the errors in our application. At the end of each day, the logs are grouped together and sent to the database so that we can study them.
Example of logs in the base:
SECURITY & MONITORING
Security is one of the most important aspects of the application. That's why we have an "encoder / decoder" in front and back because none of the information that passes between our applications is put in clear (Yes even the token 😂).
Especially the few info saved in the LocalStorage although it is not sensitive we encrypt them then compress them to make them unreadable for humans 😅 or outside the app.
In addition our Encoder allows us to have more info in the parameters of some of our GET requests while keeping it very short.
Here we have a GET request with a payload containing a DateRange matrix in its parameters. 
We also have different monitoring and tracking helpers that allow us to have all the info about the users as well as their actions in order to better monitor the app or to avoid some frauds or false claims😂. Example below users can have more advanced info on their private IP in our network (different from public ip) as well as other info that can help the IS team to intervene quickly in case of need:
For authentication we also use our own NPM Package based on Auth0 which is called: Orbit-JWT. 😏 It allows us in a few lines to manage the Auth & attach the token on the headers, without having to do interceptor and many other things...
All this makes the frontend very scalable, clear and easy to manage! 🥳
For example for a complex frontend like this:
Where we mainly have:
A Global Filter Component (parametric with more than 05 filters [date range, typeahead select, switch, etc).
A DataGrid with filters on all columns, the possibility of sorting, as well as the aggregation line at the bottom.
An export component that allows you to have the data in any format.
etc...
As complex as it is we easily build it in the application in less than 50 lines(I know! 😫 the number of lines does not define the quality of the code). Like here in the x.component.html file:
Example of the viz in our Front (Please scribble as some data may be sensitive):
All that is DataViz And So Much More !!!! :
...
During the first months, the front end was doing 70% of the processing through different helpers because I wanted to see how far I could push Angular and also because the requests were numerous and had to be done quickly. And yet the performance was very acceptable. Hence the importance of focusing on the architecture more than on the technos. Because as cool as a tech is, it is the one who uses it that will determine what he will do with it.
However, as you can imagine, the data that the front end manipulates comes from the back end. So I'm going to tell you how I was able to solve one of my biggest challenges in 2021 with incredible ease!
READ PART 3:
READ THIS HISTORY IN FRENCH ON MEDIUM :
How I built a BI platform without killing myself (VF).
🥳 Or Check out My Web Site: https://orbitturner.com