Introducing react-busser: Designing for better application data flow in ReactJS — Part 1
In October 2018, i gave a talk titled State design for complex web applications about how application state was being badly designed and constructed in many frontend browser applications that use state management solutions like Redux (or ReduxToolkit)/Zustand with centralised-reactive (flux) state or Jotai/Recoil with decentralized-atom-selector (atomic) state. This talk of mine highlighted the need to separate the application state into two broad categories and hence state contexts for the frontend. The very next year 2019, react-query made its debut on the OSS scene and was an instant hit among most Reactjs frontend developers for managing client-side state (even vue-query which inspired by react-query has been created for server-side state management).
Today, react-query has been renamed and extended to work with Vuejs and Sveltejs. Also, with the management of state for mature web application being more a significant decision in the development process, the state management landscape has exploded with a lot of tools and options that are either particular to a particular UI library or very UI library-agnostic and can be used with any. However, many developers still complain of hitting a wall at scale when using many of these state management libraries out there. So, what could be the problem ? Why is it that all state management solutions are easy to use at the start of a frontend project only when the scope and size of the application is small ? It turns out that the problem is not designing our code to enable better and more structured data flow by coupling together the 3 disparate concerns that should always remain separate. Today i make a case for react-busser as i try to expand on the answer i just gave (above 👆🏾) using Reactjs as a case study (keep reading 😀).
The coupling of concerns (Oops!)
All application state is not just only about the composition of data itself, it’s also about flow of data in software. However, the way we write our code can either impede or enable the structured flow of data in a software application. Niklaus Wirth, a famous and eminent Swiss computer scientist of the 1960s and 1970s wrote a book in 1975 titled Algorithms + Data Structures = Programs. It was a very influential book back then and still is. I feel that the title of the the book however correctly capturing what software is (or should be) made up of doesn’t shed light on how software is (or should be) built. Also, in this day and age (from the late 1970s), graphical user interfaces have been very popular for presenting the stuff of software as opposed to the early 1960s. So, assuming his book was written just after the Apple-1 was created in 1976 by Steve Wozniak, if i were to name the book in 1977, i would name it (Presentation of Data Structures + Algorithms) * Structured Flow of Data = Programs (the asterisks stands for “multiplied by”).
This is because presenting software data to a user requires its own logic (UI logic) as well as processing the data (algorithms — business logic) and structuring the data itself and its flow (data structure and data flow — data logic). Together these 3 things (UI logic, data logic and business logic) make up the 3 concerns of every piece of software ever built. This brings me to the part where i talk about the abuse of the separation of concerns or as i like to call it: the coupling of concerns.
Have you ever noticed how easy it is to recklessly couple or mix up the 3 disparate concerns when building frontend (web or mobile) applications with Reactjs ? What are those 3 concerns you ask again ? Well see below:
- The business logic (data processing based on data assembly).
- The data logic (data assembly, access & storage — network or file I/O).
- The UI logic (data rendering & presentation).
It seems that as we write code, it’s so easy to couple these 3 together so often that we don’t even notice how or when we are doing this until it’s too late (and we have to painfully refactor our code). I kinda blame Reactjs for this because as i have noticed over the years, it’s to easy to write horrible Reactjs code as there aren’t very reasonable constraints in place to guide development. For instance, why embed direct file/network read-write operation(s) logic within your business logic ? Why muddle up your business logic within your UI logic and vice-versa all inside a single Reactjs component ? Why make an Ajax request while a Reactjs component is trying to re-render or programatically change the page route all in/from one place ? These questions all deal with the topic of proper separation of concerns.
When the 3 concerns are separated properly into their own location and context in and around a Reactjs codebase, there are 3 things that can easily happen:
- It’s easier to design and structure application state properly and better.
- It’s easier to test the code written with properly designed application state.
- It’s easier to extend and add features to each concern of a well-tested code.
So, how do we separate these 3 concerns since we so easily couple them together ? Well, there is only one way to do this and it’s by decoupling along the 3 concerns. How we decouple these concerns in our codebase is as important as what we decouple. Hence, it’s important to decouple the core UI library or state management (Reactjs UI library-specific and/or Zustand/Jotai) logic from your application code logic. It’s only then that you would have properly separated these 3 concerns.
So, what does it mean to separate core UI library or state management (Reactjs UI library-specific and/or Zustand/Jotai) logic from your application code logic ? Keep reading and you’ll find out.
Transient and Non-transient state data
Before we can decouple properly we need properly classify and categorize all of the data that flows through our application frontend software. Not all data is created equally on any frontend application. For instance, let’s say you have a list of products on the database of your e-commerce app. This list of products is initially persisted and any changes made to it (due to the interactions of the frontend user via the user interface) is also persisted. However, not all changes are persisted right ? Yes, you’re right! When the user filters the list of products (fresh from the database) by color or by price, the resulting list of products is displayed to the user but never persisted (most times anyway). Also, as soon as the user closes (perhaps abruptly) the browser tab where the e-commerce app is running, that filtered list of products is lost forever. Yet, guess what ? You loose no sleep over that loss and neither does the user. The Mobx docs alludes to something similar.
This brings me to the 2 broad classes (or categories) of application data on the frontend:
- Transient data (hardly ever persisted anywhere, computed, changes often, is only valid for the current browser/app view session)
- Non-transient data (always persisted somewhere, not computed, doesn’t change often, is valid forever)
Here are examples of both classes (or categories) based on our ficticious e-commerce app earlier:
Transient data: whether a page is ready to render or not (isLoading), a resulting list of products (fresh from a database) filtered by price or color or model e.t.c, whether a button is disabled or not (isDisabled), the vertical or horizontal scroll position of the products page, the route URL of the checkout page, the unpersisted list of products in the cart and the quantity count for each product before checkout (before the user makes up their mind), text being typed continually into an input or textarea HTML element.
Non-transient data: the list (array) of products and their entity relationships on the database, the persisted list of products in a cart and the quantity count of each product at and after checkout.
A while after Redux first made its debut, software developers around the world were building small and very large frontend applications with it. They were also sticking both transient and non-transient data into the Redux store. Both of which are wrong!
Transient and non-transient data both exist only on the frontend. On the backend however, you just have non-transient data only (in the SQL or NoSQL database) and its many duplicates (in session stores like Redis or Kafka).
You see, on the frontend, if your app is really small in size and scope, then there’s no need to use a state management library. Furthermore, application state is local and never global at scale — meaning when the frontend app is small, using a centralised global store is okay. Hence, as the size and scope of the frontend app gets larger and larger, a centralised global store losses most if not all of its merits. Also, only non-transient data should go into your Redux store ever because non-transient data doesn’t change often and Redux is not optimised (at scale) for storing data that changes often plus bad things happen when you plug lots of your transient data source mutations to a Redux store in a React component. The same applies when you’re using Zustand with a single global store.
The complexity of you data also affects the choice of your state management library. If your application state data cannot be easily serialisable into JSON string and back, then you will have to choose something other than Redux or Zustand. Perhaps you might try Mobx or Jotai.
The issues with needless complexity
There is some level of needless complexity inherent in most of the tools and libraries built/created to handle state management on the frontend. This complexity makes it difficult to build and test code easily especially with Component-based UI libraries like React. This 9 miniutes video by Tanner Linsley describes the premise for this. Our job as software engineers is to bring complexity down to the bare minimum yet we find new ways to turn it up to the maximum.
Currently, solutions like Redux, Mobx, Jotai and Zustand are great because of their easy learning curve (although some are more easy to use than others due to minimal boilerplate code) and the fact that they actually try to conform to the way ReactJS is built to work as a UI library. However, they all mostly couple data logic, ui logic and business logic all together in one big thread of conjoined logic (actions, async actions, middlewares, stores e.t.c.) by making the component adapt strongly to the form of the data to be fetched and rendered and the flow of that data through the business logic and ui logic. This locks the component up into a specific context (or use cases) and makes it difficult to reuse that Component outside the use case it has been restricted to be used in. Also, it makes it difficult to write unit tests for the business logic independent of ui logic (which is the best way to test ReactJS components).
For instance, a Zustand-powered component has the horrible practice of keeping the store’s state between unit tests forcing you to mock out the entire useStore() hook if you need to test your components which defeats the purpose of proper testing. Yes, it’s also possible to test the useStore() hook by itself too but you still need mock out your own business logic implementations too so you don’t reset state for each test case (something that Redux frowns on — even as it creates more complexity to handle the issue). Again, you mostly can’t test a Redux-powered components in unit tests but only integration tests that do not give you the benefit of having unit tests for your components especially unit tests for your custom reducers, custom async actions and custom middlewares. You could test reducers and middlewares directly as distinct units or use tools like redux-mock-store which encourages mocking the Redux store and redux-actions-assertions (which i do not recommend — using mocks that have no internal implementation usually leads to more false positives and unintentional coupling in tests) to test that actions produce the right state update in the Redux store.
Mobx is a bit better at unit testing in ReactJS than Redux or Zustand but because the state isn’t local to the component like Redux and Zustand, you still have to mock your Mobx store (and sometimes what the Mobx store depends on) when testing your components and for most of the time it doesn’t play nice with Jest. Also, like Zustand, it doesn’t reset store state after each test case is run. However, you can only get it to work by creating a ReactJS context provider to inject the store down to components and hacking the render() function for React testing library. Jotai is much more better at unit testing as it correctly abides with proper testing principles than Redux, Zustand and Mobx but has it’s own area of complexity. This has to do with atomic state that serves no real purpose in the domain of the UI. As it turns out, most application data doesn’t need to be split and broken up into “atoms”.
All of this is because state isn’t localised to the ReactJS component (except in Jotai) leading to needless coupling and other needless complexities like actions, dispatch, proxies, middlewares e.t.c. Also, data usually moves in cohesive groups as it flows around in any frontend application not in disparate atoms like in Jotai. What Zustand, Redux, Mobx and Recoil try to do is present transient and non-transient data as distinct data containers while in the real world, they are not because they constantly flow in to and out to each other. Furthermore, one (transient data) flows from the other (non-transient) as so should not be persisted at all (don’t persist transient data) in any way. Application state should always be localised in the ReactJS component. This makes it easy to testing, modify and extend your components.
Getting rid of needless complexity
The react-query docs does a very good job of clearly explaining why you don’t need Redux, Zustand, Mobx or Jotai for server (non-transient) state management here. Using a tool like react-query is the very first step to redesigning the structure of your application state and hence encouraging better data flow in your frontend application. However, this is only one half of the job needed to properly remove complexity. What we have done by pulling state that should be local from a global state container to is decouple our ui logic from our data logic. The second half is to decouple our business logic from the data logic and ui logic that we initially decoupled. This is called separation of concerns.
The thing is as we build ReactJS components with tools like react-query, we mix business logic into the ReactJS components’ data access logic and ui logic such that the business logic is coupled and lost within the fetching of data from the server and the rendering of that data on the UI.
Also, one of the best paradigms for building user interfaces is to build is to build a state dependency graph. Now, this graph is a directed graph of different pieces of state that depend on one another in a graph-like manner. The issue with ReactJS is that it is extremely difficult and unintuitive to create a state dependency graph using useState() and useEffect() for state that is business-logic-driven that has aync components. However, it is easy to create a state dependency graph in React if there’s nothing async being used to build React apps.
When you analyse a lot of the popular state management solutions well enough like Jotai, Zustand or MobX, one of the things they all have in common is:
- They all make use of a core primitive called an observable under the hood.
- They all enable the creation of state dependency graph easily and intuitively (in React or other UI frontends).
When you use Mobx, you can create MobX observables that are dependent on other observables using computeds. The same also applies to Jotai where atoms can be made to depend on other atoms via composition. These are the different ways that the different state managers build state graphs or state trees (think Redux or Zustand).
The important thing to note here is that the state graph or state tree is created outside the React component lifecycle intentionally (and unfortunately can become out of sync with it usually before componentDidMount()or useEffect() and will have to be re-synced) plus for good reason.
Now, why is this necessary ? it is necessary because of how the component tree (or component graph) of any React app is fused tightly with its’ state graph. For every other UI library out there (e.g. Svelte, Solid, Vue), the state graph is usually separate from the component graph. It is for this reason that React has to render on every state change so it can recalculate the nodes of the state graph which correspond tightly with the component graph that is controlled by that node in state graph.
If React only operated with synchronous logic and state, then having to re-render on every state change will be a very sound way to update a UI. However, the ease with which a state graph can be created in React breaks down when you throw asynchronous logic and state into the mix.
On the surface, this fundamental difference in design between SolidJS and ReactJS seems to be of little consequence but it isn’t. This difference makes it clear why SolidJS has much less of what ReactJS has in order to build UIs.
More on this in Part 2.
Let’s take a look at how we can get rid of complexity in a real life ReactJS project in 7 stages.
The app we have built is a simple app that filters a list of items. However, we start at STAGE 1 with a very bad version that is complex, leaks business logic inside the components’ ui logic and end up with a well-refactored component at STAGE 6 that packages business logic within Reactjs hooks that are highly configurable and reusable. Hence, fully decoupling the application code from the React UI library code.
In STAGE 1 — The app is crude and has a lot of code that duplicates effort and needs to be decoupled. The filteredList (transient data — computed from filteredListAlgo) and list (non-transient data — passed in as a prop) are separate but they shouldn’t be and this affects the UI render logic too. Also, the filteredListAlgo() function needs to be decoupled from any calls to setAnyState() which is react-specific code. Let’s move to STAGE 2
In STAGE 2 — The app begins to take shape as the structure of the state is consolidated. So, list and searchText are merged into one state object pased to useState() instead of two separate objects. We also, update filteredListAlgo() to fuse the list (non-transient data — passed in as a prop) and filteredList (transient data — computed from filteredListAlgo) to become one and the same instead of two separate lists because filteredList only becomes relevant if searchText is not an empty string else list and filteredList converge in equivalence and value. Therefore, list and searchText are merged into one state object in obidience to the Single Responsibility principle: in a function, class or module, bring together things (data and/or logic) that change for the same reason and separate things (data and/or logic) that change for more than one reason.
In STAGE 3 — We rename a few variables that don’t feel relatable to the logic being defined. Instead of textAndList, we now prefer words. The same applies to changing setTextAndList to setWords. Also, filteredListAlgo() now has 2 argument and not one anymore. This is such that we can decouple calls to setWords() away from filteredListAlgo(). A new change event handler (a level of indirection) is created for calling setWords() directly. Finally, we setup useEffect() to persist the filtered/unfiltered list to local storage and clear it once the component is unmounted.
In STAGE 4 — We further decouple the UI logic by providing a swappable render prop called renderList which is passed words.list. Then, filteredListAlgo() is extracted out of the component and injected back in as a prop.
In STAGE 5 — A brand new custom hook useTextFilteredList is created and encapsulates all the business logic of filtering a list of strings or objects and can be reused in any other component too.
In STAGE 6 — This is the finalisation stage. Any more decoupling after this is not necessary. The only thing that can be done is write tests and extend the capabilities of what is already implemented (as we can now choose between 3 algorithms for filtering the list: specific, fuzzy & complete). In Part 2 of this article, you’ll find out how easy it is to further extend the capabilities of filtering the list. The only extra thing we do is create another brand new custom hook useBrowserStorage. The new hook helps to centralize and abstract access to browser storage (localStorage & sessionStorage).
Wrapping up and setting the stage for next steps (Part 2)
I hope i have been able to show us how to decouple the 3 concerns that are usually easily mixed up in ReactJS. Earlier, i did allude to 8 stages but for Part 1, we’ll stop at STAGE 6. So, STAGES 7 and 8 will come up in the Part 2 of this article where i discuss react-busser in more detail and how it helps separate the 3 concerns easily from the start of your ReactJS project with little hassle. I will also show you why Redux, Zustand, Mobx and all the others were (in my opinion) erroneously designed the way that they are because they are modeled after the data flow model for the backend which is different for the frontend data flow model. I will further talk about concepts like cascade broadcasts and the evented system that react-busser works with.
In the meantime, check out this live demo of react-busser in action (below 👇🏾👇🏾). It doesn’t use so much ReactJS props to communicate transient data but rather uses events to communicate. It also doesn’t re-render as much as your usual ReactJS apps do. And like react-query, react-busser is made up of just ReactJS hooks. See you in Part 2! 👋🏾
