Imagine embarking on a journey to tackle the complexities of integrating banks into your system. You start with a simple idea: gather, store, and present data. But the landscape is ever-changing, with countless unknowns lurking around every corner.
So, you make a decision: Keep it simple and focused on the present needs. Your solution must be like a living organism, constantly adapting to its environment. It’s akin to the concept of genetic algorithms, where solutions evolve over time, learning from mistakes, and becoming better with each iteration.
Picture it like this: each generation of your solution is like a new chapter in a story. You start with what works, but you’re always open to new challenges and experiences. And just like in a good story, you strive for backward compatibility — ensuring that old elements remain while introducing fresh twists to keep things exciting.
Along the way, you’re not afraid to experiment. You test out new ideas, see what works, and discard what doesn’t. It’s all about learning and growing, with each experiment bringing you closer to the perfect solution.
In this narrative, methodologies like Agile and Extreme Programming are your trusty companions, guiding you through the twists and turns of development. They provide structure and support, helping you navigate the ever-changing landscape of bank integrations with confidence.
And so, armed with simplicity, adaptability, and a willingness to learn, your journey continues. Each chapter brings new challenges, but with each challenge comes an opportunity to evolve and improve. And in the end, your solution stands as a testament to the power of storytelling in software development.
In Generation 0 of our data storage and retrieval approach, we embraced a simplistic, optimistic strategy heavily reliant on external bank APIs and network infrastructure. Dubbed the “everything or nothing” approach, it operated under the assumption that any failure during transaction receipt for an account meant the data wouldn’t be stored.
Although we implemented a retry mechanism, it often fell short, perpetuating the same loop and leading to persistent errors like data type mismatches, which ultimately prevented data storage. Despite its flaws, we chose to store the data as-is in a relational database, a decision that, while unconventional, managed to meet our initial use case requirements.
However, as we delved into the exploitation of Generation 0, we uncovered critical flaws in our overly optimistic reliance on external factors. One potential problem identified is the issue of missing data following authorization, highlighting the limitations of an “all or nothing” approach. It became apparent that we needed a more resilient and adaptable approach moving forward.
Enter Generation 1. Realizing the inadequacies of our previous strategy, we pivoted towards an “as much as possible” mindset. Given that accounts are managed under an authorization, we refined our approach to process each account individually within this framework. This method ensured that an issue with any specific account did not interrupt the processing of other accounts under the same authorization. By isolating accounts in this manner, we could effectively address and rectify issues on a case-by-case basis, enhancing our ability to manage and resolve discrepancies efficiently. This strategic adjustment not only streamlined our handling of large volumes of data but also reinforced the integrity and privacy of our clients’ data, demonstrating our commitment to maintaining distinct boundaries between each account, even within a unified authorization environment.
As we entered a relatively quiet period focused on incorporating as many banks as possible into our system, the influx of traffic shed light on the true nature of our data — its shape, volume, and size. Our initial strategy of directly storing data in its original form began to strain our resources and slow down our processing capabilities, particularly noticeable when generating reports for accounts with over 100,000 transactions. Encountering such volumes prompted a reevaluation of our approach towards data storage. To address this, we’re shifting towards a more efficient method that allows us to manage and generate reports without delving into the specifics of individual transactions. This revised strategy is designed to optimize resource use and improve processing speed while maintaining the confidentiality and integrity of the data.
After careful consideration, we made the decision to leverage the full capabilities of our relational database. Armed with a deeper understanding of the data and the Canonic Data Model discussed in a previous article, we set out to create a wide table to serve as our primary data storage solution. This shift not only allowed for more efficient data retrieval but also necessitated the implementation of the Adapter pattern to seamlessly integrate with our existing systems.
However, this transition also brought about new challenges, particularly in data migration and rollout processes. These complexities may warrant further exploration in future publications, as they represent significant aspects of our evolutionary journey.
With the release of Generation 2, we ushered in a new era of data management, leveraging the power of relational databases and embracing a more structured approach to storage. Yet, our journey continues as we navigate the ever-changing landscape of bank integrations, seeking to optimize and refine our processes with each new generation.
The importance of selecting appropriate data types and capacity cannot be overstated, as we learned through a particularly interesting case. Initially, we anticipated that our system wouldn’t encounter transactions in the multibillion range, so we opted for a decimal data type with a capacity of 12 digits, including 2 decimal places. However, as fate would have it, we encountered a scenario where this capacity proved insufficient.
Encountering transactions beyond our initial expectations prompted us to expand our data type’s capacity by adding another 2 digits. Yet, our journey didn’t end there. With the integration of our first crypto provider, we faced yet another challenge. Cryptocurrency transactions, with their inherently different scale and precision requirements, necessitated a further adjustment. We found ourselves compelled to increase the precision to 8 digits to accommodate the unique characteristics of crypto transactions.
This experience underscored the dynamic nature of our data landscape and the necessity of remaining adaptable in the face of evolving requirements. It also highlighted the importance of foresight in selecting data types and capacity, ensuring that our systems are equipped to handle a diverse range of transaction volumes and types.
As we expanded our product offerings to include Verified Analytics and Insights, our confidence in the reliability of relational databases remained steadfast.
However, our optimism was soon met with a reality check.
As we delved deeper into the development process, we encountered a significant hurdle: the overhead of pre-computation for reports during runtime. This overhead became particularly pronounced when dealing with large accounts, leading us to reassess our approach.
In response, we implemented a significant improvement in the next generation of our system. We introduced a concept we coined as “post-processing” whereby the computation for reports was shifted to a later stage, alleviating the burden on runtime operations.
This strategic shift not only streamlined our processes but also enhanced the efficiency and scalability of our system, ensuring that our products could deliver timely and accurate insights without compromising on performance. It was a pivotal moment in our evolutionary journey, highlighting the importance of adaptability and continuous improvement in navigating the complexities of data management and analysis.
Encountering our first multi-million transactions account posed several challenges that prompted a significant evolution in our data retrieval approach. Initially, the process of collecting data for such large accounts proved time-consuming, often taking several hours. Additionally, our existing storing method struggled to efficiently handle the insertion of such vast amounts of data.
Amidst these challenges, it’s worth noting that our encryption protocols remained steadfast, ensuring the security of the data within our system.
To address these issues, we embarked on the next generation of data retrieval, marked by the introduction of bank inserts. This innovation dramatically enhanced performance, reducing data collection times from tens of minutes to mere seconds or a few minutes.
Furthermore, we undertook a comprehensive rethinking and redesign of our data retrieval strategy. One key improvement involved splitting historical data into manageable chunks and simultaneously filling them. This parallelization strategy resulted in a significant performance boost, scaling up efficiency by a factor of N, where N represents the number of data chunks.
These advancements not only addressed the immediate challenges posed by multi-million transaction accounts but also laid the groundwork for a more scalable and resilient data retrieval system. By leveraging parallel processing techniques and optimizing our approach to data storage and retrieval, we ensured that our system could seamlessly handle increasing volumes of data while maintaining optimal performance levels.
Just when we thought we had reached the pinnacle of our data retrieval solution, we found ourselves confronted with a new set of challenges. Despite the efficiency gains we achieved with our latest generation system, we soon noticed an alarming trend: the size and load on our database were growing at an unprecedented rate, seemingly out of proportion to our expectations.
As we delved deeper into the root causes of this exponential growth, we uncovered a myriad of new use cases and requirements emerging from various quarters of our organization. The need for capabilities such as back-fills and many others began to emerge, compelling us to rethink and redesign our approach yet again.
But this realization didn’t dampen our spirits; instead, it ignited a newfound sense of excitement and determination. We knew that the journey to our next generation of data retrieval would be fraught with challenges, but it also held the promise of innovation and progress. And so, with eager anticipation, we set out to explore and create the next chapter in our evolutionary story.
But that, my friend, is a tale for another time — one that you’ll find unfolding before your eyes in the pages of our upcoming adventures.
Stay tuned, for the best is yet to come.
