Processing Payments Without a Database

A few days ago I had the strangest realization.

Over the past few months, I’ve been studying threads and asynchronous programming. I’m not alone: ​​the software engineering industry is slowly coming to terms with the fact that hardware capabilities are starting to plateau. The future of more efficient applications will come from code, not from machines.

But async had a built-in assumption that I didn’t really think about too much. so far.

Here’s the thing: asynchronous programming is not designed to make your program run faster. This is to avoid wasting time – time the CPU is wasting waiting for responses to network requests.

This is a well-known problem. In 2010, Ryan Dahl introduced Node.js, stating: Our I/O operations are completely wrong (I/O is roughly equivalent to “access existing content”). In most programming languages, calling functions and making external requests look very similarbut the former executes immediately, the latter blocks, and the execution time is longer than the former.

Asynchronous is an execution model that accepts this partitioning and creates a different set of keywords and evaluations to handle block operation, even if it requires code Difficult to write, read, and change.

But it’s not the only option. Async accepts this division; what if we don’t? What do we need to do if we deny the possibility of blocking operations, that is, those that have to wait for an external system to perform some action?

You may be wondering how this is possible. Can we do away with third-party systems? At least when it comes to payments, we can’t. But some offer web hookprocess means making a request to some endpoint of your choice when some pending operation completes.

But this is not enough. Is there a database? Every payment system relies on persistent storage to accomplish the tasks it needs to accomplish. We cannot have a payment system that is not database-based. Correct?

That’s when it hit me. We are forced to deal with asynchronous programming because we believe that libraries are completely necessary. Because we can choose which database to use (PostgreSQL, MongoDB, etc.), but we can only choose.

What if we didn’t have to do this?

I am Alvaro Duran,This is Pay Engineer Handbook. If you scroll on YouTube for 5 minutes, you will find many tutorials on designing payment systems. But if you want to build one for real users and real money, you’re on your own.

I know this because I have been building and maintaining payment systems for almost ten years. I was able to see all kinds of private conversations about what worked and what didn’t.

Recently, I decided to make these conversations public. That’s it Pay Engineer Handbook was born.

One reader said the book is about “deeply delving into the stacking behind the magic.” We research transfer technology so you can become a smarter, more proficient, and more successful payments engineer.

We did this by cutting out a small piece of it and extracting tactics from it. Today we’re looking at event sourcing and how it can eliminate the need for persistent storage when we process payments.

If I had no idea how data systems work, I would never have imagined that we would have to store data about payments.

Think about it. Most payment systems are message brokers: they take data from the customer and use it to send a message to the bank indicating that the customer and merchant have agreed to the purchase.

Yes, some information will be retained. Accountants need it to balance the ledger, and finance needs it to perform reconciliations and keep records in the event of a chargeback dispute. Analytics require it to achieve business intelligence. But the payment system itself requires little of this data—it just produced.

Storing payment data is an afterthought. It doesn’t make the payment system better.

Therefore, in theory, we can use the information temporarily saved in RAM to process payments.

If there is data but no database, there must be a fundamental shift in the way systems process and react to information. This shift is largely about healing As a first-class citizen, changes in the country, not the country itself.

this is called Event sourcing.

You can always reconstruct the status of your payment by viewing your payment history.

Therefore, collecting facts is equivalent to storing the latest version of the customer’s payment.

You are already familiar with events. event notification, or publish/subscribeis a design pattern whose goal is to separate the producers of information from the systems that use the information.

This is great for PSPs because it doesn’t make sense to make any code changes every time a new customer wants to integrate with their system. In fact, some message queues sit between the PSP system and the end client so that the PSP emits events, and the message queues ensure that the appropriate client receives the event.

Event sourcing is the ultimate conclusion of this approach: why not let Even the payment system itself In terms of events?

Pub/sub decouples producers from consumers. Event sourcing decouples the system from any persistent storage.

Ultimately, the payment system is constantly reacting to new facts, with the expectation that the state of this payment will be rebuilt every time a new event arrives.

I know I’m being too abstract, so let me clarify this with an example. The payment system collects events about a given payment:

• The first activity, payment_createdstart the process.

• another event, 3DS_requiredcausing the system to send a 3DS challenge to the client.

• this 3DS_completed will cause the system to attempt authorization.

• one pending_authorization The event will not trigger any action until completed_authorization Event received. Therefore, the rest of the system will continue to fulfill orders.

• when order_fulfilled Once received, the payment system will request to capture the payment to collect the funds.

• And when completed_capture After receiving the event, the payment is completed.

I want you to notice two things. First, there is no need to preserve any of these events immediately. They can be stored in memory while another process retrieves them in the background.

The second thing is, at each step, whether that step is likely to be determined by what happened before.

Therefore, we can treat these events as temporary records in the payment system, eliminating the need for a repository. Once we complete the payment, we no longer need them. But if a payment system fails, we must ensure that no events are lost and that we can resume operations immediately.

There is a solution: hot backup.

One of the advantages of events is that you can have multiple systems running the same event stream. So they have a very good hot backup method. They basically run two systems all the time, one is the dominant system and if it fails the second system will take over immediately.

—Martin Fowler, The many meanings of event-driven architecture

This is why event sourcing is so popular high frequency trading industry. Systems such as Maximum LMAX Process millions of transactions per second on a single-threaded program by keeping all operations in memory, taking hot backups in the background, and taking periodic snapshots.

By the way, this shouldn’t surprise you. That’s what prompted this quest all along. High frequency trading firms process millions of financial transactions per second, so I asked myself, “What is their secret?”, “Do they handle threads and desynchronization in a special way?”.

They did; they abolished them entirely.

I consulted many references to put this article together. The video that lit the fire inside me was This lecture on mechanical empathy and LMAX high-frequency trading. I’ve heard the concept of mechanical empathy on the Jane Street Podcast (They are not HFT firms, but they create markets, which is the same in many ways). Which brings me to Martin Fowler’s video On the various meanings of “event-driven”and His website has an article on memory imagesand ended with a shocking mention of LMAX. How likely is it?

If you want to learn more about event sourcing, Martin Fowler provides A good primer on the subject.

this is already Pay Engineer Handbook. I’ll be taking next week off and will publish my next article on January 2nd, for paying subscribers only.

If you’ve been living under a rock these past few months, this newsletter will be paid for in 2025.

Wishing you a Merry Christmas and a Happy New Year.