Why Big Data is Becoming a Big Deal for Payments

By Ernie Beernink, Senior Systems Architect

If you follow payments trends closely enough, you’re bound to hear a few phrases get tossed around more casually than a beachball at a Dead concert: machine learning, real-time payments, unicorns, and of course, big data. 

Aside from being the center square in “Buzzword Bingo”, big data is playing a key role in the evolution of finance. It sits behind many of the innovations that are changing how merchants, consumers and tech platforms interact with payments. 

But what exactly is big data? What are some of the real-world use cases that can illustrate its value? 

Big Data Basics

Let’s start by oversimplifying. Big data is a set of information so vast that it would be nearly impossible for humans to make sense of without the aid of serious computing power. Think millions, billions or trillions of pieces of data, all connected and yet seemingly disparate at the same time. 

Big data exists even when you wouldn’t expect it to. For example, take a small to mid-sized lender that accepts 10,000 payments a month. This doesn’t seem like a large data set at face value, until you begin to think of all the data points related to a payment. 

You have a customer, making a payment, with some form of tender, on a particular device, at a certain time and date, using one of many browsers, in a specific location…and on and on. Add them up, and sheer the number of data points for that “smaller” lender can balloon into the millions. Start tracking back through the years and the data set continues to grow. 

The amount of big data isn’t the only issue. Most of what you need often lurks in the darkest corners of the company, hiding in separate databases, spreadsheets and log files, as well as in proprietary software like account management or accounting systems.

At this point, you might be thinking “how could this be useful?” The number of variables involved is staggering, and it can be difficult to imagine how to turn all this insignificant information into important insights. 

And that’s where things get…fun. The true power of big data is consolidating all these disparate data sources into a single place so that it can be easily queried and analyzed in different ways by different people for different purposes.

To give you some more practical examples, let’s take a look at some of the ways PayNearMe is using big data both today and in the near future. 

Making Data Accessible with Warehouses

Ever wonder how you can place an Amazon order and have it delivered later that day? It’s because Amazon builds large warehouses stocked with its most popular products within 1-2 days drive of most Americans. But it also uses smaller shipping hubs stocked with items that are more likely to be needed by people who live close by. This makes it incredibly easy for you to get what you want, fast. 

Data warehouses are used similarly. Data from all your sources are collected into a giant staging area called a ‘data lake’. A data lake stores massive amounts of historical data in a single place in a clean and consistent format. That data is then pulled into one or more data warehouses, designed to be queried and analyzed on a massive scale. If needed, smaller data warehouses, referred to as ‘data marts’ can be assembled, intended for a single purpose (compliance, sales, accounting), or client. Data marts allow the fastest access to the data you need, because they only contain what you need.

Data warehouses allow for incredibly speedy data retrieval and processing. In plain English, this means you can get more of the information you want faster. 

Why you should care: Imagine having access to years of customer payment data, and having the ability to access that data for whatever purposes your business requires. Data warehousing makes that dream a reality. 

Turn Boring Logs into Improved Authorization Rates

Many business users only see a fraction of the data produced by their operations. Sure, you might see payment dates, amounts paid, and account numbers. But how often are you exposed to the web browsers, devices or operating systems used to complete payments? 

Much of this information, along with a hoard of other background data, is stored in server log files. This data is sometimes used by IT departments to troubleshoot issues, but is largely kept separate from usable business data. 

Data warehouses now make it realistic to store all this data together in a single location, and software like Splunk is making it possible to turn technical logs into a treasure trove of useful information. 

Why you should care: What if you could run an analysis of failed payments and learn that a significant number of them happened on the same devices or at certain times? This type of information could help you identify the root cause of the issue, helping to drive up authorization rates over time. 

This is exactly the type of optimizations that PayNearMe is making across our portfolio, allowing all our clients to benefit from anonymized log data. When authorization rates go up, the cost of processing payments goes down. 

Realtime Business Intelligence

When we covered data warehousing, we hinted at the possibilities it unlocked for making fast data processing possible. Combining your warehouse with business intelligence tools turn this from an abstract concept into something that can drive serious bottom line improvements. 

With tools such as Looker, Tableau or Microsoft BI, you can turn mountains of big data into real time dashboards and business drivers. For example, you can build a dashboard that gives you near-instant monitoring of your payments, allowing you to identify anomalies before they fester into problems. 

PayNearMe is working on new ways to allow you unprecedented access to your payments data so that you can use it to improve your business and create better outcomes for your customers. 

Why you should care: You can’t manage what you don’t measure, and business intelligence tools allow you to measure whatever your heart desires. It gives your analysts the ability to create new insights while giving your executives the ability to understand them at a glance. Oh, and you can finally retire the complex Excel spreadsheets that only 3 people in your company can actually understand. 

Advanced Security and Reliability

Big data is beginning to play an important role in data security, compliance and risk analysis. As we’ve seen countless times in recent years, bad actors are becoming more sophisticated in their ability to infiltrate, breach and abuse financial data. Fighting fraud and protecting your customers data is extremely difficult, even for the most seasoned security experts. 

Big data allows the constant stream of information—login attempts, page accesses, port accesses and much more—to be collected and analyzed in real time. This allows us to quickly detect and mitigate bad actors, and equally importantly allows us to detect when something isn’t working correctly. By monitoring an expected stream of data, we can detect anomalies of all sorts.

Why you should care: Even the smallest security threats can pose significant risks for your organization. Staying one step ahead of bad actors with malicious intent is required to protect your customers data and the health of your business. 

The Big Finale

As you can see, big data isn’t some obscure “future of” concept. It’s here right now, and it’s fueling the growth and maturation of the payments industry. By understanding what big data is, and what it can do, you’ll be able to make more intelligent decisions about how to use it to improve your business. 

To learn more about how PayNearMe is taking advantage of all the benefits of big data, and to see how we can help you do the same, request a demo with our team (I swear, marketing made me print this last line).