The simple truth about Web Scraping for Open Finance

Why web scraping is essential for bringing Open Finance to life

Anyone who’s worked in the financial services industry for at least the past 10+ years has likely heard about web scraping in some way, shape, or form. After all, until the latest PSD2 (Revised Payment Services Directive) regulations were put into action, thereby allowing banks to connect with third party providers (TPP) via secure, standardized APIs, web scraping was pretty much the only way the Open Banking movement could get itself off the ground.

Unfortunately, those APIs had a limited scope: they only allowed for the secure sharing of checking account data. Of course, that makes complete sense for the bulk of Open Banking-related needs. But as we all know, understanding an individual’s or a business’s full financial picture requires digging into a lot more than just checking account data alone.

This is why web scraping is largely an essential tactic for moving the world towards and enabling an Open Finance—and eventually, an Open Data—future. And while the term “web scraping” itself may sound somewhat intrusive, invasive, or even insecure, when done right and with the proper precautions in place to protect a user’s data, it can be even safer, more secure, and more effective than API connectors. Powens has actually mastered this process.
However, knowing that there is a lot of misinformation and misunderstanding about how users can safely share their financial data with TPPs—either via APIs or web scraping-based solutions—we figured it was time to clear up many of the misconceptions that have surrounded the practice of web scraping for Open Finance once and for all.

 

What is web scraping for Open Finance?

If you’re new to this, you might be asking yourself, “What is web scraping for Open Finance exactly?” Calling out “Open Finance” in this instance is an important distinction because there are actually a lot of other use cases where web scraping—sometimes also referred to as screen scraping—is often used outside of the financial services world. And for the purposes of this article, we want to focus squarely on web scraping for Open Finance. That’s why we thought it would be useful to offer up a quick refresher to make sure we’re all on the same page.

Web scraping for Open Finance is more or less a way for users—whether individuals or businesses—to give consent to TPPs, like payment service providers such as Powens, to be able to access to an array of useful financial data. This data could include anything from checking account transactions to savings and investment details in a wealth management portfolio.

To access this data, the user must share their financial account login details with a TPP, who then “logs into” the account—with the user’s explicit permission—to “scrape” their data. The end goal here is simple: by having access to this data, companies can provide their customers with better user experiences or more personalized products, services, and offers. For example, many financial service providers or insurance companies may want to see this data before making credit decisions, finalizing insurance policies, processing instant payments, and so on.

Unfortunately, not all web scraping is carried out with the user’s protection and safety in mind, especially outside of the financial services industry, which is why the practice itself has become stigmatized as a less safe alternative to APIs. For this reason, Powens has always made a point to consistently go above and beyond to ensure that web scraping-enabled connectivity solutions are aligned to established PDS2 regulations. Part of this includes doing regular audits with banks to ensure that the customer data being shared is being used for the right purposes.

 

How is web scraping for Open Finance different from APIs?

Since not all web scraping solutions for Open Finance can promise the same level of safety, privacy, and security as those created by Powens, you would think that industry regulators would demand that only APIs be used to share financial data, right? Not exactly. Global regulatory bodies are well aware that existing APIs have limitations and, therefore, see the value and innovation that web scraping for Open Finance can bring to the industry as whole. So if both data sharing solutions must cohabitate, what really differentiates APIs from web scraping?

Simply put, an Open Banking API is a secure, stable, and regulated connector—between a bank or financial institution and a TPP—that does not require users to share credentials. Additionally, during the authentication and consent steps of the connection process, users can specify exactly which datasets a TPP is authorized to access on their behalf. In other words, an API can limit the scope of data sharing by not giving TPPs free reign to scrape all data from a user’s account.

Because APIs are supported by PSD2 regulations in an effort to help Open Banking flourish, they provide a cleaner, simpler, and (assumed) safer way for users to share their data. That’s why APIs are always the preferred method for sharing financial data, if available. However, because PSD2 regulations only established standards for APIs enabling the sharing of checking (or payment) account data, web scraping still remains the gold standard for accessing other financial data sources that support Open Finance solutions (beyond Open Banking alone).

Adding yet another layer of complexity here, APIs falling under PSD2 regulations needed to be created in-house by banks and then made available to TPPs. Developing, testing, and deploying these APIs was not an easy task for banks across Europe. This was on top of the fact that some banks were already reticent to lose their so-called monopoloy over the banking data in their customers’ accounts. But as we’ve discussed before, financial data belongs to the customer; banks are merely service providers that “house” this information. So even though they may have taken their time to deploy custom-made API connectors, they knew that playing nicely in the Open Banking “sandbox” was essential for retaining customers in the long-term. Plus, for many, it served as a catalyst for embracing digital transformation from the inside out.

But seeing how much of an uphill battle it was for banks to successfully deploy their APIs, the team at Powens knew that web scraping would need to be a perennial fallback solution should APIs ever crack under pressure. That’s why we made a decision early on to keep our pre-API connectors in place—even after connecting to bank APIs—to ensure that banks and TPPs could remain seamlessly connected should a country’s fintech infrastructure ever fail. Obviously, this is a situation we hope to avoid, but at least there’s a safeguard in place should it happen.

 

5 common web scraping for Open Finance myths debunked

Doing web scraping for Open Finance correctly is not an easy task, especially doing it in accordance with existing regulations around user safety and data protection. These solutions take a lot of time to develop and require constant improvement and refining over time. In other words, not all web scraping is created equal—and that’s why it can have a negative reputation.

But we’ve developed some of the most sophisticated and secure web scraping-based Open Banking and Open Finance solutions for well over 10 years now and, as a result, can say with confidence that we do it better than any other player in the industry. This is our secret sauce. It’s also a big reason why our solutions have been able to unlock the power and potential of Open Finance across multiple use cases. Other industry players can’t make the same claim.

Because we know just how powerful web scraping for Open Finance can be—and the good it can bring to users when done right—we wanted to clear up a few lingering myths.

 

 

Myth 1: Web scraping for Open Finance ended with PSD2 regulations

Far from it. Existing PSD2 regulations cleared the way for the development and deployment of APIs that would be used specifically for simplifying and streamlining how individuals and businesses share only their checking account data with TPPs. These regulations had no purview over anything outside of the realm of checking accounts. To expand beyond Open Banking and create solutions for an Open Finance world, web scraping must still remain the gold standard. This doesn’t mean that future regulations won’t create standards for sharing and accessing other kinds of financial data, but until then, web scraping is the go-to catalyst for innovation.

 

Myth 2: Web scraping for Open Finance is not safe or secure

This isn’t true at all, but some TPPs recently created like to say this on repeat. They are simply jealous about what Powens can offer via web scraping for Open Finance that they can’t.

We’ve spent years perfecting our web scraping-based solutions to ensure that they not only meet the safety and security standards of existing regulations but also provide users with added peace of mind throughout the authentication and consent process. We have also put added measures in place—like regular audits with both banks and our clients—to ensure that the data being shared via web scraping connectors is being used for the right purposes.

For example, is someone is applying for a loan, they may only want to share their data in that moment for the purpose of enabling an immediate credit decision and then close that connection after. For accounting solutions, on the other hand, users must consent to keep that connection open to ensure that their accountant has real-time access to transactional data.

While there’s certainly not a one-size-fits-all for how web scraping for Open Finance can be used across multiple use cases, the solutions we deploy on behalf of our clients—for the benefit of their users—are developed to meet the highest safety and security standards at all times.

 

Myth 3: Web scraping for Open Finance is not reliable

Although in most cases we would prefer to work with APIs, especially because they tend to offer simpler and cleaner workflows, they are not fool-proof. When many banks across Europe began deploying and testing their recently developed Open Banking APIs, a number of bugs and tech issues caused them to break. While many of these issues were fixed early on, they certainly can rear their ugly head again. Quite to the contrary, we spent over 10 years perfecting our web scraping-based solutions to ensure near-zero downtime. This is why, as mentioned above, we’ve kept our web scraping connectors in place so that there’s a back-up plan should APIs ever fail.

 

Myth 4: Web scraping is for Open Banking only

Before PSD2 regulations created standards around APIs for sharing checking account data, web scraping was how the Open Banking revolution got started. But now that those APIs exist, we’ve taken everything we learned by building the Open Banking model and applied it to unlocking all sorts of possibilities for an Open Finance future. The truth is, we’ve consistently stayed one step ahead of banks, regulations, and the financial services industry as a whole in order to create innovations that have the power to transform user experiences for the better—and also motivate key industry players to embrace digital transformation.

 

Myth 5: Web scraping for Open Finance is another name for “web crawler”

A web crawler implies that there has been no consent whatsoever. It’s basically something that just sits idly on the web and scavages for data as you go about your online journey. That is not what web scraping for Open Finance is, much less how we’ve used it in our own solutions. Whenever Powens serves as a connector between users, their banks or financial institutions, and the businesses interested in using their data to provide a specific product or service, there is a clear and transparent workflow that explicitly asks users to authenticate—verifying that they are, indeed, the account owner—and give permission to share their data. It’s important for users to know what they’re sharing and who they’re sharing it with. We simply are there to facilitate this interaction in the safest, most secure, and most effective ways possible.

 

Tap into the future of Open Finance with Powens

When we talk about our promise of “Open Banking with Superpowers,” we’re actually talking about web scraping for Open finance head on. Our ability to build technologies and solutions to help usher in the Open Banking movement couldn’t have been done without advanced web scraping expertise. And seeing that Open Banking has practically become omnipresent in users lives today, so much that it’s now table takes across the industry, we’ve taken our web scraping prowess to imagine, build, transform, and innovate the future of Open Finance and beyond.

From our perspective, none of this would have ever been possible without web scraping for Open Finance. Perhaps it’s about time to change the (almost entirely) false narrative around it.

 

Ready to transform your business with the power of Open Finance

Contact our team today to schedule a demo of our Open Finance products and services.