WePay’s Bill Clerico talks DevOps at Rackspace Solve

Monday, we at WePay had the pleasure of attending Rackspace Solve in San Francisco, the first of several major conferences by Rackspace aimed at showing how various companies are using the cloud to solve tough business problems.

Our CEO Bill Clerico gave one of the first talks of the morning on a problem that every startup eventually runs into: scaling up to meet massive user demand without falling flat on your face.

Anybody who’s not gone through it would probably put that in the “good problems to have” category, but make no mistake: scaling is one of the toughest challenges in the technology business, something that takes a robust DevOps strategy to tackle. We would know, since WePay has faced some unique scaling challenges. Put it this way — the payment volume we process is set to more than double this year, and it will double again next year if we have anything to say about it. To keep up with demand, we’ve hired about 20 people in the last month, mostly in engineering, at a company that’s only about 60 people strong.

In other words, getting to scale is a major initiative here at WePay, one on which we’ve worked closely with Rackspace. We were one of the first customers for Rackspace’s DevOps Automation Services, which helped us to keep our service running at a time when we were facing unprecedented demand, and has continued to help us as we look to take on even more volume.

You can watch Bill’s full talk at the video above, but here’s some takeaways in the meantime:

The time to think about DevOps is now: Bill said that if he had one do-over at WePay, it would be to have started thinking about building great infrastructure earlier in the process, because it’s something that sets a company up for success at every stage. In the early days, the understandable temptation is to focus technical talent exclusively on product, but if you don’t also spend time building out great infrastructure, it’s easy to get stuck with early mistakes that will hurt you down the road, Bill said.

“As a technical founder, you think you can just go on Stack Overflow and find what you need solve any problem, but that’s really not the case,” he said. “You can’t have great DevOps just by reading some blog.”

DevOps isn’t just a technical problem: Because DevOps is a problem with a technical solution, it’s tempting to think of it as just a problem for the technical side of the business. But a poor or non-existent DevOps strategy hurts every part of a company. It’s a customer service problem, because especially in the enterprise  zero downtime is the one of the No. 1 demands that your customers make of you. It’s a product problem, because bad infrastructure limits how fast product teams can work and how well they can innovate. (With due respect to Mark Zuckerberg, you need to move fast, but if you break things every time you do, that’s a problem.) DevOps even affects recruiting, Bill said.

“I’ve certainly found it to be the case that the best engineers, the ones you really want to hire, won’t want to work on crappy infrastructure,” he said.

Just because DevOps is important doesn’t mean it needs to be entirely in-house: At the same time, DevOps is often a problem with peaks and valleys — sometimes your servers are getting hammered and you need a full team working around the clock, and sometimes nothing is happening and you’ll struggle to find work for just one person. That being the case, don’t be afraid of working with an outside team, Bill said. In WePay’s case, working with Rackspace enabled the company to be very agile, scaling up to meet demands and get support when it was needed and then scaling down when things were more stable. Outsourcing can be an important tool in the arsenal.

12 data sources that help WePay detect and fight fraud

Online businesses in the US lose an estimated $3.5 billion to fraud every year. That number is only set to rise as long as we keep having security breaches like the Target hack, which put an estimated 40 million credit card numbers in the hands of fraudsters.

The payments industry has typically fought fraud by making merchants go through a lengthy and involved process to prove they aren’t risky before they accept their first cent from customers. Yet in today’s digital economy, that just doesn’t cut it anymore. The platform businesses that are the real drivers of growth in the new economy — things like crowdfunding sites, marketplaces, and small business software providers — need to sign on new merchants fast and start processing payments immediately. That requires a new kind of payments system — one that’s faster, more secure, more flexible and backed by more machine intelligence than ever before.

At WePay, we’re building that payments system. It’s a tough engineering challenge, something our VP of Risk John Canfield laid out recently for attendees to Q Conference in New York at his talk “Leveraging Big Data for Payment Risk Management.”

You can watch the whole talk here, but one of the more interesting bits was John’s take on what data to actually look at — a bigger problem than you might think. Machine intelligence approaches rely on having data points that are actually predictive; so choosing what to focus on is an important first step in creating a system for assessing payment risk.

Here are some of the data points that can help a payments company assess risk:

·    Know Your Customer, or KYC info: This is the classic information like name, address, date of birth, and social security number that is required by all banks to open a merchant account. Everyone requires this because it works — although fraudsters can gain this info too, so it can’t be the whole of a fraud assessment strategy. It’s good for a first pass —You can check it against records held by companies like Experian and Equifax to verify that a person by that name actually exists.

·    Traditional business credit reports: Assuming you can get a business credit report, this is a great source of insight into a potential merchant. The problem is that this isn’t usually available in the Bottom Up Economy, the fast-growing e-commerce space in which “merchants” are often individuals able to punch above their weight thanks to small business software, online marketplaces and crowdfunding. If a merchant isn’t a traditional business, then the credit reporting bureaus generally won’t have the same level of data about them.

·    Business License: Again, this isn’t going to be available for many payers. But if a business is registered, that’s a signifier that it might be legitimate.

·    Business Social Media: This varies from business to business because businesses use social media to greater or lesser extent. However, if a business has a social media profile, that can be a good thing to look at. It’s not a 100 percent signal, because it can be faked, but if a social profile has accumulated a great deal of likes or followers over a long period of time and sees regular engagement, that’s an excellent sign of legitimacy.

·    Editorial reviews and ratings: This can be even better than social media, because these are outsiders evaluating a business, and even businesses that aren’t especially active on the Internet might have garnered reviews or been mentioned in newspaper articles. The issue is that it can be a difficult process to gather this info, because it’s not usually packaged neatly into an API for you. Often, checking for this can be a very manual process, but parts of it can be automated as you get a better understanding of the sources for this data.

·    Street view addresses: Also a manual process that can be automated somewhat later on. Street view is useful because it allows a payment company to answer a simple question: does the building at the address the merchant has given me look like it matches the kind of business they’re representing themselves as? A word of warning, however: small businesses very often try to make themselves look larger than they are by doing things like giving a mail forwarding address that belongs to a large, professional looking building.

·    Personal Facebook pages: A person’s social media profile is an incredibly valuable source of data — it establishes their identity online in much the same way a drivers license does in the offline world. What’s more, it’s hard to fake. Even a person who doesn’t use Facebook much will look very different from a fraudster — they’ll likely have had the account for years and have many followers that have built up over time. This is doubly so if you can confirm the merchant has control of this account.

·    Device ID: These are newer technologies that try to tie a transaction to a specific device offered by companies like ThreatMetrix, Iovation and Experian/41st Parameter. This goes beyond just looking at the IP address — these technologies look at a variety of data points to establish a unique fingerprint for each device. And that’s useful because it lets one establish blacklists, allowing one to prevent further fraud from the same device once fraud is found.

·      Google: When you do a Google query for the name of the business or individual, does it return results? Is their website or social profile in those results? It’s a simple test, but still useful. Google search results provide 3rd party verification of the existence of a business or an individual, and they could lead to other potentially useful sources of data like reviews and blog posts about the business.

·      Control Verification: This requires users to take an additional step to sign into an account by entering a code sent to their cell phone. This protects against takeover attacks, because in order to takeover an account an attacker would have to have the victim’s cell phone in addition to their username and password. Passing a control verification is thus a very good sign, from a risk perspective.

·      Transaction History: Not all sources of data are external. If a merchant has been using your service for a decent period of time, then their transaction history is an excellent source of insight. Fraud might look very different than the baseline behavior you see from this merchant — think a Halloween store that normally does all of its business in October which suddenly sees a spike of transactions in March. It can also give you early warning that a merchant is starting to go down hill. If a usually good merchant starts generating an unusual number of chargebacks, that’s a sign that fraud or something like it is occurring.

·      Partner Data: It turns out that most platform companies actually have a lot of data about the people using their service that would be very helpful for a payments company trying to assess fraud risk — things like the kind of payment being made, what’s actually being bought, how the service will be delivered and 3rd party data that they’ve collected themselves. Yet tradition payments vendors have had no way to see this data, so they can’t use it.  

WePay’s answer to that problem is Veda Risk API, our patented method for collecting a range of data from our partner platforms easily, securely and without any impact on the user experience. Not to toot our own horn, but we think it’s pretty great. Obviously, the kinds of data varies a lot from platform to platform — a crowdfunding site knows different things about its users than a small business accounting software. That’s why we’ve built Veda to be extremely flexible. Here’s a sampling of some of the things Veda can look at:


If you want to learn more about how our risk assessment system works, contact our API team at api@wepay.com


Rasmus Lerdorf Talks PHP 5.6, 6.0 and Beyond

Rasmus Lerdorf likes to joke that he’s the most famous programmer from Greenland that we’re likely to meet this month.  Truth is, the creator of PHP is one of the great legends of open source.  The software he created is one of the most widely used technologies for server-side web programming. We were fortunate to catch up with Rasmus at the WePay office, listen to his talk on the present and future of PHP.

His presentation is available here.  It covers PHP 5.5, PHP 5.6 and web deployment.  Rasmus also responded to questions from the audience about PHP 6.0.

The hot topics were performance, security, web deployment, compatibility with different PHP versions, and his perspective on Facebook Hack.


Here’s a sample of the great discussion that Rasmus had with the audience.

Q: Tell us about the direction of PHP, in particular PHP 6.

Rasmus:  First, I want to emphasize that PHP is driven by the needs of developers.  People will present features, including test cases.  If it’s reasonable, then the feature will get into PHP.  It’s not about my wishlist. I don’t have a team of developers. This is a community of developers building PHP for other developers.

Q: But, if you did have a feature wish list, what would you wish for?

Rasmus: If I had a wishlist, I would probably wish to clean up the internal API. It’s gotten a bit nasty over the years. The other thing is of course, performance.  

Q: What about backward compatibility?  I’m worried that PHP might go down the path that Python went down with Python 3 not compatible with Python 2.

Rasmus: We’re not planning a Python type of break. Chances are that anything that breaks in PHP 6 will be features that resulted in pretty hard warnings in PHP 5, which means that it wasn’t in the best interest of the developer to do that in the first place. No drastic changes are planned in PHP 6.

Q: What’s your take on the static code analysis?

Rasmus: Etsy uses static analysis. I would love to see it. The community is hoping to get Facebook to release their static analysis system as a standalone tool. This type of tool would make it much easier to be compatible between PHP 5.x and PHP 6.

Q: Any thoughts on Facebook Hack?

Rasmus: Facebook has done some amazing things. That said, not all of their work is going to be applicable to the entire PHP community. If we’re going to put new features into PHP, it has to be something that is good for the developer, not a special case for a specific company.

WePay API Documentation Revamp

WePay’s API has always been designed with platforms in mind - marketplaces, crowdfunding sites and small business software providers. In January, with our new Series C funding, we doubled-down and focused solely on platforms and the API. Now our documentation more clearly reflects this focus as well and makes it even easier for platforms to integrate our API.

Step by Step Examples

We’ve provided step-by-step examples for our top platform types to quickly integrate based on their unique needs.

Each of these examples:

  • Explains the overall scenario use case

  • Gives code samples you can copy & paste

  • Gives both the Simple and Advanced integration options

Announcing Our New Merchant Portal User Interface

We’re pretty excited to share an upcoming change that helps our partners and their merchants view WePay account balances and notifications. The new Merchant Portal provides better visibility into a user’s cash flow. 

For the past few months, we’ve been asking our partners how we can make it even easier for you to do your business. We learned that their merchants want to see:

  • pending payments
  • next scheduled withdrawals
  • current balance
  • amounts subject to reserves, disputes or chargebacks