Edit: A day and a half after publishing this post, the source of the data was eventually identified and a statement issued. Do see the updates at the end of this post.
I see a lot of data breaches. I see a lot of legit ones and I see a lot of fake ones and because of that, I always verify them before making any claims that an organisation has been hacked. Usually I'll verify and then in conjunction with journalists I know and trust, there'll be a private disclosure to the company involved. Good journos are very adept at getting answers to these things and when it's going to be a story that hits the news anyway, it ensures there's a way of getting responses from the impacted organisation before it hits the interwebs. Every so often though, we all get left totally stumped as to what actually went on.
Such has been the case recently for a data breach that I'm highly confident is legitimate but nobody wants to "own". I've worked with a couple of different trusted journos who are very good at getting answers but have ultimately been unable to draw the saga to a conclusion, largely because neither of the parties I believe are involved believes the breach originated from them. So I'm just going to write about the whole thing here, lay the facts out as they stand then see if anyone wants to own it once the details are public.
It all began with this tweet a couple of months ago on 10 July:
This isn't an embedded tweet because it has since been deleted. However, that happened more than a month later which was plenty of time for people to access the alleged BlueSnap database on the Mega hosting service before that link was also disabled. I grabbed a copy of it for later review then headed off on travels, not returning to look at it properly until late August.
BlueSnap is a payment provider which allows websites to take payments from customers by offering merchant facilities. BlueSnap was founded in Israel back in 2001 where it was originally known as Plimus (both of these facts have later relevance I'll come back to). It was later acquired in 2011 for $115M and rebranded as BlueSnap which is both the present day trading name and the alleged source of the breach in 0x2Taylor's tweet.
Obviously the first thing anyone is going to do when verifying a data breach is look at the contents so here's what I found: The data is in a single file named "Bluesnap_324K_Payments.txt" and as the name suggests, it has 324,380 rows in it with a total of 105k unique email addresses. The first transaction is on 10 March 2014, the last on 20 May 2016. Each row appears to be a payment record which looks like this:
The grey obfuscation is personal information relating to an Have I been pwned (HIBP) subscriber who assisted me with the verification process. The red obfuscation is card data and the arrow points to the "security-code" field which is the CVV. This is the CVV too but again, I'll come back to that.
This is actually only a small porting of the row, in fact it's a mere 14% of the entire record. Every row begins with "0x2Taylor" and contains pipe delimited values along with XML you see above. I've actually decoded a portion of this; the original file included encoding as follows:
\u003ccard-type\u003eVISA\u003c/card-type\u003e
Which decodes as follows:
<card-type>VISA</card-type>
This gives us a bit of a sense of where the data may have been used as the encoding could be used in the JavaScript context.
The other clue in the file here is the word "Plimus" which as you'll recall, was the name BlueSnap went by before 2011. That's two positive indicators of the source but they're also easily fabricated indicators and I wanted some hard facts. So I asked for them.
I've just passed 700k verified subscribers to HIBP, that is people who've come to the site, added their email address to the free notification service then received a confirmation email and clicked on the link to opt in. These are people who are interested in their exposure online, exactly the sort of exposure that this breach here has led to. What I do these days when I need to verify a data breach that's a bit harder than usual or is particularly sensitive is email some of the most recent HIBP subscribers who are in the alleged data breach and ask them if they're willing to assist in verifying the incident. When they respond (and it's always a positive response because they're naturally curious), I send them an email with questions along these lines:
- Do you live on [redacted]?
- Did you have a Visa card that expired in [redacted]?
- There is a purchase against your record from 2014-06-15 for the value of $160 USD; do you recognise the name beginning with "JCC-Maccabi-Games"? This is possibly the service you paid.
- This may be a harder one given the card has expired, but if you recall, did the CVV end with the number [redacted]?
Let's talk about that CVV for a moment. The Card Verification Value is an extremely important piece of data because it's used to verify the card in scenarios where it's not present, such as when making an online purchase. When the retailer requests the CVV, it means that even if someone has your card number and expiry, without that 3 or 4 digit code the data should be useless as far as making online purchases go. For example, if a database of transactions is leaked then so long as there's no CVV then the cards should be useless on any site that requests it (most do, Amazon is a notable exception to this). When the CVV is in the hands of a malicious party, the very mechanism that was put in place to protect consumers in "card not present" scenarios falls apart. PCI DSS is very clear about how the CVV (or CVV2 as it is these days) should be stored:
It shouldn't be stored and that's what makes this breach such a big issue. Violation of PCI DSS guidelines can lead to pretty serious fines and even loss of merchant facilities; the card providers take this very seriously. I take it seriously as well which is why I also asked HIBP subscribers to verify their CVV by providing me with an additional digit to avoid any confirmation bias (I didn't want them just answering "yes" to each of my questions). It checked out - this is the CVV.
I still wanted to be certain the transactions themselves were clear though but it was tricky to identify the actual source from the raw data alone. The one indicator of the source that was present in the file was an attribute named "soft-descriptor" which in the example above was "JCC-Maccabi-Games". I wondered initially whether this might just be a case of one particular site losing a bunch of data, that was until I aggregated the attribute and looked at the spread of records. In total, there were 899 unique values with the top 20 by prevalence appearing as follows:
- EntourageManageme : 6299
- regpackclients : 6084
- Kidventure : 3728
- METNY2015201 : 2660
- Group-RX-New-Camp : 2535
- Wild-Whatcom : 2453
- CampKeeTov2016 : 2232
- garinusa : 2178
- JCC-Maccabi-Games : 2163
- USY-Summer-Program : 2088
- AvaAndersonNonT : 2005
- National-College-T : 1986
- High-Sierra-Pools : 1919
- Dedicated-To-Learn : 1846
- METNY-2014-2015 : 1761
- Dedicated-to-Learn : 1717
- EastBaySPCA : 1700
- SanDomenicoSummerC : 1684
- SAEP : 1642
- USY-International : 1548
The record I was looking at was merely the 9th most common result, clearly there were many others involved too. But it still wasn't clear precisely what these websites were nor what was purchased from them. The answer to that lie further down in the data within a Plimus URL formatted as follows:
https://www.plimus.com/jsp/show_invoice.jsp?ref=[redacted]
As the URL suggests, this then takes you through to an online invoice like this:
There are many interesting things about the invoice, the first of which is that it obviously identifies BlueSnap quite clearly both by virtue of their brand and the Plimus URL. It also matches the individual's identity and address from the data breach file which goes a long way to establishing authenticity. Then we can see the website itself where the payment was made which is at jcca.org. The site has a donation page complete with a payment form:
As you can see, the logo clearly indicates that this is "Secure Credit Card Processing"...
There's nothing on the site or the structure of the payment form that indicates BlueSnap though and it looks as though the integration with the payment provider is done entirely on the server side without exposing that information publicly. But there was another piece of information on the invoice which didn't initially stand out at me and only later piqued my interest after another HIBP subscriber made this comment:
I still have the conformation email (a Summer Camp). It referenced http://www.regpacks.com so that might be a possible source too.
Now this is interesting because the invoice in the earlier image refers to a support email address on the regpacks.com domain. Regpack offers a registration service and part of the feature set is this:
Receive payments during registration rather than post-registration
Dealing with payment info is serious business so they also offer some assurances as to their security position:
Another piece of relevant information on the Regpack website is a list of just a few of their customers, including JCC Maccabi Games:
Every single HIBP subscriber I contacted had an invoice referencing a Regpack email address for support. It was looking more and more like they were taking the registrations then passing them downstream to BlueSnap for payment processing. In fact, that's precisely what was happening and it was easily verified via a press release a few years ago:
Waltham, Mass.---April 2, 2013---BlueSnap™, the most flexible and advanced buying platform for online companies selling goods and services over the web and mobile, today announced that Regpack, a global online enrollment platform serving the private education industry, has selected BlueSnap to process the financial transactions for its online enrollments. Regpack integrates with BlueSnap’s flexible and advanced payments platform to provide a complete enrollment and payments solution for organizations such as private schools, camps, educational tourism, faith community organizations, seminars and professional conferences.
In that press release, the Regpack CEO goes on to say:
Moreover, BlueSnap’s strict security measures for online transactions mean that we can use BlueSnap to process payments and conduct business without going through the expense of becoming PCI-compliant level one on our own.
Now by this stage you'd think the whole thing was wrapped up; either Regpack or BlueSnap have had a data breach and leaked a few hundred thousand transactions replete with partial card data and CVVs. The problem is though, neither party believes the breach came from them. I worked with two separate journalists on this and they both had feedback from BlueSnap and Regpack suggesting another party was responsible. I also reached out to them both yesterday for comment and got this from BlueSnap:
Based on an investigation we initiated as soon as we heard about the data set, we hired a top PCI-certified Incident Response firm. Based on that investigation we confirmed that BlueSnap did not experience a system breach or any data loss.
And got this from Regpack:
As a preventive measure, we ran a full forensic investigation and it has concluded that there was no data breach on Regpack servers. In spite of that, we have run the full security protocol implemented in these cases and conclusively determined that our servers were not involved.
Personally, I see indicators implicating both of them. On those that point to BlueSnap losing the data, there's the name of the file itself and 0x2Taylor's original assertion that it came from them in the first place. The file wasn't named "Regpack_324K_Payments.txt", it was BlueSnap's name in there and whilst a file name alone is not proof of an incident, it's an indicator. Then there's the nature of the sites that were involved; when I checked with HIBP subscribers, we identified sources such as the Jewish Community Centers Association of North America mentioned above, Liberal Judaism and Passages America Israel. There were other non-Jewish organisations involved as well (such as the East Bay SPCA), but it's hard to ignore the coincidence of the organisation being implicated as having lost the card data to have its origins in Israel then see such a prevalence of Jewish websites using their services. But then again, they all had Regpack support email addresses on them, so onto them...
Regpack's name is associated with every one of the HIBP subscribers I contacted. I'd expect that if BlueSnap was the source of the breach then we'd be seeing a mix of downstream consumers in the file, unless they store the data in such a way that Regpack's records are isolated from other customers and they alone got breached. Another indicator pointing to Regpack as the source of the incident is that per the statement above, they don't need to be PCI complaint and thus haven't gone through the rigour of audits. (Edit: I've put a strike through this because the CEO's comment was around level one PCI compliance. Regpack may be compliant with a lower level requiring less rigour.) Now by no means does merely being PCI compliant guarantee a breach won't happen, but when the transgressions are as egregious as storing the CVV, something is majorly amiss. And finally, "regpackclients" features as the second most common "soft-descriptor" in the earlier bulleted list with over 6k entries. That's slightly odd because there are many other descriptors which then have invoices referring Regpack's email address for support, but it's yet another indicator of how heavily they feature in the data.
Now it's possible that the data has come from another unnamed party, but it's highly unlikely. Not only could I not pick a pattern in the data suggesting it was sourced from elsewhere, but the CVVs just shouldn't have been there. We've got 899 totally separate consumers of the Regpack service (so it's not from one of them) who send their data direct to Regpack who pass payment data onto BlueSnap for processing. Unless I'm missing a fundamental piece of the workflow (and I'm certainly open to suggestions on what this might be), it looks like accountability almost certainly lies with one of these two parties.
Lastly, just to absolutely, positively avoid any remaining doubt that this is a legitimate data breach, let me share a collection of responses from HIBP subscribers (note also the responses regarding the CVV):
Address is correct and yes I did have a card that expired in 2014
That all seems right
Yes, that information is correct
I had a Visa card ending in 10 and I am pretty sure it expired in 2013
Yes, we do have a visa that expires in 2020, and yes the CVV ends in 8
This is genuine information that you have provided
I don’t know how they got the CVV either
So that's where it stands at the moment - it's highly likely that either BlueSnap or Regpack lost the data - but frankly, I'm more concerned about those who have their info floating around the web which includes:
- Names
- Physical addresses
- Email addresses
- IP Addresses
- Phone numbers
- Last 4 digits of their credit cards (remember, this is identity verification data and it's enormously useful for hijacking accounts)
- CVV
- Online invoices which then include details of their purchases
These people need to know that their data was posted publicly to Twitter and none of us have any idea how many people now have it. They need to cancel impacted cards (full card data wasn't leaked, but refer to the link above re partial data being used to hijack accounts) and be aware that their personal info has been exposed. The sites using these facilities also need to be notified because they're the ones that have the relationship with the customers. This requires the cooperation of BlueSnap and Regpack, the former of which is still hosting those invoices publicly on the plimus.com domain where anyone who has the invoice numbers from the breach can simply enumerate them and pull down even more personal data. It may not be a pleasant experience for them, but they need to step up and take responsibility.
I've now loaded all 105k email addresses into HIBP so if you think you may have been impacted, you can search for your address on the site. I've indicated that it's a BlueSnap breach and linked through to this post simply because that's the name it was represented as but will change that if it's determined otherwise. Right now the priority should be in supporting those whose personal data has been disclosed and attribution can follow later.
Update 1 (12 hours later): I've had further feedback from BlueSnap who remain adamant the data hasn't come from them and have issued the statement below to their merchants. I've asked point-blank if they believe Regpack is the source of the breach and will post an update here if there's any feedback I can share. As yet, I don't believe the individuals in the breach whose data is been publicly circulated have been notified by either party.
Update 2 (24 hours after initial post): There's been a lot of discussion on this incident both in the comments below and via email. A number of people have said they've reached out to Regpack and received responses indicating that they weren't the source of the breach and offering little support beyond there. I want to reiterate a few immutable facts:
- The data in the breach is legitimate and contains personal information
- There are hundreds of thousands of transactions out in the wild including details on over 100k customers
- The data contains the last four digits of the card which are frequently used for identity verification purposes
- The data contains the CVV which should never have been stored by anyone
- BlueSnap has known about the incident since at least the 21st of August
- Regpack has known about the incident since at least the 26th of August
- Websites who had customer data exposed were using the services of Regpack
- Regpack may not have lost the data, but they're accountable to their customers which means the sites using their service
- As yet, to the best of my knowledge those impacted in the data breach have not been notified and that includes both websites using Regpack and customers who made purchases
Given there's still no resolution to this and neither BlueSnap nor Regpack believe they're responsible, I'm listing all 899 "soft-descriptor" values below complete with the number of transactions each has in descending order (these are the websites using the Regpack service). If your site is amongst that list and you're concerned for your customers, contact the organisation you sent the transaction to as they're the party you have the relationship with and entrusted with the data.
Update 3 (a day and a half after initial post):
I've had further communication with both BlueSnap and Regpack since writing this post and the source of the data has now been identified as originating from Regpack. Let me share a statement from them here:
Further to the article Troy Hunt published both Regpack and BlueSnap have looked into the presented data loss. Reviewing the post by Troy Hunt assisted our engineers in reaching this conclusion: Regpack has confirmed that all payments information passed to the payment processor is encrypted on its databases. Nonetheless, periodically, this information is decrypted and kept internally for analysis purposes. We identified that a human error caused those decrypted files to be exposed to a public facing server and this was the source of the data loss. This was identified by our teams going back and reviewing some of the log files as indicated in the blog discussion post. We have changed our approach to handling this data and are confident that this one-time mistake will not occur again. To reiterate our security stance: 1. The source of the data loss was a procedural human error. 2. Neither Regpack nor BlueSnap had our systems breached. This has been confirmed by independent forensic experts retained by each company after the initial data loss. As a further security measure, RegPack has rebuilt all servers and run full security scans on the new servers. 3. Both Regpack and BlueSnap have conducted thorough reviews of the environments and found that all systems are secure. 4. Regpack and Bluesnap have updated all internal security procedures and processes to ensure that no data can leave internal environments. This will prevent the loss we saw in this case. Regpack is notifying vendors whose customers were potentially affected so they can make the appropriate communications.
Obviously they now have various processes to go through including reaching out to impacted customers who will in turn need to contact their customers (the ones who made the purchases) and notify them of the data exposure. I've just updated HIBP to reflect the source of the data as being Regpack and adjusted the description accordingly.
If you run a website that uses Regpack services then you should hear from them directly. If you believe that your personal information was exposed then you should hear from the site you provided it to (yes, I know they didn't lose the data but that's the chain of relationships here).
Thank you to everyone who commented and provided input on this post, I'm glad the source has now been identified and steps can be taken to protect those who were exposed.