I think stories like these highlight the need for clearer (not necessarily more) regulations around contractor/subcontractor/client relationships and what happens when one of them goes tits up.
There were rampant issues in the fintech world that exploded when Synapse, a banking as a service provider, went bankrupt and their ledger didn't match what partner banks had in their accounts. End users were told "your deposits are FDIC insured", but in bankruptcy court the judge was sort of at a loss over how to rectify things - the banks weren't insolvent, and the FDIC (rightfully) said "hey, this isn't our problem, our regulated entities are in compliance". Looks like a similar situation happened here, where the contractors are both doing the "not it" thing.
I feel like a lot of tech innovation and "business process innovation" over the past 15 years was just ignoring regulations that were built up over decades, only discovering the reasons for those regulations when the tide went out and seeing that lots of companies had been pantsless the whole time.
Whoever claims "your deposits are FDIC insured" needs to be prosecuted as fraud and scam artists. “Your” the company’s deposits in the bank is FDIC insured. My deposit with you the company is not. When the bank goes belly up, your deposit is FDIC secured up to the account limit which is tiny in the scale of things. When your company goes belly up, my deposit is gone.
My understanding is that they split the accounts to keep up with the limit. Otherwise though yeah, FDIC will only step in for the extreme minority scenario of failure by the underlying bank. You absolutely should not be allowed to advertise FDIC insurance unless it goes all the way to the consumer.
> When the bank goes belly up, your deposit is FDIC secured up to the account limit which is tiny in the scale of things.
The FDIC is meant to protect individual people from loosing all of their money from the collapse of a bank, currently at $250k. If you have more wealth than that yet have it all as cash in a single account, then, you're pretty much an ID10T. For regular mere mortals, that's a helluva lot better than a bank telling you to pound sand when they collapse. If you're a business thinking the gov't is meant to protect you, then you are also delusional.
> If you have more wealth than that yet have it all as cash in a single account, then, you're pretty much an ID10T.
SVB collapse has shown that the 250k limit is basically not relevant. Maybe if a big consumer bank like Chase failed then 250k would be the max paid out, but we haven't seen that.
TBF, a solid 50% of regulation is probably government overreach. The problem is that it's intrinsically tangled with the other 50% (crossing various specializations/laws/departments) that isn't; you can't undo one without undoing both.
love the use of "solid" when both numbers (50, and the other 50) are pure unobtanium pulled out of an orifice.
"a lot" would be less rhetorically satisfying, but probably more appropriate.
I suspect the number is in fact, far less than 50%. In mining engineering, I'd be amazed if a significant majority of the rules don't stem from a significant accident or death, or forseeable need to avoid them. In medicine, the stakes are equally high. Building codes? It depends. The cost of tunnelling in NY isn't because of government compliance.
In building codes, sometimes there's a tragedy that causes government overreach. A death with big public support often causes regulation that "does something" even if the rule itself doesn't help or make sense.
Why do you think the laws and regulations are unclear?
This seems to be following the completely standard and expected process.
Contractor goes belly up, so you go to court and a judge who confirms they were a custodian of your data and you are entitled to retreive it.
Same would be the case if I was leasing equipment to someone and they had it at a storage lot. If the middle party dies or goes bankrupt, I get a court order to claim it from their other possessions.
Its been 3 weeks, and they got what they need from the courts.
> The St. Louis station sued the information management company July 28
How much do you think it cost? A few grand?
Maybe we could replace this by reinventing the wheel, but i dont think it would be better. We could put it on blockchain and let a Peter Theil company handle arbitration.
If you talk to a lawyer (or, worse, a legislator, many of whom are also lawyers and all of whom are surrounded by them), they will insist up and down that this is a pretty standard custodial arrangement dispute and that the system worked as intended. And I would, very begrudgingly agree with that.
The legal system is perfectly capable of recognizing stolen property no matter how many layers of abstraction you put it through. The problem is always in the fact that the dispute resolution process is too expensive[0] to be useful. If you are defrauded for $10,000; but the legal fees for your representation will exceed that; then that juice ain't worth the squeeze. See also: Bricks and Minifigs.
In the Nine PBS case the judge correctly recognized Iron Mountain as a constructive bailee of Nine PBS's property and created a framework to retrieve their data. The problem is that this took almost half a year of legal work to get to the obvious outcome to make Nine PBS whole.
In Synapse's case, the problem is slightly different, because Synapse is not a bank, they are a reseller of banking services. That's the whole idea behind "fintech[1]" - that we can sell banking services while dodging all the regulatory compliance designed specifically to stop these kinds of issues so long as a real bank is involved. Saying their deposits are FDIC insured is like saying you have auto insurance because you happen to be riding a taxi. Technically correct but misleading and fraudulent. FDIC insurance doesn't cascade into your customers' accounts, because if it did, you'd be a bank.
[0] There's a similar problem with Bitcoin, where only a certain number of transactions can ever be processed per hour and thus it bottlenecks any higher-layer process that intends to use the Bitcoin blockchain as a settlement or dispute resolution system.
[1] "Fintech" in particular is meaningless as all banks are tech companies. They were one of the first adopters of electronic computers, online transaction processing, and a whole load of other things that seem utterly quaint now.
EDIT: changed "years" to "almost half a year", I was too lazy to do another Google search
> The problem is that this took years of legal work to get to the obvious outcome to make Nine PBS whole.
Not years. This whole "saga" has been going on for 5 months, and the suit against Iron Mountain was only filed on 28 July, so it just took weeks to come to this current arrangement.
EDIT: Toned down the comment. Leaving the rest, though, since I can't delete it with the reply below.
The problem in the case of Synapse was that they said they were doing that stuff, but were lying. (I think. The details of what happened there are apparently still not public. Also their bank seems to have been doing some sketchy things too.)
The biggest mess with Synapse seems to be the ledgers disagreeing: The banks, Synapse, Synapse's customers, and Synapse's customer's customers don't all agree on whose money is whose, and worse, the totals don't seem to add up.
> FDIC insurance doesn't cascade into your customers' accounts, because if it did, you'd be a bank.
It does cascade. It's called pass-through deposit insurance, it's codified in 12 CFR § 330.5 and 330.7. It has existed since the founding of the FDIC. Its enabling statute expressly provided that deposit insurance should be calculated based on the beneficial owners of a deposit account, regardless of in whose name the account is. [0]
Common arrangements include: HSAs, HOA accounts, UTMA/UGMA accounts, guardians and conservators, mortgage servicing accounts, escrow and title agents, payroll processors, brokerage cash sweep programs, prepaid cards, and yes, fintechs. [1]
Open Source Storage (OSS), the vendor in question, was around for two decades before going out of business last year. The first and last archived versions of its web site:
> William Cravens, the attorney representing Iron Mountain, told the judge his client doesn’t know the format of Nine PBS’ materials that were stored by OSS. He expressed concern about whether Nine PBS’ archival material is lumped together with data from other OSS clients. Iron Mountain wants to avoid potentially corrupting the other data, Cravens added.
> Elliff ordered the immediate return of any physical devices that hold Nine PBS’ data once access to OSS’ storage system is granted…
> Once Nine PBS retrieves its data, the station must work with a third party to ensure that no data from other OSS customers is among those materials.
I don’t love this call. Handing over any disk has several problems. It could easily contain very sensitive data from another client. It’s inherently one-sided (another client can’t make the same request if the disk is already gone). And it’s being handed over to a PBS station that doesn’t seem to have the technical chops to manage data backups/security.
I want them to get their data back, but this needs a technical intermediary that handles all client data equally.
Is that statement by Iron Mountain's attorney believable, though?
If cage or server access was shared between Iron Mountain customers, how can Iron Mountain not have a record of what's in what cage or what's on what server?
If Iron Mountain customers like OSS were free to add or remove things without Iron Mountain's knowledge, then their (OSS's) access would've been limited to cages or servers dedicated to their (OSS's) own clients' data. Allowing an OSS ex-employee the same access they previously would've had should be a non-issue, from Iron Mountain's perspective.
The court got this right. This is when you need a special master to help with cleanup after a bankruptcy.
We saw something like this when TechShop went bankrupt, with member property in storage on the premises. The bankruptcy trustee set up a procedure where former members could make an appointment to retrieve their property, escorted by a representative of the trustee. Huge pain, but necessary.
I'd still argue that I.M. should have anticipated this situation, and have some process in place to address it. "See a court" might well be part of that process.
I'm a bit confused from Iron Mountain's response about how they are worried that data could be co-mingled with other data? That would be an interesting way to back up data.
Iron Mountain probably only knows what data was owned by Open Source Storage, but not on the level of which data belongs to which Open Source Storage customer.
If so, it would be incredibly reckless for Iron Mountain to provide all of the data stored by Open Source Storage to one of Open Source Storage's former customers (PBS). Presumably that is why they wanted to go through the legal process so that someone else would be vested with the legal authority to determine what data PBS can legally access.
No idea how OSS and Iron Mountain were doing things so this is pure speculation, but if for example Iron Mountain were providing an object storage platform on which OSS then ran some secondary platform that stored all the data in one place with separate metadata to figure out what's what then it's entirely reasonable that they might not be able to easily separate it out. Not saying that'd be a good way to do it, but it's easily plausible
I do something similar with my clients backups, each individual client gets their own bucket on B2 for many obvious reasons but to sort out anything beyond what's obvious from the bucket name you need the backup software and appropriate encryption keys.
Sometimes these things can go better for companies like Iron Mountain when there's a court order/decision in place to cover them. Nine PBS, in order to access their own data, could end up accessing data belonging to other people and that may not be properly covered under existing contracts and policies. It's a risk for Iron Mountain if this happens. Having a court decision and court set procedures that essentially force them to participate and also establishes third party review of the data will give them some cover if one of those other clients of the now defunct OSS discovers that Nine PBS accessed or inadvertently retained their data.
A court is needed for cover since it is possible that sometimes else's data will be seen or even corrupted in the process. This way they can say they were doing what the court asked for. That is legal cover for everyone to do what they want. Also legal limits on what they can do.
They couldn't really - I doubt Iron Mountain actually objected to giving the data, but in the end it probably wasn't in a position to know what data belonged to PBS and what belonged to other clients of OSS and you'd get very worried if a data storage company gave a companies data to someone else without authorisation.
It’s likely that Iron Mountain needed a court order in order to do this with legal cover.
It’s just a data warehouse and OSS likely had multiple clients data and backups. Iron Mountain can’t let one third party go searching through a defunct customers stuff that has a bunch of additional third parties property intermingled with it. Too many unknowns and potentially litigious third parties.
I previously commented on this when a different, earlier news article and it wasn't clear at the time if the storage company was a colocation customer, or a dedicated server customer of Iron Mountain:
The news article is really not clear about whether this was, relative to the company "OSS":
a) OSS is a colocation customer with its own hardware colocating it inside an Iron Mountain datacenter. In which OSS owned the bare metal and paid iron mountain for rack space and power.
or
b) OSS is a dedicated hardware customer of Iron Mountain running a service on bare metal owned by Iron Mountain, and has gone defunct, leaving behind a bunch of servers/storage arrays that would in normal circumstances get wiped/reprovisioned.
From the point of view of a customer of OSS (PBS), that's two extremely different things.
If it's scenario A, I don't see how PBS has any claim against Iron Mountain. Your typical datacenter colo host for bare metal hardware owned by a customer has no involvement whatsoever in the condition or operation of the data, operating system, filesystems, RAID arrays, ZFS, etc of how the customer has set up their environment. Nor any ability to do anything with it. A colocation host that hasn't been paid for its rack space and power will typically have clauses in its colo contracts allowing for seizure and sale of abandoned hardware after a certain period of time.
====================================
New commentary:
The new news article seems to shed a bit more light on it, it sounds like it's scenario B, in which OSS never owned the hardware (they just set up the operating system/software config on it), and was renting it from Iron Mountain. In this scenario it's much more realistic to expect to be able to get some data back. And as the new news article says, PBS is paying all of the overdue bills in order to be able to do so.
Ordinarily a dedicated server provider that is renting rackmount bare metal hardware to people has a very automated and fast re-provisioning process, if you don't pay your bill, the terms of the contract let them wipe the servers within a fairly short amount of time and reallocat them to new customers.
Now if it had been scenario A, I think that there would have been very little opportunity for judicial remedy in an order requiring Iron Mountain to do anything, because Iron Mountain would have had no control over what a failing/bankrupt/dead colocation customer did with the data on their servers. PBS might have had a claim against the principals of the OSS company, but then you get into the "blood from a stone" problem of trying to enforce a judgment against people who have no assets or ability to pay.
For some reason, despite these things being rather simple and concrete distinctions, all the reporting around the case keeps being confusing, vague, and contradictory. I had been rather convinced by the Ars Technica article [1] that Iron Mountain was just the data center operator and this was colocation of completely OSS-owned and managed equipment. Iron Mountain's statements there certainly seem to say that. There's the complexity here that, in general, Iron Mountain does apparently provide both data storage, and colocation.
Part of the problem would be that we have "tech" journalists writing this who have never been on either side of the transaction directly as a colocation customer, or an ISP/datacenter/hosting company, and drafted/reviewed contracts for such services. Nor have they ever gone and like, personally laid hands on a 2U rackmount server in a cabinet in a colo.
I'm not sure that I could reasonably expect a "journalist" to have those qualifications, but they could at least attempt to interview a neutral third party in the colo industry who can explain the distinction.
This station was clearing over a million dollars after expenses, and they couldn't be arsed to take a backup of their data on this cloud service. I have a hard time feeling much sympathy for them. And now Iron Mountain has to referee the data being recovered without handing over others data in the process.
The utter cluelessness all around from start to finish is impressive.
If Iron Mountain made a deal with PBS-affiliate, they'd be breaking the terms of the contract with their customer (the fact it is defunct is just an asterisk). If you were an Iron Mountain client and one of your customers made an end run around you to go directly to Iron Mountain because they refused to pay your bill, you'd be pissed at Iron Mountain.
By solving this as they have done, Iron Mountain can assure other clients they will not just let a third person circumvent their clients. They can now say they only did it by a court order even if they were more than willing to accept the asterisk and do it on principle. Everyone is happy. Everyone is whole.
Thats my impression too. Iron mountain is doing exactly what I would want if I had data with them.
If you are wharehousing data, you dont just let your customer's customer come in, look around, and take whatever they want. Even if they have a sob story.
Also some articles and people on social media were unclear that it was just a single PBS station involved, rather than all PBS content. I saw comments that thought that all the archives for Sesame Street were lost.
Apparently PBS doesn't have some kind of massive conglomerate backup or archives of things that its member stations produce. It's up to every station to archive or back up their own stuff. No standardization as to storage formats, NASes, tapes, or anything.
Sometimes things are transmitted in painful ways like rebroadcast in off hours over microwave links between members so they can re-record what they lost.
Plus the endless game of "hey does anyone have that one obscure episode of this one program we made?" to your colleagues...
What is unfair? It seems completely normal and what courts are for sorting out. Iron mountain is acting exactly how I would want them to if they were storing data for me.
If you are wharehousing data, you dont just let your customer's customer come in, look around, and take whatever they want. Even if they have a sob story.
How about before the company went out of business? Wasn’t there an avenue for the CEO to send a letter to Iron Mountain mapping the data and its owners or make an arrangement to legally transfer the data or something?
It's unfair when you do the best you can and suddenly you have to talk to lawyers.
> It's unfair when you do the best you can and suddenly you have to talk to lawyers.
Well, PBS didn't do the best they could. Aside from the whole "one backup is no backup" problem, they should've immediately made a new backup as soon as Open Source Storage didn't want to renew their contract and stopped responding to them.
That's the most screaming-red, horn-blaring, strobe-flashing warning that they were about to go under and it was the job of the PBS CTO (who makes $180k/yr) to recognise that and respond immediately (i.e., send an intern the nearest Best Buy to buy 70TB of HDDs, they'd probably still have enough space in their bag to do a coffee run on the way back).
> Wasn’t there an avenue for the CEO to send a letter to Iron Mountain mapping the data and its owners or make an arrangement to legally transfer the data or something?
And if they made a mistake in providing the mapping (given they were a company going bankrupt, they might have trouble getting staff to do a complete audit of their customer data)? I would be surprised if the CEO would not be personally on the hook if they violated HIPPA or some other privacy requirement by allowing PBS access to something by accident.
Once a company is in dire straits like this, sussing out problems like this is one of the main purposes of the legal system. Ironically, 70TB of HDDs probably cost less than the first demand letter PBS sent to Iron Mountain.
This is so much their own damn fault. I have no idea what level of utter incompetency it takes to think that a cloud storage system is a safe way to store the only copy of your data. And as you said, on top of that all the warning signs were there that you need to take a backup right away.
Holy hell, I wanted to see them get this suit dismissed just to rub it into their faces what mindless idiots were in charge of this fiasco.
> These archives represent an important part of our region’s history, and we look forward to ensuring their preservation and protection through the Court-approved process.
Apparently you didn't find them important enough to have backups. Hundreds of thousands of dollars a year in net income and they couldn't afford a cheap NAS with 4 disks. Incredible.
They did, they contracted with a vendor to store and backup the data. If that is insufficient then just about every customer of AWS, Azure, GCP, OCP, and every other managed storage provider is guilty of the same sin.
I have to agree with GP here, this is pretty incredible. They contracted with a vendor to store their data, not to back it up. It's not a backup if you can lose it in the same instant you lose the data.
You're right, they are. If your data can't survive a single storage host vanishing from existence, you don't have a backup. If your data can't survive one predictable or regularly occurring catastrophic act of nature, you don't have a backup. If your data can't survive a piece of malware -- with all of the credentials you have -- erasing it, you don't have a backup.
EDIT: Also, if you don't regularly test that your backups actually work, you probably don't have a backup. Lots of companies learn that one the hard way.
> If that is insufficient then just about every customer of AWS, Azure, GCP, OCP, and every other managed storage provider is guilty of the same sin.
Yes, they are as guilty! You say that as if it's a ridiculous assertion. Have you not read the occasional HN submission where someone is locked out of their cloud provider for dubious reasons?
I don't understand why you were downvoted. There is no excuse for them to not have local copies. In the context of video production 55TB is nothing. They must have local storage that's significantly larger than that already.
I think stories like these highlight the need for clearer (not necessarily more) regulations around contractor/subcontractor/client relationships and what happens when one of them goes tits up.
There were rampant issues in the fintech world that exploded when Synapse, a banking as a service provider, went bankrupt and their ledger didn't match what partner banks had in their accounts. End users were told "your deposits are FDIC insured", but in bankruptcy court the judge was sort of at a loss over how to rectify things - the banks weren't insolvent, and the FDIC (rightfully) said "hey, this isn't our problem, our regulated entities are in compliance". Looks like a similar situation happened here, where the contractors are both doing the "not it" thing.
I feel like a lot of tech innovation and "business process innovation" over the past 15 years was just ignoring regulations that were built up over decades, only discovering the reasons for those regulations when the tide went out and seeing that lots of companies had been pantsless the whole time.
Whoever claims "your deposits are FDIC insured" needs to be prosecuted as fraud and scam artists. “Your” the company’s deposits in the bank is FDIC insured. My deposit with you the company is not. When the bank goes belly up, your deposit is FDIC secured up to the account limit which is tiny in the scale of things. When your company goes belly up, my deposit is gone.
My understanding is that they split the accounts to keep up with the limit. Otherwise though yeah, FDIC will only step in for the extreme minority scenario of failure by the underlying bank. You absolutely should not be allowed to advertise FDIC insurance unless it goes all the way to the consumer.
> When the bank goes belly up, your deposit is FDIC secured up to the account limit which is tiny in the scale of things.
The FDIC is meant to protect individual people from loosing all of their money from the collapse of a bank, currently at $250k. If you have more wealth than that yet have it all as cash in a single account, then, you're pretty much an ID10T. For regular mere mortals, that's a helluva lot better than a bank telling you to pound sand when they collapse. If you're a business thinking the gov't is meant to protect you, then you are also delusional.
> If you have more wealth than that yet have it all as cash in a single account, then, you're pretty much an ID10T.
SVB collapse has shown that the 250k limit is basically not relevant. Maybe if a big consumer bank like Chase failed then 250k would be the max paid out, but we haven't seen that.
And OP is referring to Synapse, where the FDIC could not help any of the americans who lost their savings because the underlying banks didn't fail. https://www.cnbc.com/2024/11/22/synapse-bankruptcy-thousands...
> SVB collapse has shown that the 250k limit is basically not relevant
Only as long as you're too big to fail...
> … was just ignoring regulations that were built up over decades, only discovering the reasons for those regulations …
yeah it’s unreal to me how many people who imagine themselves intelligent are just now discovering the equivalent to why we make wheels round.
they never think to ask “why does regulation x exist?”
its absolutely crazypants.
> they never think to ask “why does regulation x exist?”
They do, but they think the answer is always some riff on, "government overreach because bureaucrats need to justify their jobs."
That attitude has been firmly ingrained into (at least) a generation of people.
TBF, a solid 50% of regulation is probably government overreach. The problem is that it's intrinsically tangled with the other 50% (crossing various specializations/laws/departments) that isn't; you can't undo one without undoing both.
love the use of "solid" when both numbers (50, and the other 50) are pure unobtanium pulled out of an orifice.
"a lot" would be less rhetorically satisfying, but probably more appropriate.
I suspect the number is in fact, far less than 50%. In mining engineering, I'd be amazed if a significant majority of the rules don't stem from a significant accident or death, or forseeable need to avoid them. In medicine, the stakes are equally high. Building codes? It depends. The cost of tunnelling in NY isn't because of government compliance.
In building codes, sometimes there's a tragedy that causes government overreach. A death with big public support often causes regulation that "does something" even if the rule itself doesn't help or make sense.
Propaganda works.
"your deposits are FDIC insured" claim was a lie, for anyone building on top of Synapse
Why do you think the laws and regulations are unclear?
This seems to be following the completely standard and expected process.
Contractor goes belly up, so you go to court and a judge who confirms they were a custodian of your data and you are entitled to retreive it.
Same would be the case if I was leasing equipment to someone and they had it at a storage lot. If the middle party dies or goes bankrupt, I get a court order to claim it from their other possessions.
It’s already been months in this case. And, court cases often take many years to resolve.
The issue is more that the court is slow and expensive even in the best case.
Its been 3 weeks, and they got what they need from the courts.
> The St. Louis station sued the information management company July 28
How much do you think it cost? A few grand?
Maybe we could replace this by reinventing the wheel, but i dont think it would be better. We could put it on blockchain and let a Peter Theil company handle arbitration.
If you talk to a lawyer (or, worse, a legislator, many of whom are also lawyers and all of whom are surrounded by them), they will insist up and down that this is a pretty standard custodial arrangement dispute and that the system worked as intended. And I would, very begrudgingly agree with that.
The legal system is perfectly capable of recognizing stolen property no matter how many layers of abstraction you put it through. The problem is always in the fact that the dispute resolution process is too expensive[0] to be useful. If you are defrauded for $10,000; but the legal fees for your representation will exceed that; then that juice ain't worth the squeeze. See also: Bricks and Minifigs.
In the Nine PBS case the judge correctly recognized Iron Mountain as a constructive bailee of Nine PBS's property and created a framework to retrieve their data. The problem is that this took almost half a year of legal work to get to the obvious outcome to make Nine PBS whole.
In Synapse's case, the problem is slightly different, because Synapse is not a bank, they are a reseller of banking services. That's the whole idea behind "fintech[1]" - that we can sell banking services while dodging all the regulatory compliance designed specifically to stop these kinds of issues so long as a real bank is involved. Saying their deposits are FDIC insured is like saying you have auto insurance because you happen to be riding a taxi. Technically correct but misleading and fraudulent. FDIC insurance doesn't cascade into your customers' accounts, because if it did, you'd be a bank.
[0] There's a similar problem with Bitcoin, where only a certain number of transactions can ever be processed per hour and thus it bottlenecks any higher-layer process that intends to use the Bitcoin blockchain as a settlement or dispute resolution system.
[1] "Fintech" in particular is meaningless as all banks are tech companies. They were one of the first adopters of electronic computers, online transaction processing, and a whole load of other things that seem utterly quaint now.
EDIT: changed "years" to "almost half a year", I was too lazy to do another Google search
> The problem is that this took years of legal work to get to the obvious outcome to make Nine PBS whole.
Not years. This whole "saga" has been going on for 5 months, and the suit against Iron Mountain was only filed on 28 July, so it just took weeks to come to this current arrangement.
EDIT: Toned down the comment. Leaving the rest, though, since I can't delete it with the reply below.
I changed it.
IIUC "FDIC insurance doesn't cascade into your customers' accounts" is not true in all cases, pass-through deposit insurance is a thing (https://www.fdic.gov/financial-institution-employees-guide-d...).
The problem in the case of Synapse was that they said they were doing that stuff, but were lying. (I think. The details of what happened there are apparently still not public. Also their bank seems to have been doing some sketchy things too.)
The biggest mess with Synapse seems to be the ledgers disagreeing: The banks, Synapse, Synapse's customers, and Synapse's customer's customers don't all agree on whose money is whose, and worse, the totals don't seem to add up.
> FDIC insurance doesn't cascade into your customers' accounts, because if it did, you'd be a bank.
It does cascade. It's called pass-through deposit insurance, it's codified in 12 CFR § 330.5 and 330.7. It has existed since the founding of the FDIC. Its enabling statute expressly provided that deposit insurance should be calculated based on the beneficial owners of a deposit account, regardless of in whose name the account is. [0]
Common arrangements include: HSAs, HOA accounts, UTMA/UGMA accounts, guardians and conservators, mortgage servicing accounts, escrow and title agents, payroll processors, brokerage cash sweep programs, prepaid cards, and yes, fintechs. [1]
[0] https://www.fdic.gov/notice-proposed-rulemaking-custodial-de...
[1] https://www.fdic.gov/financial-institution-employees-guide-d...
Open Source Storage (OSS), the vendor in question, was around for two decades before going out of business last year. The first and last archived versions of its web site:
https://web.archive.org/web/20040628023451/https://www.ossto...
https://web.archive.org/web/20250329140721/https://www.ossto...
(The first version isn't too exciting. It's a broken Flash site.)
> William Cravens, the attorney representing Iron Mountain, told the judge his client doesn’t know the format of Nine PBS’ materials that were stored by OSS. He expressed concern about whether Nine PBS’ archival material is lumped together with data from other OSS clients. Iron Mountain wants to avoid potentially corrupting the other data, Cravens added.
> Elliff ordered the immediate return of any physical devices that hold Nine PBS’ data once access to OSS’ storage system is granted…
> Once Nine PBS retrieves its data, the station must work with a third party to ensure that no data from other OSS customers is among those materials.
I don’t love this call. Handing over any disk has several problems. It could easily contain very sensitive data from another client. It’s inherently one-sided (another client can’t make the same request if the disk is already gone). And it’s being handed over to a PBS station that doesn’t seem to have the technical chops to manage data backups/security.
I want them to get their data back, but this needs a technical intermediary that handles all client data equally.
Is that statement by Iron Mountain's attorney believable, though?
If cage or server access was shared between Iron Mountain customers, how can Iron Mountain not have a record of what's in what cage or what's on what server?
If Iron Mountain customers like OSS were free to add or remove things without Iron Mountain's knowledge, then their (OSS's) access would've been limited to cages or servers dedicated to their (OSS's) own clients' data. Allowing an OSS ex-employee the same access they previously would've had should be a non-issue, from Iron Mountain's perspective.
The court got this right. This is when you need a special master to help with cleanup after a bankruptcy.
We saw something like this when TechShop went bankrupt, with member property in storage on the premises. The bankruptcy trustee set up a procedure where former members could make an appointment to retrieve their property, escorted by a representative of the trustee. Huge pain, but necessary.
Earlier coverage (St. Louis KETC / Nine PBS sues Iron Mountain for data access):
"Nine PBS sues Iron Mountain over blocked access to archival data" <https://news.ycombinator.com/item?id=49285418>
This is working out largely as I'd suggested it should, albeit with a court's intervention. See: <https://news.ycombinator.com/item?id=49293058>.
I'd still argue that I.M. should have anticipated this situation, and have some process in place to address it. "See a court" might well be part of that process.
I'm a bit confused from Iron Mountain's response about how they are worried that data could be co-mingled with other data? That would be an interesting way to back up data.
Iron Mountain probably only knows what data was owned by Open Source Storage, but not on the level of which data belongs to which Open Source Storage customer.
If so, it would be incredibly reckless for Iron Mountain to provide all of the data stored by Open Source Storage to one of Open Source Storage's former customers (PBS). Presumably that is why they wanted to go through the legal process so that someone else would be vested with the legal authority to determine what data PBS can legally access.
No idea how OSS and Iron Mountain were doing things so this is pure speculation, but if for example Iron Mountain were providing an object storage platform on which OSS then ran some secondary platform that stored all the data in one place with separate metadata to figure out what's what then it's entirely reasonable that they might not be able to easily separate it out. Not saying that'd be a good way to do it, but it's easily plausible
I do something similar with my clients backups, each individual client gets their own bucket on B2 for many obvious reasons but to sort out anything beyond what's obvious from the bucket name you need the backup software and appropriate encryption keys.
The ruling seems fair and reasonable, and I'm surprised they couldn't come to the same conclusion without a court.
Sometimes these things can go better for companies like Iron Mountain when there's a court order/decision in place to cover them. Nine PBS, in order to access their own data, could end up accessing data belonging to other people and that may not be properly covered under existing contracts and policies. It's a risk for Iron Mountain if this happens. Having a court decision and court set procedures that essentially force them to participate and also establishes third party review of the data will give them some cover if one of those other clients of the now defunct OSS discovers that Nine PBS accessed or inadvertently retained their data.
A court is needed for cover since it is possible that sometimes else's data will be seen or even corrupted in the process. This way they can say they were doing what the court asked for. That is legal cover for everyone to do what they want. Also legal limits on what they can do.
That is how I interpret the article anyway
They couldn't really - I doubt Iron Mountain actually objected to giving the data, but in the end it probably wasn't in a position to know what data belonged to PBS and what belonged to other clients of OSS and you'd get very worried if a data storage company gave a companies data to someone else without authorisation.
An Iron Mountain employee is quoted in this article saying basically that.
It’s likely that Iron Mountain needed a court order in order to do this with legal cover.
It’s just a data warehouse and OSS likely had multiple clients data and backups. Iron Mountain can’t let one third party go searching through a defunct customers stuff that has a bunch of additional third parties property intermingled with it. Too many unknowns and potentially litigious third parties.
https://archive.is/OBGDn
I previously commented on this when a different, earlier news article and it wasn't clear at the time if the storage company was a colocation customer, or a dedicated server customer of Iron Mountain:
=============================================
(from 4 days ago)
https://news.ycombinator.com/item?id=49293326
The news article is really not clear about whether this was, relative to the company "OSS":
a) OSS is a colocation customer with its own hardware colocating it inside an Iron Mountain datacenter. In which OSS owned the bare metal and paid iron mountain for rack space and power.
or
b) OSS is a dedicated hardware customer of Iron Mountain running a service on bare metal owned by Iron Mountain, and has gone defunct, leaving behind a bunch of servers/storage arrays that would in normal circumstances get wiped/reprovisioned.
From the point of view of a customer of OSS (PBS), that's two extremely different things.
If it's scenario A, I don't see how PBS has any claim against Iron Mountain. Your typical datacenter colo host for bare metal hardware owned by a customer has no involvement whatsoever in the condition or operation of the data, operating system, filesystems, RAID arrays, ZFS, etc of how the customer has set up their environment. Nor any ability to do anything with it. A colocation host that hasn't been paid for its rack space and power will typically have clauses in its colo contracts allowing for seizure and sale of abandoned hardware after a certain period of time.
====================================
New commentary:
The new news article seems to shed a bit more light on it, it sounds like it's scenario B, in which OSS never owned the hardware (they just set up the operating system/software config on it), and was renting it from Iron Mountain. In this scenario it's much more realistic to expect to be able to get some data back. And as the new news article says, PBS is paying all of the overdue bills in order to be able to do so.
Ordinarily a dedicated server provider that is renting rackmount bare metal hardware to people has a very automated and fast re-provisioning process, if you don't pay your bill, the terms of the contract let them wipe the servers within a fairly short amount of time and reallocat them to new customers.
Now if it had been scenario A, I think that there would have been very little opportunity for judicial remedy in an order requiring Iron Mountain to do anything, because Iron Mountain would have had no control over what a failing/bankrupt/dead colocation customer did with the data on their servers. PBS might have had a claim against the principals of the OSS company, but then you get into the "blood from a stone" problem of trying to enforce a judgment against people who have no assets or ability to pay.
For some reason, despite these things being rather simple and concrete distinctions, all the reporting around the case keeps being confusing, vague, and contradictory. I had been rather convinced by the Ars Technica article [1] that Iron Mountain was just the data center operator and this was colocation of completely OSS-owned and managed equipment. Iron Mountain's statements there certainly seem to say that. There's the complexity here that, in general, Iron Mountain does apparently provide both data storage, and colocation.
[1]: https://arstechnica.com/information-technology/2026/08/pbs-s...
Part of the problem would be that we have "tech" journalists writing this who have never been on either side of the transaction directly as a colocation customer, or an ISP/datacenter/hosting company, and drafted/reviewed contracts for such services. Nor have they ever gone and like, personally laid hands on a 2U rackmount server in a cabinet in a colo.
I'm not sure that I could reasonably expect a "journalist" to have those qualifications, but they could at least attempt to interview a neutral third party in the colo industry who can explain the distinction.
This station was clearing over a million dollars after expenses, and they couldn't be arsed to take a backup of their data on this cloud service. I have a hard time feeling much sympathy for them. And now Iron Mountain has to referee the data being recovered without handing over others data in the process.
The utter cluelessness all around from start to finish is impressive.
Im curious why this topic has been popping up in the news. It seems like a bog standard contractor/subcontractor obligation case.
If Iron Mountain made a deal with PBS-affiliate, they'd be breaking the terms of the contract with their customer (the fact it is defunct is just an asterisk). If you were an Iron Mountain client and one of your customers made an end run around you to go directly to Iron Mountain because they refused to pay your bill, you'd be pissed at Iron Mountain.
By solving this as they have done, Iron Mountain can assure other clients they will not just let a third person circumvent their clients. They can now say they only did it by a court order even if they were more than willing to accept the asterisk and do it on principle. Everyone is happy. Everyone is whole.
Thats my impression too. Iron mountain is doing exactly what I would want if I had data with them.
If you are wharehousing data, you dont just let your customer's customer come in, look around, and take whatever they want. Even if they have a sob story.
You ask for a court order with specifics.
The initial reporting implied it was some catastrophic data loss, like https://en.wikipedia.org/wiki/2008_Universal_Studios_fire. The truth, while concerning, is a lot more boring.
Also some articles and people on social media were unclear that it was just a single PBS station involved, rather than all PBS content. I saw comments that thought that all the archives for Sesame Street were lost.
Apparently PBS doesn't have some kind of massive conglomerate backup or archives of things that its member stations produce. It's up to every station to archive or back up their own stuff. No standardization as to storage formats, NASes, tapes, or anything.
Sometimes things are transmitted in painful ways like rebroadcast in off hours over microwave links between members so they can re-record what they lost.
Plus the endless game of "hey does anyone have that one obscure episode of this one program we made?" to your colleagues...
It's the original torrent network
Archival/secure storage is big business with big liability headaches if you make a bad decisiion.
A non-profit in a deeply unfair situation tugs at the heartstrings.
What is unfair? It seems completely normal and what courts are for sorting out. Iron mountain is acting exactly how I would want them to if they were storing data for me.
If you are wharehousing data, you dont just let your customer's customer come in, look around, and take whatever they want. Even if they have a sob story.
You ask for a court order with specifics.
How about before the company went out of business? Wasn’t there an avenue for the CEO to send a letter to Iron Mountain mapping the data and its owners or make an arrangement to legally transfer the data or something?
It's unfair when you do the best you can and suddenly you have to talk to lawyers.
> It's unfair when you do the best you can and suddenly you have to talk to lawyers.
Well, PBS didn't do the best they could. Aside from the whole "one backup is no backup" problem, they should've immediately made a new backup as soon as Open Source Storage didn't want to renew their contract and stopped responding to them.
That's the most screaming-red, horn-blaring, strobe-flashing warning that they were about to go under and it was the job of the PBS CTO (who makes $180k/yr) to recognise that and respond immediately (i.e., send an intern the nearest Best Buy to buy 70TB of HDDs, they'd probably still have enough space in their bag to do a coffee run on the way back).
> Wasn’t there an avenue for the CEO to send a letter to Iron Mountain mapping the data and its owners or make an arrangement to legally transfer the data or something?
And if they made a mistake in providing the mapping (given they were a company going bankrupt, they might have trouble getting staff to do a complete audit of their customer data)? I would be surprised if the CEO would not be personally on the hook if they violated HIPPA or some other privacy requirement by allowing PBS access to something by accident.
Once a company is in dire straits like this, sussing out problems like this is one of the main purposes of the legal system. Ironically, 70TB of HDDs probably cost less than the first demand letter PBS sent to Iron Mountain.
This is so much their own damn fault. I have no idea what level of utter incompetency it takes to think that a cloud storage system is a safe way to store the only copy of your data. And as you said, on top of that all the warning signs were there that you need to take a backup right away.
Holy hell, I wanted to see them get this suit dismissed just to rub it into their faces what mindless idiots were in charge of this fiasco.
The middleman fucked PBS, but everyone else is doing the reasonable job provided the lemons they have been given. It doesn't have to be a big deal.
This audience thinks data center operators are Jedi.
> These archives represent an important part of our region’s history, and we look forward to ensuring their preservation and protection through the Court-approved process.
Apparently you didn't find them important enough to have backups. Hundreds of thousands of dollars a year in net income and they couldn't afford a cheap NAS with 4 disks. Incredible.
They did, they contracted with a vendor to store and backup the data. If that is insufficient then just about every customer of AWS, Azure, GCP, OCP, and every other managed storage provider is guilty of the same sin.
I have to agree with GP here, this is pretty incredible. They contracted with a vendor to store their data, not to back it up. It's not a backup if you can lose it in the same instant you lose the data.
You're right, they are. If your data can't survive a single storage host vanishing from existence, you don't have a backup. If your data can't survive one predictable or regularly occurring catastrophic act of nature, you don't have a backup. If your data can't survive a piece of malware -- with all of the credentials you have -- erasing it, you don't have a backup.
EDIT: Also, if you don't regularly test that your backups actually work, you probably don't have a backup. Lots of companies learn that one the hard way.
If you do not have at least one, but usually two copies of a backup in your physical possession, you do not have a backup. |o-o|
> If that is insufficient then just about every customer of AWS, Azure, GCP, OCP, and every other managed storage provider is guilty of the same sin.
Yes, they are as guilty! You say that as if it's a ridiculous assertion. Have you not read the occasional HN submission where someone is locked out of their cloud provider for dubious reasons?
A single vendor to handle the primary storage and all backups is insufficient for your most critical data.
But where are you seeing this vendor was tasked with backups? This looks like a single copy situation.
Then that is just the 1 in 3-2-1 backups, and neglecting the 3 & 2 is a sin (wrt backup strategies).
I wonder the proportion of companies who store backups with a minimum of two separate vendors. Low?
I don't understand why you were downvoted. There is no excuse for them to not have local copies. In the context of video production 55TB is nothing. They must have local storage that's significantly larger than that already.