To the best of my knowledge much of this originated with SecDB/Slang at Goldman - SecDB (securities db I believe) being the object store and slang the somewhat quirky C like language that ran with it (also the only language I’ve used professionally that let you have spaces in the variable names).
Some of the folk that built that (or worked on it) ended up at JPM and Merrill where they built the Python centric version - Alpha and Quartz respectively. Barclays Capital has/had a similar system as well I think, but it’s not one I know about offhand - they did though, memorably, have a system that was pretty much Haskell-in-Excel.
What a well-written account of "how things are done".
> Time to drop a bit of a bombshell: the [Barbara] source code is in Barbara too, not on disk. Remain composed. It's kept in a special Barbara ring called sourcecode.
When first encountering these ecosystems and looking at the various pieces they contain, one may repeatedly ask: "why didn't they just use <off-the-shelf solution> for this problem instead of writing this component/subsystem from scratch"?
The answer is often that the battle-hardened mature off-the-shelf solution did not exist at the time the code was written. You're doing software archaeology.
That's only half the answer. These large investment banks' value-add is partly that they can integrate everything they know into these closed-world environments (kind of like a Smalltalk image), which is something that simply isn't done in the wider world because you can't accrete it out of smaller pieces and it doesn't make sense at all for smaller entities.
People turning up in hedge funds (i.e. much smaller) and trying to rewrite the bit of a bank they used to work in's equivalent of this article is so annoying.
Morgan Stanley's version is open-source at https://github.com/morganstanley/optimus-cirrus , although I don't know how practical it is to actually run yourself. (They don't go quite as far as having the code itself be bitemporal and kept in the datastore, but most of the stuff in the article exists there)
Well I doubt these solutions are very useful outside; from what I read what they have is a universal data store (that isn't hard to implement using current off-the-shelf OSS), something for financial instruments that has a compositional nature (you won't encounter much of that in the outside world), plus some other quirky features.
I've seen similar inside large financial orgs - what struck me was how there are these huge amounts of people that spend their entire working life inside this alternate IT reality. It's not unlike SAP consultants where their skillset is tied to one company.
Also...these things tend to have fuckin terrible documentation. Good luck figuring any of this out. And you can't google it and your AI is just as lost as you
lolol. Actually, I find AI has a reasonable chance to figure it out, as long as you point to the right source code.
BTW to me these quirks actually can be used as some kind of job security. If it takes a year to onboard someone to do meaningful work, it sure raise the cost of firing.
>Applications also commonly store their internal state in Barbara - writing dataclasses straight in and out with only very simple locking and transactions (if any).
Right out of the gates, it's crazy how this contrasts with Mercury's Haskell infra
I may be reading between lines, but temporal seems to be a virtual machine like the evm, it handles computation.
Barbara is a company wide database, it handles data storage.
When I read about internal app state being stored in Barbara I'm interpreting that the policy is for the data to be centralized for more vertical control.
While the Temporal thing sounds like if something is written, it's done so in a containerized like manner, and other processes can't just read it.
> temporal seems to be a virtual machine like the evm, it handles computation.
It stores the app's work-in-progress state as well (probably as a blob full of serialised internal datastructures, at least in some cases):
> You write your workflow as ordinary sequential code, and the platform records every step in an event history. If a worker crashes mid-workflow, another worker replays the deterministic prefix to reconstruct the state, then continues from where it left off.
> When I read about internal app state being stored in Barbara I'm interpreting that the policy is for the data to be centralized for more vertical control.
That wasn't the way I experienced it, if anything it was the opposite: app developers would push to use Barbara for their internal state because it was easy: the app is already accessing it, the APIs are simple, and since it's just pickled objects you can just store your state without having to worry about serialisation (much) or ORM. Whereas policy and leadership would if anything prefer you to use a separate traditional database. The point of Barbara is to provide a unified interface onto "everything the bank knows", it's primarily for data that multiple teams use, not internal state owned by a single team.
Eh, to be fair, this post is about a _bank_, and the one you've linked is about _fintech_. They are not even close to the same space, even though they both deal with money.
But also I suppose you may be saying exactly this?
Well, I say "more or less" :). But the fact that there is a single global database doesn't mean that you can read every key or value (but generally, yes, you can _read_ everything). I mention in passing prolog-style permission systems for evaluating perms.
But anyway, specific trades are rarely private to one part of the bank for many reasons. For example regulatory: these days you have to notify the regulator about every trade.
What I imply from the description is that, the default ring contains some shared global public data (e.g. a cache of bloomberg informations), and each individual team will have their own rings. Afterall there's no that many you can fit into 16mb
To the best of my knowledge much of this originated with SecDB/Slang at Goldman - SecDB (securities db I believe) being the object store and slang the somewhat quirky C like language that ran with it (also the only language I’ve used professionally that let you have spaces in the variable names).
Some of the folk that built that (or worked on it) ended up at JPM and Merrill where they built the Python centric version - Alpha and Quartz respectively. Barclays Capital has/had a similar system as well I think, but it’s not one I know about offhand - they did though, memorably, have a system that was pretty much Haskell-in-Excel.
JPM's version was Athena (not Alpha) [0]
[0] https://www.slideshare.net/slideshow/managing-python-at-scal...
What a well-written account of "how things are done".
> Time to drop a bit of a bombshell: the [Barbara] source code is in Barbara too, not on disk. Remain composed. It's kept in a special Barbara ring called sourcecode.
When first encountering these ecosystems and looking at the various pieces they contain, one may repeatedly ask: "why didn't they just use <off-the-shelf solution> for this problem instead of writing this component/subsystem from scratch"?
The answer is often that the battle-hardened mature off-the-shelf solution did not exist at the time the code was written. You're doing software archaeology.
That's only half the answer. These large investment banks' value-add is partly that they can integrate everything they know into these closed-world environments (kind of like a Smalltalk image), which is something that simply isn't done in the wider world because you can't accrete it out of smaller pieces and it doesn't make sense at all for smaller entities.
The previous discussion was fascinating: https://news.ycombinator.com/item?id=29104047
Does anyone working at one of these banks or similar know if this information still holds true?
And have any of the banks started using uv yet? Or will they forever be using pip?
They don't use pip, you just import the module and it is pulled from barbara
People turning up in hedge funds (i.e. much smaller) and trying to rewrite the bit of a bank they used to work in's equivalent of this article is so annoying.
I think it is a pity they’ll likely never open source any of this stuff
Of course, financial institutions have a lot of “secret sauce” - such as financial models - you’d never expect them to release.
But this kind of underlying infrastructure isn’t really “secret sauce”
Morgan Stanley's version is open-source at https://github.com/morganstanley/optimus-cirrus , although I don't know how practical it is to actually run yourself. (They don't go quite as far as having the code itself be bitemporal and kept in the datastore, but most of the stuff in the article exists there)
Well I doubt these solutions are very useful outside; from what I read what they have is a universal data store (that isn't hard to implement using current off-the-shelf OSS), something for financial instruments that has a compositional nature (you won't encounter much of that in the outside world), plus some other quirky features.
This infrastructure is more secret sauce than the financial models, which change rapidly.
> I think it is a pity they’ll likely never open source any of this stuf
The more they use cloud-hosted LLMs, the more likely it will get leaked into training data.
> This is because clients generally do not ring up about pennies.
I’ve had clients ring up about pennies… it can be crazy what some people are motivated by
Precision?
Quite common in accounting, the accounting equation must balance, it's like a checksum
Prior discussion: https://news.ycombinator.com/item?id=29104047
And I thought rewriting 3rd party packages to work with AFS was crazy
I've seen similar inside large financial orgs - what struck me was how there are these huge amounts of people that spend their entire working life inside this alternate IT reality. It's not unlike SAP consultants where their skillset is tied to one company.
Also...these things tend to have fuckin terrible documentation. Good luck figuring any of this out. And you can't google it and your AI is just as lost as you
lolol. Actually, I find AI has a reasonable chance to figure it out, as long as you point to the right source code. BTW to me these quirks actually can be used as some kind of job security. If it takes a year to onboard someone to do meaningful work, it sure raise the cost of firing.
Weirdly not dissimilar from MUMPS systems.
>Applications also commonly store their internal state in Barbara - writing dataclasses straight in and out with only very simple locking and transactions (if any).
Right out of the gates, it's crazy how this contrasts with Mercury's Haskell infra
https://blog.haskell.org/a-couple-million-lines-of-haskell/
It sounds pretty similar actually. Barbara fills the same role that Temporal is doing at Mercury.
I may be reading between lines, but temporal seems to be a virtual machine like the evm, it handles computation.
Barbara is a company wide database, it handles data storage.
When I read about internal app state being stored in Barbara I'm interpreting that the policy is for the data to be centralized for more vertical control.
While the Temporal thing sounds like if something is written, it's done so in a containerized like manner, and other processes can't just read it.
> temporal seems to be a virtual machine like the evm, it handles computation.
It stores the app's work-in-progress state as well (probably as a blob full of serialised internal datastructures, at least in some cases):
> You write your workflow as ordinary sequential code, and the platform records every step in an event history. If a worker crashes mid-workflow, another worker replays the deterministic prefix to reconstruct the state, then continues from where it left off.
> When I read about internal app state being stored in Barbara I'm interpreting that the policy is for the data to be centralized for more vertical control.
That wasn't the way I experienced it, if anything it was the opposite: app developers would push to use Barbara for their internal state because it was easy: the app is already accessing it, the APIs are simple, and since it's just pickled objects you can just store your state without having to worry about serialisation (much) or ORM. Whereas policy and leadership would if anything prefer you to use a separate traditional database. The point of Barbara is to provide a unified interface onto "everything the bank knows", it's primarily for data that multiple teams use, not internal state owned by a single team.
Eh, to be fair, this post is about a _bank_, and the one you've linked is about _fintech_. They are not even close to the same space, even though they both deal with money.
But also I suppose you may be saying exactly this?
to be fair, it is a fintech that wants to become a bank
It's a bank, that is, a bank in all but name for regulatory purposes.
> but the default ring is more or less a single, global, object database for the entire bank.
Is this really the case? I'm sure there are plenty of transactions that for umpteen different reasons must not be exposed on a global level.
Well, I say "more or less" :). But the fact that there is a single global database doesn't mean that you can read every key or value (but generally, yes, you can _read_ everything). I mention in passing prolog-style permission systems for evaluating perms.
But anyway, specific trades are rarely private to one part of the bank for many reasons. For example regulatory: these days you have to notify the regulator about every trade.
What I imply from the description is that, the default ring contains some shared global public data (e.g. a cache of bloomberg informations), and each individual team will have their own rings. Afterall there's no that many you can fit into 16mb
No, no, there is a single world ring. the 16mb limit is for values