Thanks! "OLT" was also new to me. In case others find it helpful:
> OLT = Optical Line Terminal.
> In ISP fiber (typically GPON/EPON) infrastructure, it’s the provider-side device at the central office/headend that terminates and controls the passive optical network: it connects upstream into the ISP’s aggregation/core network and downstream via fiber (through splitters) to many customers’ ONTs/ONUs, handling PON line control, provisioning, QoS, and traffic aggregation.
I have been worked for a regional ISP 10 years ago and having an architecture like that one, would be a godsend.
With centralized BNGs we were not able to apply upstream QoS policies for subscribers on the backhaulings and we had to apply policies on DSLAM access ports.
We ended using a couple of cheap Mikrotik as PPPoE concentrators for every access room, in a similar way as you did. But the reliability of Mikrotik routers was not the best
The code is mostly vibe coded and under the BSL. I think the interesting bit here is a single developer can write something like this with an agent. Does it make sense to open source such a thing or just each ISP write their own to their requirements?
I also don’t get the focus on handling DHCP renewals in the kernel fast path. With 2000 subscribers per OLT and say a 5 minute lease time that’s only a few renewals a second.
I'm curious as to what actually is the CPU <-> NPU bandwidth in these whitebox OLTs? Traditionally that has been sized for small amounts of punted control plane packets, then programming a fast path into the NPU for revenue traffic.
There's a commercial product available from 6WIND that makes this much more supportable for mission-critical networks. It leverages DPDK and delivers excellent performance at scale.
This is very elegant, but is treading some ground that for various reasons never got commercial traction.
- Cisco tried distributed BNG about ten years ago, their "cnBNG" running on their x86 UCS server line. See [0]
- A UK company called Metaswitch tried doing this with eBPF and some home-grown tech (VPP meets fd.io and special sauce) in about 2018. Interestingly they pivoted the tech to work on 5G where blazing fast user plane is essential [1]. They got bought by Microsoft, ground into glass, and wiped out five years later.
- There was a lot of talk in ~2020 about whether wireline (fiber) and wireless (5G cellular) infrastructures could converge, with the BNG becoming another node in the system, like an AGF, and authenticating users against a UDR. 5G was already very distributed and it made a lot of sense at least on market-techture slide decks.[2]
Looking back, the difficulty making this commercial was not splitting up the function, making it performant, or running it disagg on commodity hardware. The difficulty was finding a set of anchor customers who were experiencing such pain on their existing BNG that they would be prepared to jump ship from their big iron to something new knowing full well that the new system would only support 10% of what their old Lucent 7750s or Ericsson boxes could do.
Taking disagg as an example, it makes little sense unless your network is above a certain size. But if you run a big network, like DT or AT&T, say, then you will demand hundreds of features be present before you will look at an alternative. Does it work with my OSS. Does it support all the features of RFC XYZ and the special tweaks that only we have. Will it keep the three-letter agencies happy when they serve a warrant. Can it pass muster with my security people. Can the developer survive working with my procurement people long enough to make enough money to fund development.
No disruptive vendor --none-- has ever made it past this barrier into the network core, despite operators saying for years that they want to work with disruptors. That's why Nokia, Ericsson, and Huawei reign supreme and telcos haven't innovated in decades.
In ossified companies like telcos there's also the issue that the limitations of the existing equipment are being worked around with people. Those people derive their salaries from it, their manager derives his salary + prestige from managing such a headcount, and so on.
While the top brass might indeed be interested and benefit from more automation and a network that mostly runs itself, it's a bad deal for effectively everyone else in the company, so any attempts in that direction will never end up anywhere.
That's why legacy companies have been talking about "digital transformation" for decades now, yet it never progresses past simply digitizing the paperwork (and often creating more of it due to reduced friction), because enough people derive their job from said paperwork to make actual digital transformation politically untenable and impossible to deliver due to constant sabotage.
I mean, you see this with MikroTik all the time. The recent L3HW-enabled devices (up to 400G now) are so good it's crazy, and European onshore manufacturing, too. However, it doesn't support a subset of legacy "Enterprise" features, even though there's always a way to do the same thing using different architecture to how ISP guys have been trained many years ago, so instead we hear all the time that it's inadequate.
5G is a breath of fresh air in the sense that a lot of new techniques and broadly-applicable architectures were introduced to ISP's. I'm telling you, they HATED it. They absolutely hate learning new thing and that may as well be the largest blocker for disruptive players in the market.
I love this you and the other guy conspiracy lol. Telco bas, Ericsson bad. Okay if your stuff is so good why is it not dominant? Ah yes its all a conspiracy.
"Hate learning new stuff" = This ISP, LTE, NR stuff is all fairly new lol
Not all ISP's are like that. MikroTik is used by many ISP's in Europe. Dominant? Not yet, but it's getting there! U.S. is not really popular these days, you know? Trust in the pesky, backdoored Cisco switches is at an all-time low, cost notwithstanding. This is pushing telcos to consider alternative architectures that do not require certain proprietary features pushed by the big-three switch and router manufacturers.
Sounds like a really cool endeavour. I had no idea that ISP infrastructure was so heavily centralised. Hope the author succeeds in their quest to improve on this. I love that they're using simpler, cheaper hardware for this. Essentially, it sounds like it could reduce vendor lock-in for ISPs.
The [ONT → OLT(+BNG)] → Internet] sections of the paths will continue to be owned by commercial entities that can still be the subject of court orders and/or government pressure.
Even if you were to roll your own cable in the ground to your own ONT/OLT/BNG at some point you will need to acquire IP transit or peering from other commercial entities.
The latter usually isn't that difficult, just expensive. You can usually rent a leased line from anywhere to anywhere. The government will still come knocking if they think you're evading their censorship.
A leased line though will only get you A<->B where sure, A and B can be anywhere but have to be concrete locations/hand off points when provisioned. It does ultimatley come down to the service that one orders from a commercial entity.
A hypothetical court order saying something like "kill internet access" would likely cause an IP transit service to stop working (implemented by said provider no longer announcing global IP routing tables to that service) but a leased line between two locations would likely remain untouched since that isn't an "internet" service. So they might not need to come knocking if they're reasonably confident that all such edge cases like leased lines end up at dead-ends because any internet-capable product they might be enabling access to is sufficiently disabled.
I do imagine though that if they get as far as "kill the internet" that obtaining a subsequent court order to go after some suspicious leased line would be trivial.
As a side note, I find that IP transit is typically the cheapest aspect of providing an internet service since a cross-connect at a well connected DC will cost well under $1/Mbps/month unmetered. Plus the cost is very well amortized when residential users are the target. This has tended to hold when one takes into account the co-lo costs as well since network gear doing relatively basic packet forwarding/internet table routing doesn't take up that much space or power.
Because TFA never bothered to define it:
Broadband Network Gateway (BNG)[1]
[1]: https://github.com/codelaboratoryltd/bng#bng-broadband-netwo...
Thanks! "OLT" was also new to me. In case others find it helpful:
> OLT = Optical Line Terminal.
> In ISP fiber (typically GPON/EPON) infrastructure, it’s the provider-side device at the central office/headend that terminates and controls the passive optical network: it connects upstream into the ISP’s aggregation/core network and downstream via fiber (through splitters) to many customers’ ONTs/ONUs, handling PON line control, provisioning, QoS, and traffic aggregation.
Thanks.. was reading the article like WTF is "BNG"
Is it the FTTX equivalent of a BRAS?
Yes, exactly. BRAS is functionally the same as BNG.
So what is BRAS?
https://en.wikipedia.org/wiki/Broadband_remote_access_server
I have been worked for a regional ISP 10 years ago and having an architecture like that one, would be a godsend. With centralized BNGs we were not able to apply upstream QoS policies for subscribers on the backhaulings and we had to apply policies on DSLAM access ports.
We ended using a couple of cheap Mikrotik as PPPoE concentrators for every access room, in a similar way as you did. But the reliability of Mikrotik routers was not the best
OpenWISP: https://news.ycombinator.com/item?id=42953414
From "Open Hardware Ethernet Switch project, part 1" https://news.ycombinator.com/item?id=43969857 :
> There are 48+2 port switches with OpenWRT support
Are there 48 port switches with 8 or more cores?
Edit:
From "Show HN: Spliff – Correlating XDP and TLS via eBPF (Building a Linux EDR)" (2026) https://news.ycombinator.com/item?id=46663446 :
> the current "Golden Thread" correlation architecture fundamentally requires userspace + kernel cooperation that can't be fully offloaded.
The code is mostly vibe coded and under the BSL. I think the interesting bit here is a single developer can write something like this with an agent. Does it make sense to open source such a thing or just each ISP write their own to their requirements?
I also don’t get the focus on handling DHCP renewals in the kernel fast path. With 2000 subscribers per OLT and say a 5 minute lease time that’s only a few renewals a second.
You might have many OLTs per rack and many racks per site.
I'm curious as to what actually is the CPU <-> NPU bandwidth in these whitebox OLTs? Traditionally that has been sized for small amounts of punted control plane packets, then programming a fast path into the NPU for revenue traffic.
There's a commercial product available from 6WIND that makes this much more supportable for mission-critical networks. It leverages DPDK and delivers excellent performance at scale.
https://www.6wind.com/vrouter-vsr-solutions/virtual-broadban...
This is very elegant, but is treading some ground that for various reasons never got commercial traction.
- Cisco tried distributed BNG about ten years ago, their "cnBNG" running on their x86 UCS server line. See [0]
- A UK company called Metaswitch tried doing this with eBPF and some home-grown tech (VPP meets fd.io and special sauce) in about 2018. Interestingly they pivoted the tech to work on 5G where blazing fast user plane is essential [1]. They got bought by Microsoft, ground into glass, and wiped out five years later.
- There was a lot of talk in ~2020 about whether wireline (fiber) and wireless (5G cellular) infrastructures could converge, with the BNG becoming another node in the system, like an AGF, and authenticating users against a UDR. 5G was already very distributed and it made a lot of sense at least on market-techture slide decks.[2]
Looking back, the difficulty making this commercial was not splitting up the function, making it performant, or running it disagg on commodity hardware. The difficulty was finding a set of anchor customers who were experiencing such pain on their existing BNG that they would be prepared to jump ship from their big iron to something new knowing full well that the new system would only support 10% of what their old Lucent 7750s or Ericsson boxes could do.
Taking disagg as an example, it makes little sense unless your network is above a certain size. But if you run a big network, like DT or AT&T, say, then you will demand hundreds of features be present before you will look at an alternative. Does it work with my OSS. Does it support all the features of RFC XYZ and the special tweaks that only we have. Will it keep the three-letter agencies happy when they serve a warrant. Can it pass muster with my security people. Can the developer survive working with my procurement people long enough to make enough money to fund development.
No disruptive vendor --none-- has ever made it past this barrier into the network core, despite operators saying for years that they want to work with disruptors. That's why Nokia, Ericsson, and Huawei reign supreme and telcos haven't innovated in decades.
[0] https://www.cisco.com/c/en/us/td/docs/routers/cnBNG/cnBNG-CP...
[1] https://www.linkedin.com/pulse/why-new-approach-pure-softwar...
[2] https://www.broadband-forum.org/pdfs/tr-470-2-0-0.pdf
In ossified companies like telcos there's also the issue that the limitations of the existing equipment are being worked around with people. Those people derive their salaries from it, their manager derives his salary + prestige from managing such a headcount, and so on.
While the top brass might indeed be interested and benefit from more automation and a network that mostly runs itself, it's a bad deal for effectively everyone else in the company, so any attempts in that direction will never end up anywhere.
That's why legacy companies have been talking about "digital transformation" for decades now, yet it never progresses past simply digitizing the paperwork (and often creating more of it due to reduced friction), because enough people derive their job from said paperwork to make actual digital transformation politically untenable and impossible to deliver due to constant sabotage.
I mean, you see this with MikroTik all the time. The recent L3HW-enabled devices (up to 400G now) are so good it's crazy, and European onshore manufacturing, too. However, it doesn't support a subset of legacy "Enterprise" features, even though there's always a way to do the same thing using different architecture to how ISP guys have been trained many years ago, so instead we hear all the time that it's inadequate.
5G is a breath of fresh air in the sense that a lot of new techniques and broadly-applicable architectures were introduced to ISP's. I'm telling you, they HATED it. They absolutely hate learning new thing and that may as well be the largest blocker for disruptive players in the market.
I love this you and the other guy conspiracy lol. Telco bas, Ericsson bad. Okay if your stuff is so good why is it not dominant? Ah yes its all a conspiracy.
"Hate learning new stuff" = This ISP, LTE, NR stuff is all fairly new lol
Not all ISP's are like that. MikroTik is used by many ISP's in Europe. Dominant? Not yet, but it's getting there! U.S. is not really popular these days, you know? Trust in the pesky, backdoored Cisco switches is at an all-time low, cost notwithstanding. This is pushing telcos to consider alternative architectures that do not require certain proprietary features pushed by the big-three switch and router manufacturers.
This is not rocket science.
Ericsson and Nokia are Swedish and Finnish.
I am Swedish.
American? What are you on about?
Sounds like a really cool endeavour. I had no idea that ISP infrastructure was so heavily centralised. Hope the author succeeds in their quest to improve on this. I love that they're using simpler, cheaper hardware for this. Essentially, it sounds like it could reduce vendor lock-in for ISPs.
Can Iran like internet ban happen? i feel the answer is no. We can finally escape govt sponsored censorship
The [ONT → OLT(+BNG)] → Internet] sections of the paths will continue to be owned by commercial entities that can still be the subject of court orders and/or government pressure.
Even if you were to roll your own cable in the ground to your own ONT/OLT/BNG at some point you will need to acquire IP transit or peering from other commercial entities.
The latter usually isn't that difficult, just expensive. You can usually rent a leased line from anywhere to anywhere. The government will still come knocking if they think you're evading their censorship.
A leased line though will only get you A<->B where sure, A and B can be anywhere but have to be concrete locations/hand off points when provisioned. It does ultimatley come down to the service that one orders from a commercial entity.
A hypothetical court order saying something like "kill internet access" would likely cause an IP transit service to stop working (implemented by said provider no longer announcing global IP routing tables to that service) but a leased line between two locations would likely remain untouched since that isn't an "internet" service. So they might not need to come knocking if they're reasonably confident that all such edge cases like leased lines end up at dead-ends because any internet-capable product they might be enabling access to is sufficiently disabled.
I do imagine though that if they get as far as "kill the internet" that obtaining a subsequent court order to go after some suspicious leased line would be trivial.
As a side note, I find that IP transit is typically the cheapest aspect of providing an internet service since a cross-connect at a well connected DC will cost well under $1/Mbps/month unmetered. Plus the cost is very well amortized when residential users are the target. This has tended to hold when one takes into account the co-lo costs as well since network gear doing relatively basic packet forwarding/internet table routing doesn't take up that much space or power.
It can always happen. The government would just have to arrest everyone who doesn't comply, like they do in Iran.
That could never happen here /s