> Censys ARC identified 4,148 Internet-exposed hosts that respond to EtherNet/IP and self-identify as Rockwell Automation/Allen-Bradley. The United States remains dominant at 71.0% (2,945 hosts), with Canada a clear second at 11.5% (476 hosts).
Describe the network security of the industrial automation industry and their customers in a single statement. Lol.
And how many more are on the same internal networks as dozens or hundreds of ordinary Windows desktops on which municipal workers are checking their email? Hardly better.
It’s far worse, just last week I was assessing some architecture and there’s still dial up and 3G connected devices in some of the most critical infrastructure around..
I don't see the issue with either of those things? At least as long as they're properly secured. (Which they probably aren't but that's neither here nor there.)
Well I think it speaks to the age of the equipment they are speaking of in the industrial sector.
How do you lockdown something that may have not been taken offline for decades because it will cost downtime or harm. Or something that can’t be locked down without tossing new tech around it that may not be compatible with the protocols etc.
Dial up into an air gapped network defeats the purpose of being air gapped. I would think the bare minimum standard is that there is no way to change anything in the control systems without being physically present at the facility, past it's physical security boundary.
It is an issue, dial up lacks tunnel encryption and if you managed to make it, it will be useless in real scenarios, and 3g is being phased out and obsolete
Secure the contents of the tunnel and it doesn't matter. Provide a secure backbone and it doesn't matter. Realize that dialup is so slow that you can transparently tunnel it across ~any modern network (for which you can encrypt the tunnel) and it doesn't matter.
Tunnel your dialup within your obsolete 3g network, and then tunnel that obsolete 3g network within something modern. The obsolete technologies are not the issue here.
Also the thing about dialup is that it's point to point so the attack surface isn't even remotely comparable to exposing a port on the open internet. I should generally be able to trust the link that my phone company provides. Faxes are still used in many secure settings in preference to email.
For 3G, it’s phased out, so the bands will be gone, you have to upgrade it.
The dial up part is far more involved, especially when they have auto answering connected directly to PLC or HMI, mostly with shared passwords. Also, you can’t trust the network operator either, insider threats and rogue employees are a threat. Additionally, dial ups are less monitored compared to modern network, and usually you end up with duct tape solutions like jump server to have strong authentication and continuous logging in firewall and such, plus proper encrypted tunnels so even physical wiretapping isn’t possible, and the assumption of air gap isn’t there because it’s reachable through public telephone, and the worst part, these dial ups are usually connected to windows XP Scada developers machines.
To add, obsolete is bad too, when your device cease to have vulnerability patches, you are screwed regardless of whatever configs you put.
I much prefer the non-vendor perspective on this. Andy Krapf, co-chair of the Water ISAC, has a great breakdown about the status quo systemic problems that water faces today.
There can be surprisingly few critical components and processes in a water and sewerage network due to gravity-fed designs (far more cost efficient), large buffers (water reservoirs, sewage sumps, etc) allowing intermittent operation of otherwise critical components, retained ability for humans to manually operate equipment, and also the fields of availability and safety engineering which typically prefer elimination of software failure modes by designing equipment to not rely upon software.
The aim of a water network is to:
1. Take water from a water source (elevated dam -- strongly preferred, river, ocean) and as much as possible, gravity feed it to a treatment plant.
2. Treat the water using processes that are simplified/fail-safe as much as possible and could be operated manually by humans if necessary. This is where availability and safety engineers would design equipment to not be dependent on software and instead use mechanical or analogue electronics control.
3. Pre-position treated water as much as possible at ~50-90m hydraulic head (~500-900kPA) above the water faucets where people want to use the treated water and provide a buffer for X days of usage. Any pumps between the treatment plant and elevated storage therefore only need to operate intermittently to refill the buffer.
Sewerage networks have similar aims:
1. Let sewage flow as much as possible downhill to the treatment plant via gravity. Where a rising main (elevation gain) is required, place a large enough sump for X hours/days of usage and pump up to higher elevation from the sump.
2. Treat the sewage using processes that are simplified/fail-safe as much as possible and could be operated manually by humans if necessary. For example, a compressor used for aeration can be manually switched on/off with a mechanical switch and plugged into a diesel generator, and not require someone logging in with multi-factor authentication to a laptop to issue a command to a PLC to turn on the compressor.
3. Design overflows into the system for emergency release of partially or untreated sewage, and practice this process as part of disaster recovery exercises. This is generally an aim arising due to risk assessment process that says building a $1bn sump with 8 independent pumps is cost prohibitive versus the 1-in-200 year chance of untreated sewage messing up a downstream river for a few weeks.
Ultimately a lot of the cybersecurity risk comes down to government appetite to accept 1-in-1000 (or whatever) year failure modes. Is it worth investing now in triple modular redundant automated control systems (mostly used in safety-critical sectors such as aviation and space), or installing a just-in-case diesel generator at every one of 500 pumping stations across a region, or building $10bn of sewage sumps to hold sewage for up to a month, or building 2 treatment plants instead of 1 and using different technology for each, or hiring and training more humans to regularly exercise manual control and operation of a network, etc? Or just accept that once every 1000 years, some water rationing may be required, or a downstream river will be polluted for a few weeks?
Sadly this instantly became a political football, with the states pointing fingers at Iran, but Trump was not wrong in this case. This is gross incompetence at all levels — IT malpractice if you will.
CISA and its predecessors have been warning utility operators about critical infrastructure vulnerabilities for what, 15 years at this point? That goes back to the first Obama administration.
Yet here we are in 2026 and these utilities are still connecting these things to the raw Internet with default passwords. You cross a threshold where you're being deliberately careless.
When you are putting more effort into securing your Plex server on your home network then public utilities are taking on machinery that dumps chemicals into the local water, something is not right and finger-pointing isn't going to fix it.
I think it's less carelessness and more the inability to attract (pay) people who have the technical knowhow to properly secure infrastructure. Even a lot of developers are poor network engineers and treat IT like magic at their own companies.
I've met info-sec / vulnerability researcher types that were egregiously reckless, like plugging Raspberry Pi's into the production network kind of thing.
Public sector has always paid low. But the problem is widespread, almost universal, and they've had a 15 year head start of the federal government telling them to get their shit together.
At some point it just became standard industry practice is my guess.
Upgrades to waste water are project based. Lowest bidder will not provide security for free. Security may be mentioned in spec but in hand waved language that can be hand waved away. That company doing the improvement project will have next to no documentation from the previous engineering effort. Just do bare minimum and move on to next job, because no one is getting paid enough to do put in more effort.
In all things when it fails in a drastic way the system will be corrected. People will die, it will suck, changes will be made.
Government is supposed to respond to these things and create incentives to correct. Telling a small municipality to do something without a carrot ir stick does nothing.
In this instance someone else will be providing the stick.
You’re on the right track, I think. But, I wouldn’t say it’s about the pay to attract competent workers. I think it has more to do with the incentive structures once you’re in. Incentives and performance management are fundamental problem in civil service. The incentives to set high standards and hold individuals accountable simply do not exist.
The one and only exception is the military, because lives are literally on the line in a way that is not at all abstract.
Usually when people say this they are dog whistling privatization. Which is the exact opposite thing people need in infrastructure, ask anyone who has to deal with PG&E.
Paying skilled people highly does actually incentivize people to do better work, especially if they are actually embedded into the community they are essentially working for. If being a civil servant was as "glorious" as being a techie is SF there would be a very different attitude around the work.
In an environment without accountability, higher pay is just more incentive to lay low and not take any personal risk. Why do anything other than the bare minimum when there is no upside? In a bureaucracy where responsibility is diffused and the culture is purely political and not merit/performance-based, few are willing to step out of line to do the right thing.
The people who climb to the top aren’t the ones who took a risk and got the reward. Quite the opposite, they’re the ones who learned to play the game and didn’t upset the power structure by rocking the boat. No one is going to tell the emperor he has no clothes when the path to power is political and has no grounding in reality.
Highly conscientious, intrinsically motivated people will do the right thing in any environment. And those people don’t last long in political/bureaucratic environments where the incentives are misaligned.
Your comments show, beyond a shadow of a doubt, that you have never worked for the federal government and likely have never worked in state or local government.
>In a bureaucracy where responsibility is diffused and the culture is purely political and not merit/performance-based, few are willing to step out of line to do the right thing.
Federal employment is merit based. Advances are earned, not doled out to the teacher's pet or the ass-kisser with the most perfect pucker. You have no idea what you're talking about.
I have no idea where you or anyone else got the idea that a government employee has no responsibility or incentives and works strictly for political ends and even worse, where none are willing to speak up. You clearly have no experience in that space and should focus your comments on things that you better understand through direct experience.
>And those people don’t last long in political/bureaucratic environments where the incentives are misaligned.
This is bullshit and you are grossly misinformed. Plenty of people fight the polarization of the system on political lines from within the system. They use the laws that are in place in order to prevent systemic abuses. They may not win every internal fight but they do go to battle for the right reasons.
I don't think your attitude here is in keeping with the guidelines (or constructive discourse for that matter). You've made a number of uncharitable assumptions about the other party on the back of which you then launched into baseless personal attacks.
Notably everything in the comment you replied to applies equally to the public and private sector. They are neutral observations about systemic motives and the associated perverse incentives.
Also it's not clear to me that any of the utilities in question have anything to do with the federal government so I'm not sure why you dragged them into this.
>I don't think your attitude here is in keeping with the guidelines (or constructive discourse for that matter).
I'll make a note of that. Any comment that I make on this site is made in the hopes that there will be discussion generated. Sometimes that happens and other times it doesn't. I don't spend any part of my life wondering why people choose to or choose not to engage. Most of us have busy lives and this (HN) for us is an opportunity to catch up on interesting things.
>You've made a number of uncharitable assumptions about the other party on the back of which you then launched into baseless personal attacks.
I am not sure that noting that a poster has no experience in a subject and has no idea how it really works in practice becomes an uncharitable assumption. It may look like a personal attack but it is based entirely on the content of the user's own posts, which support the conclusion that I reached.
>They are neutral observations about systemic motives and the associated perverse incentives.
From /u/moscoe original comment in the thread:
>Incentives and performance management are fundamental problem in civil service. The incentives to set high standards and hold individuals accountable simply do not exist.
These two sentences here demonstrate that /u/moscoe does not have any experience working in the public sector for federal, state, or local government entities yet somehow they feel qualified to speak confidently (incorrectly) that those who have worked in that capacity are ham-strung by the system and disincentivized to make waves. That is false.
I mentioned federal employees specifically because they are civil servants (referenced by /u/moscoe above) and someone close to me has worked a long career in the federal government under several administrations from both political parties.
I think you have your opinions and you are entitled to those opinions. I am also sure that you are wrong about the content of my post. It was not uncharitable nor were there any assumptions. The conclusions that I drew were supported by the statements made by /u/moscoe. If it ends up looking like a personal attack then there was a basis for that personal attack.
It's possible that you read my comment without reading the full thread so some of the context was missing.
Corporations have even less accountability to the public than public utility districts. If you don’t like what the organization is doing, vote in different commissioners and make them aware of your concerns. But if you don’t like what your water service is doing, good luck drilling a well to get away from them.
Why do you assume that PG&E has a good incentive structure, or that the public sector must necessarily have a bad one? My only objection to the preceding comment would be that struggling with perverse incentives isn't limited to civil service. Incentive structures are a core struggle of approximately all large groups of people.
I think the big problem is administrative capacity.
There are better run governments than we have in the US.
The contempt for the state is a self-fulfilling prophecy. The state is incompetent because many of us believe it is inevitable that it will be. Compensation is just a part of it; coherent administration with continuity is even more important.
> Yet here we are in 2026 and these utilities are still connecting these things to the raw Internet with default passwords.
I work with PLCs. Default passwords of not, the idea that such weakly secure devices are being made accessible from the public internet boggles my mind.
Absolutely 100% spot on. It’s not a political issue, it’s a technical issue. Disconnect them from the internet. Run your security patches. Check your logs. Water supplies are pretty important, do your job.
“Run your security patches” is easier said than done in the case of OT and it’s actually an issue that is further upstream than this. Policies, procedures, culture, and resources to execute. None of which are technical.
can't reboot, the machinists have the windows sized and positioned on screen just like they like them since 1997, so if you reboot it will cause downtime
Kind of feels like national security is the job of the federal government. Seems fair to say the federal government should do their job. They started a war for no reason and failed to anticipate not only these infrastructure breach but also the closure of the Hormuz strait.
The federal government could potentially oversee the most populous areas of the state and those near a military installation pretty easily. They don’t have to look at every one.
The federal government makes sure everyone who sends a venmo for $60 pays taxes on it so yeah I think securing our national infrastructure is not an unreasonable expectation.
There's no "national infrastructure" for water. Aside from what the EPA does, it isn't within the remit of the federal government to manage municipality water systems.
My wording was ambiguous. “national infrastructure” can refer to either the infrastructure in our nation or infrastructure managed by the federal government. I meant the former.
Usually these commenters are not from the US but get energized by US topics and misunderstand scale of things. E.g., they come from countries with a single nationalized entity for many things.
There are 150 million taxpayers and the federal government regulates all of them. I don’t see why they can’t audit 0.1% of that. If there are 150,000 utilities then absent some regulation, some of them will fuck up. If we want fewer fuckups, you need regulation.
It’s the same regulatory/incentive toolbox as any industry, including possibly accepting lower security standards for tiny treatment plants just like we accept less security for podunk airports.
> make the feds do it
National Security has always been a federal government responsibility. You make it sound like I’m expecting the federal government to take on some new responsibility. If the federal government starts a war with another country they’re absolutely responsible for minimizing by collateral damage at a fucking minimum.
National Security has always been a federal government responsibility yes. But what does that fundamentally mean for boots on the ground?
NSA doesn’t do IT for the DoD/W, DHS doesn’t do IT for the government, CISA only gives guidance where they can. And IT does not equal OT. The issue comes down to actual skilled people hours to do the work and resource constraints to do so.
I agree that in theory this would not be a stretch if the stars aligned, but these are for the most part, not federal government funded entities nor government controlled even at a state level. They are usually clooged together by 100 years of paper maché. And that’s just water. What about Energy? Data Centers? Pharma? Regulation is way too far behind to just instantly drop a silver bullet.
And 100% agree that we are witnessing repercussions of leadership that did not have much forethought but that ain’t new and goes back quite a ways especially in CI.
> The issue comes down to actual skilled people hours to do the work and resource constraints to do so.
It comes down to incentives. If you want broad security you have to do more than hope that every water utility will both hire good people and also allow them to do their jobs properly.
> But what does that fundamentally mean for boots on the ground?
How does any regulation look on the ground? How does the federal government regulate banks and airports?
> these are for the most part, not federal government funded entities nor government controlled even at a state level
Neither are banks or airports
> What about Energy? Data Centers? Pharma?
Energy and pharma are already regulated. Maybe data centers will be eventually if they are deemed sufficiently critical.
> Regulation is way too far behind to just instantly drop a silver bullet.
I don’t know what this even means in the context of securing our water system. Do you mean to say that regulation can’t ensure that these software systems don’t use default passwords and so on?
> And 100% agree that we are witnessing repercussions of leadership that did not have much forethought but that ain’t new and goes back quite a ways especially in CI
What is new is that we started a war with a country with a respectable technology competency without doing anything to shore up our defenses.
Not IT malpractice and this where the industry diverges. IT folks usually don’t work on or understand these systems.
Which is one of MANY problems OT faces. IT best practices don’t suffice in OT and even when they do, most of these orgs are too resource hamstrung to do anything about all of the fires they have to put out.
Not to mention all of the OT vendors who flooded the market with tools instead of people being taught the boring process driven work.
CISA was formed in 2018, so not quite 15 years but closer to half that. The security industry as a whole has been yelling for longer than 15 years about the vulnerability of utilities. They've been marked as soft targets before the Bush administration restructured the government.
Yes, utilities shouldn’t be negligent, but national security is 100% the federal government’s responsibility. If the vulnerabilities were so trivial, then it’s even more damning that the federal government was caught with its pants down, particularly since they were the only ones who knew they would be starting a war.
There are 1000x as many tax payers and the government still makes sure every single mom who gets a venmo payment for $60 pays taxes. National security is a federal responsibility, they should absolutely do their jobs.
Until the people in charge face jailtime for hurting innocent people, why would they care? The government shouldn’t be warning, it should be ordering and imprisoning. And funding and educating where there are genuine gaps.
"I blame it on Minnesota because they are grossly incompetent."
"I think Minnesota is behind it."
The first quote makes it the state's responsibility to secure local water systems, which I'm not sure that it is. The second makes it at least sound like the state of Minnesota is the entity running the attack on local water systems within their state, which is off in paranoid conspiracy territory.
> Censys ARC identified 4,148 Internet-exposed hosts that respond to EtherNet/IP and self-identify as Rockwell Automation/Allen-Bradley. The United States remains dominant at 71.0% (2,945 hosts), with Canada a clear second at 11.5% (476 hosts).
Describe the network security of the industrial automation industry and their customers in a single statement. Lol.
And how many more are on the same internal networks as dozens or hundreds of ordinary Windows desktops on which municipal workers are checking their email? Hardly better.
It’s far worse, just last week I was assessing some architecture and there’s still dial up and 3G connected devices in some of the most critical infrastructure around..
I don't see the issue with either of those things? At least as long as they're properly secured. (Which they probably aren't but that's neither here nor there.)
Well I think it speaks to the age of the equipment they are speaking of in the industrial sector.
How do you lockdown something that may have not been taken offline for decades because it will cost downtime or harm. Or something that can’t be locked down without tossing new tech around it that may not be compatible with the protocols etc.
Dial up into an air gapped network defeats the purpose of being air gapped. I would think the bare minimum standard is that there is no way to change anything in the control systems without being physically present at the facility, past it's physical security boundary.
Is there reason to assume someone even tried to air gap it?
If.
It is an issue, dial up lacks tunnel encryption and if you managed to make it, it will be useless in real scenarios, and 3g is being phased out and obsolete
Secure the contents of the tunnel and it doesn't matter. Provide a secure backbone and it doesn't matter. Realize that dialup is so slow that you can transparently tunnel it across ~any modern network (for which you can encrypt the tunnel) and it doesn't matter.
Tunnel your dialup within your obsolete 3g network, and then tunnel that obsolete 3g network within something modern. The obsolete technologies are not the issue here.
Also the thing about dialup is that it's point to point so the attack surface isn't even remotely comparable to exposing a port on the open internet. I should generally be able to trust the link that my phone company provides. Faxes are still used in many secure settings in preference to email.
For 3G, it’s phased out, so the bands will be gone, you have to upgrade it.
The dial up part is far more involved, especially when they have auto answering connected directly to PLC or HMI, mostly with shared passwords. Also, you can’t trust the network operator either, insider threats and rogue employees are a threat. Additionally, dial ups are less monitored compared to modern network, and usually you end up with duct tape solutions like jump server to have strong authentication and continuous logging in firewall and such, plus proper encrypted tunnels so even physical wiretapping isn’t possible, and the assumption of air gap isn’t there because it’s reachable through public telephone, and the worst part, these dial ups are usually connected to windows XP Scada developers machines.
To add, obsolete is bad too, when your device cease to have vulnerability patches, you are screwed regardless of whatever configs you put.
Used to drive me mad that the clipper terminals in the SF Caltrain station used dial up, in 2017!
https://www.linkedin.com/pulse/end-complacency-i-can-hope-an...
I much prefer the non-vendor perspective on this. Andy Krapf, co-chair of the Water ISAC, has a great breakdown about the status quo systemic problems that water faces today.
There can be surprisingly few critical components and processes in a water and sewerage network due to gravity-fed designs (far more cost efficient), large buffers (water reservoirs, sewage sumps, etc) allowing intermittent operation of otherwise critical components, retained ability for humans to manually operate equipment, and also the fields of availability and safety engineering which typically prefer elimination of software failure modes by designing equipment to not rely upon software.
The aim of a water network is to:
1. Take water from a water source (elevated dam -- strongly preferred, river, ocean) and as much as possible, gravity feed it to a treatment plant.
2. Treat the water using processes that are simplified/fail-safe as much as possible and could be operated manually by humans if necessary. This is where availability and safety engineers would design equipment to not be dependent on software and instead use mechanical or analogue electronics control.
3. Pre-position treated water as much as possible at ~50-90m hydraulic head (~500-900kPA) above the water faucets where people want to use the treated water and provide a buffer for X days of usage. Any pumps between the treatment plant and elevated storage therefore only need to operate intermittently to refill the buffer.
Sewerage networks have similar aims:
1. Let sewage flow as much as possible downhill to the treatment plant via gravity. Where a rising main (elevation gain) is required, place a large enough sump for X hours/days of usage and pump up to higher elevation from the sump.
2. Treat the sewage using processes that are simplified/fail-safe as much as possible and could be operated manually by humans if necessary. For example, a compressor used for aeration can be manually switched on/off with a mechanical switch and plugged into a diesel generator, and not require someone logging in with multi-factor authentication to a laptop to issue a command to a PLC to turn on the compressor.
3. Design overflows into the system for emergency release of partially or untreated sewage, and practice this process as part of disaster recovery exercises. This is generally an aim arising due to risk assessment process that says building a $1bn sump with 8 independent pumps is cost prohibitive versus the 1-in-200 year chance of untreated sewage messing up a downstream river for a few weeks.
Ultimately a lot of the cybersecurity risk comes down to government appetite to accept 1-in-1000 (or whatever) year failure modes. Is it worth investing now in triple modular redundant automated control systems (mostly used in safety-critical sectors such as aviation and space), or installing a just-in-case diesel generator at every one of 500 pumping stations across a region, or building $10bn of sewage sumps to hold sewage for up to a month, or building 2 treatment plants instead of 1 and using different technology for each, or hiring and training more humans to regularly exercise manual control and operation of a network, etc? Or just accept that once every 1000 years, some water rationing may be required, or a downstream river will be polluted for a few weeks?
Earlier: https://news.ycombinator.com/item?id=49135507
Sadly this instantly became a political football, with the states pointing fingers at Iran, but Trump was not wrong in this case. This is gross incompetence at all levels — IT malpractice if you will.
CISA and its predecessors have been warning utility operators about critical infrastructure vulnerabilities for what, 15 years at this point? That goes back to the first Obama administration.
Yet here we are in 2026 and these utilities are still connecting these things to the raw Internet with default passwords. You cross a threshold where you're being deliberately careless.
When you are putting more effort into securing your Plex server on your home network then public utilities are taking on machinery that dumps chemicals into the local water, something is not right and finger-pointing isn't going to fix it.
I think it's less carelessness and more the inability to attract (pay) people who have the technical knowhow to properly secure infrastructure. Even a lot of developers are poor network engineers and treat IT like magic at their own companies.
I've met info-sec / vulnerability researcher types that were egregiously reckless, like plugging Raspberry Pi's into the production network kind of thing.
Public sector has always paid low. But the problem is widespread, almost universal, and they've had a 15 year head start of the federal government telling them to get their shit together.
At some point it just became standard industry practice is my guess.
Upgrades to waste water are project based. Lowest bidder will not provide security for free. Security may be mentioned in spec but in hand waved language that can be hand waved away. That company doing the improvement project will have next to no documentation from the previous engineering effort. Just do bare minimum and move on to next job, because no one is getting paid enough to do put in more effort.
In all things when it fails in a drastic way the system will be corrected. People will die, it will suck, changes will be made.
Government is supposed to respond to these things and create incentives to correct. Telling a small municipality to do something without a carrot ir stick does nothing.
In this instance someone else will be providing the stick.
You’re on the right track, I think. But, I wouldn’t say it’s about the pay to attract competent workers. I think it has more to do with the incentive structures once you’re in. Incentives and performance management are fundamental problem in civil service. The incentives to set high standards and hold individuals accountable simply do not exist.
The one and only exception is the military, because lives are literally on the line in a way that is not at all abstract.
Usually when people say this they are dog whistling privatization. Which is the exact opposite thing people need in infrastructure, ask anyone who has to deal with PG&E.
Paying skilled people highly does actually incentivize people to do better work, especially if they are actually embedded into the community they are essentially working for. If being a civil servant was as "glorious" as being a techie is SF there would be a very different attitude around the work.
In an environment without accountability, higher pay is just more incentive to lay low and not take any personal risk. Why do anything other than the bare minimum when there is no upside? In a bureaucracy where responsibility is diffused and the culture is purely political and not merit/performance-based, few are willing to step out of line to do the right thing.
The people who climb to the top aren’t the ones who took a risk and got the reward. Quite the opposite, they’re the ones who learned to play the game and didn’t upset the power structure by rocking the boat. No one is going to tell the emperor he has no clothes when the path to power is political and has no grounding in reality.
Highly conscientious, intrinsically motivated people will do the right thing in any environment. And those people don’t last long in political/bureaucratic environments where the incentives are misaligned.
Your comments show, beyond a shadow of a doubt, that you have never worked for the federal government and likely have never worked in state or local government.
>In a bureaucracy where responsibility is diffused and the culture is purely political and not merit/performance-based, few are willing to step out of line to do the right thing.
Federal employment is merit based. Advances are earned, not doled out to the teacher's pet or the ass-kisser with the most perfect pucker. You have no idea what you're talking about.
I have no idea where you or anyone else got the idea that a government employee has no responsibility or incentives and works strictly for political ends and even worse, where none are willing to speak up. You clearly have no experience in that space and should focus your comments on things that you better understand through direct experience.
>And those people don’t last long in political/bureaucratic environments where the incentives are misaligned.
This is bullshit and you are grossly misinformed. Plenty of people fight the polarization of the system on political lines from within the system. They use the laws that are in place in order to prevent systemic abuses. They may not win every internal fight but they do go to battle for the right reasons.
You just have no idea.
I don't think your attitude here is in keeping with the guidelines (or constructive discourse for that matter). You've made a number of uncharitable assumptions about the other party on the back of which you then launched into baseless personal attacks.
Notably everything in the comment you replied to applies equally to the public and private sector. They are neutral observations about systemic motives and the associated perverse incentives.
Also it's not clear to me that any of the utilities in question have anything to do with the federal government so I'm not sure why you dragged them into this.
>I don't think your attitude here is in keeping with the guidelines (or constructive discourse for that matter).
I'll make a note of that. Any comment that I make on this site is made in the hopes that there will be discussion generated. Sometimes that happens and other times it doesn't. I don't spend any part of my life wondering why people choose to or choose not to engage. Most of us have busy lives and this (HN) for us is an opportunity to catch up on interesting things.
>You've made a number of uncharitable assumptions about the other party on the back of which you then launched into baseless personal attacks.
I am not sure that noting that a poster has no experience in a subject and has no idea how it really works in practice becomes an uncharitable assumption. It may look like a personal attack but it is based entirely on the content of the user's own posts, which support the conclusion that I reached.
>They are neutral observations about systemic motives and the associated perverse incentives.
From /u/moscoe original comment in the thread:
>Incentives and performance management are fundamental problem in civil service. The incentives to set high standards and hold individuals accountable simply do not exist.
These two sentences here demonstrate that /u/moscoe does not have any experience working in the public sector for federal, state, or local government entities yet somehow they feel qualified to speak confidently (incorrectly) that those who have worked in that capacity are ham-strung by the system and disincentivized to make waves. That is false.
I mentioned federal employees specifically because they are civil servants (referenced by /u/moscoe above) and someone close to me has worked a long career in the federal government under several administrations from both political parties.
I think you have your opinions and you are entitled to those opinions. I am also sure that you are wrong about the content of my post. It was not uncharitable nor were there any assumptions. The conclusions that I drew were supported by the statements made by /u/moscoe. If it ends up looking like a personal attack then there was a basis for that personal attack.
It's possible that you read my comment without reading the full thread so some of the context was missing.
Corporations have even less accountability to the public than public utility districts. If you don’t like what the organization is doing, vote in different commissioners and make them aware of your concerns. But if you don’t like what your water service is doing, good luck drilling a well to get away from them.
Why do you assume that PG&E has a good incentive structure, or that the public sector must necessarily have a bad one? My only objection to the preceding comment would be that struggling with perverse incentives isn't limited to civil service. Incentive structures are a core struggle of approximately all large groups of people.
I think the big problem is administrative capacity.
There are better run governments than we have in the US.
The contempt for the state is a self-fulfilling prophecy. The state is incompetent because many of us believe it is inevitable that it will be. Compensation is just a part of it; coherent administration with continuity is even more important.
We just started replacing our PLCs. They absolutely were setup with default passwords, but weren't put on the public internet.
What policies do y’all have in place for this? Is there a program in place or the beginnings of one at least?
> Yet here we are in 2026 and these utilities are still connecting these things to the raw Internet with default passwords.
I work with PLCs. Default passwords of not, the idea that such weakly secure devices are being made accessible from the public internet boggles my mind.
Yeah I meant the default passwords are simply the cherry on top of already egregiously poor security.
What industry? Very relevant.
Absolutely 100% spot on. It’s not a political issue, it’s a technical issue. Disconnect them from the internet. Run your security patches. Check your logs. Water supplies are pretty important, do your job.
“Run your security patches” is easier said than done in the case of OT and it’s actually an issue that is further upstream than this. Policies, procedures, culture, and resources to execute. None of which are technical.
Not really. Just patch and reboot. Pretty simple.
can't reboot, the machinists have the windows sized and positioned on screen just like they like them since 1997, so if you reboot it will cause downtime
Kind of feels like national security is the job of the federal government. Seems fair to say the federal government should do their job. They started a war for no reason and failed to anticipate not only these infrastructure breach but also the closure of the Hormuz strait.
So the federal government should be responsible for every rinky-dink water well in Bumblefuck, Minnesota?
> failed to anticipate not only these infrastructure breach
They've been warning them for close to two decades.
Minnesota chose the path of no locks on their front doors and are now crying that someone walked in without knocking first.
The federal government could potentially oversee the most populous areas of the state and those near a military installation pretty easily. They don’t have to look at every one.
The federal government makes sure everyone who sends a venmo for $60 pays taxes on it so yeah I think securing our national infrastructure is not an unreasonable expectation.
There's no "national infrastructure" for water. Aside from what the EPA does, it isn't within the remit of the federal government to manage municipality water systems.
My wording was ambiguous. “national infrastructure” can refer to either the infrastructure in our nation or infrastructure managed by the federal government. I meant the former.
I will repeat a comment from below. There are over 150k water utilities alone in the US.
Passing the buck to the Federal Government is not understanding the problem.
Usually these commenters are not from the US but get energized by US topics and misunderstand scale of things. E.g., they come from countries with a single nationalized entity for many things.
There are 150 million taxpayers and the federal government regulates all of them. I don’t see why they can’t audit 0.1% of that. If there are 150,000 utilities then absent some regulation, some of them will fuck up. If we want fewer fuckups, you need regulation.
What’s the penalty you would impose on a rural water system that has under 10 employees that services thousands of people for water and/or wastewater?
What does that audit look like and how frequent does that happen? It’s a security assessment? A quality assessment? A risk assessment?
I’m open to the idea, but I will repeat, I don’t think the problem is well enough understood for a “make the feds do it” type of comment.
It’s the same regulatory/incentive toolbox as any industry, including possibly accepting lower security standards for tiny treatment plants just like we accept less security for podunk airports.
> make the feds do it
National Security has always been a federal government responsibility. You make it sound like I’m expecting the federal government to take on some new responsibility. If the federal government starts a war with another country they’re absolutely responsible for minimizing by collateral damage at a fucking minimum.
National Security has always been a federal government responsibility yes. But what does that fundamentally mean for boots on the ground?
NSA doesn’t do IT for the DoD/W, DHS doesn’t do IT for the government, CISA only gives guidance where they can. And IT does not equal OT. The issue comes down to actual skilled people hours to do the work and resource constraints to do so.
I agree that in theory this would not be a stretch if the stars aligned, but these are for the most part, not federal government funded entities nor government controlled even at a state level. They are usually clooged together by 100 years of paper maché. And that’s just water. What about Energy? Data Centers? Pharma? Regulation is way too far behind to just instantly drop a silver bullet.
And 100% agree that we are witnessing repercussions of leadership that did not have much forethought but that ain’t new and goes back quite a ways especially in CI.
> The issue comes down to actual skilled people hours to do the work and resource constraints to do so.
It comes down to incentives. If you want broad security you have to do more than hope that every water utility will both hire good people and also allow them to do their jobs properly.
> But what does that fundamentally mean for boots on the ground?
How does any regulation look on the ground? How does the federal government regulate banks and airports?
> these are for the most part, not federal government funded entities nor government controlled even at a state level
Neither are banks or airports
> What about Energy? Data Centers? Pharma?
Energy and pharma are already regulated. Maybe data centers will be eventually if they are deemed sufficiently critical.
> Regulation is way too far behind to just instantly drop a silver bullet.
I don’t know what this even means in the context of securing our water system. Do you mean to say that regulation can’t ensure that these software systems don’t use default passwords and so on?
> And 100% agree that we are witnessing repercussions of leadership that did not have much forethought but that ain’t new and goes back quite a ways especially in CI
What is new is that we started a war with a country with a respectable technology competency without doing anything to shore up our defenses.
> IT does not equal OT
I'll say, you should see the hours I have to work sometimes. Honestly I've rarely seen IT jobs pay OT.
Not IT malpractice and this where the industry diverges. IT folks usually don’t work on or understand these systems.
Which is one of MANY problems OT faces. IT best practices don’t suffice in OT and even when they do, most of these orgs are too resource hamstrung to do anything about all of the fires they have to put out.
Not to mention all of the OT vendors who flooded the market with tools instead of people being taught the boring process driven work.
CISA was formed in 2018, so not quite 15 years but closer to half that. The security industry as a whole has been yelling for longer than 15 years about the vulnerability of utilities. They've been marked as soft targets before the Bush administration restructured the government.
Yes, utilities shouldn’t be negligent, but national security is 100% the federal government’s responsibility. If the vulnerabilities were so trivial, then it’s even more damning that the federal government was caught with its pants down, particularly since they were the only ones who knew they would be starting a war.
> finger-pointing isn't going to fix it.
Your entire comment was finger pointing…
There are over 150k water utilities alone in the US.
Passing the buck to the Federal Government is not understanding the problem.
There are 1000x as many tax payers and the government still makes sure every single mom who gets a venmo payment for $60 pays taxes. National security is a federal responsibility, they should absolutely do their jobs.
> There are 1000x as many tax payers and the government still makes sure every single mom who gets a venmo payment for $60 pays taxes.
That's a join in an existing database.
Where's the database and what's the code for this case?
Until the people in charge face jailtime for hurting innocent people, why would they care? The government shouldn’t be warning, it should be ordering and imprisoning. And funding and educating where there are genuine gaps.
a broken clock, yada yada.
"I blame it on Minnesota because they are grossly incompetent."
"I think Minnesota is behind it."
The first quote makes it the state's responsibility to secure local water systems, which I'm not sure that it is. The second makes it at least sound like the state of Minnesota is the entity running the attack on local water systems within their state, which is off in paranoid conspiracy territory.
Trump was absolutely wrong.
Some people allegedly say that the US should treat heads of companies like they do in China, when found guilty of certain crimes. Allegedly.