As a kid I used to substitute in an injection molding plant. We worked for four hours, pause, four hours. The plant ran three shifts, 7 days a week.
I worked in a big hall with two rows of molding machines. Each one had a pole with a yellow beacon light at the top.
The machines had fixed cycle times and the operator had to repeat the same steps every cycle. For example on one machine I had to put four small metal cylinders into the mould, close the door, wait, open the door and remove the finished parts. Cycle time was only a couple of seconds usually, with the shortest one I remember being 8 seconds. If you were too slow the machine stopped and the yellow light went on.
You also could stop the machine on purpose. There was a field with 3 by 6 buttons or something with different stopping reasons, toilet break being one of them.
So far, this probably doesn't sound too bad, but to complete picture you have to know two things:
1. Every restart meant throwing away the first one or two minutes of production
2. Foreman had to keep a quota.
That meant, yellow light, foreman came and shouted at you. The buttons were never used. You spent four hours straight doing the same routine every couple of seconds without skipping a beat.
Whenver I think my job is bad I remember that time and I'm glad for what I have.
I have worked on production lines and I’ve worked in the manufacturing and quality engineering side of things.
I know a lot of companies do have that old school production at all cost approach, but the better places I’ve worked at have put quality metrics ahead of productivity metrics.
Usually the metrics are safety, quality, delivery, and cost in that priority.
I know Goodhart’s law, “when a measure becomes a target, it ceases to become a good measure,” gets quoted a bunch on HN, but then nothing else is offered up. In my opinion, having these additional metrics with their own targets serves to balance out and constrain a single metric screwing things up. For example, if productivity is going up by producing lower quality parts, this will be shown in either the quality metrics (e.g., scrap rate, yield) or the cost metrics (e.g., material costs have increased, rework costs have increased).
The other component of it all is that the operators can only produce to the level of the systems they have. They can make some small, point improvements to help, but generally there need to be larger process or system level changes supported to make significant improvements.
> Whenver I think my job is bad I remember that time and I'm glad for what I have.
I generally suspect folks would be a lot happier if they'd had a few more crap jobs as a baseline!
I think my lowest point was vacuum packing smoked salmon. It wasn't dissimilar in some ways (repeat simple task endlessly), but the real crux was the absolute inability to evade the smell. It came home with you and lingered hard. The greasy aroma of kitchen fryers had nothing on this.
* Unloading fish on a wharf during a violent snowstorm, at night.
* Doing the stop/go signs on roadworks for 12 hours in the rain.
* Carrying heavy boxes of government records up 5 flights of stairs from a basement for eight hours, and the next day, carrying them up 2 flights of stairs to their new home - blew out the medial meniscus in my right knee doing that
* Carrying what Americans call drywall, but like the kind with noise deadening material in it that makes it heavier, non-stop for three weeks during the construction of an art gallery. Ever had that thing where your body just says "no" because you've been pushing your muscles too hard and your fingers go nerveless and involuntarily release? I got a written warning for that because I dropped the expensive drywall. Only kind of drywall that was worse was the drywall used to line X-ray rooms in medical centres.
* Working the recycling trucks - the bins we emptied into the sorting tray had drainage holes in the bottom, and you had to lift them above your head to tip them into the sorting tray, so when you did, a lovely mixture of coagulating milk / wine / beer / whatever other fluids came from unclean recycling would run down your arm and until I learned to tape up the cuffs of my overalls, inside your overalls. And when we went to offload, rubbish dump dust would stick to your sweat / recycling juice, so every day ended up stinking and filthy.
* Kitchenhand in an airport kitchen so poorly managed I was the longest serving dishpig they'd ever had when I stayed there six months. I outlasted about 8 cooks, I suppose they had options.
* Case worker in social security - understaffed, pressured to achieve unrealistic numbers, and often directed by policy to not help people who needed help.
* Oh, and roguing for wild oats in wheat fields in near 100°F heat.
Those jobs really help me appreciate how privileged and lucky I am to be a developer.
I've done some odd jobs too; worked in an assembly plant for toilet freshener plastic things, most of the time was spent making sure the components were supplied to the machines and their workers and collecting, weighing, spot checking, stacking and wrapping the finished product boxes on pallets, but for a few weeks I was at one of the machines, putting a little foam insert rough-side-up on a plastic frame. Two hours on end, break, two hours, lunch, etc. It was earbuds in, brain off, blink, day finished, part of me misses it.
Other jobs were harvesting lettuce (on your knees with a knife behind a conveyor, my knees were fooked after a few days), building scaffolding in 30+ degree weather (lasted two weeks I think?), working at a DIY store (restocking, mixing paint, cleaning, it was a decent job, I was there for over a year working part-time), delivering newspapers, etc.
> I generally suspect folks would be a lot happier if they'd had a few more crap jobs as a baseline!
I'll go one further. Not only should everybody have to do a crap job or two, but they should have to work with the public. Imagine how much more chill people would be with service workers having had the experience of being one themselves.
Cheese warehouse: the feta buckets always leaked onto to wooden pallets, making an unholy stink.
My step dad's worst was shoveling iron shot used to grind cement out of the grinder, lifting them up over his head, next to the kiln on an already boiling hot Californian summer day.
> I generally suspect folks would be a lot happier if they'd had a few more crap jobs as a baseline!
Fully agree. I cleaned a potato processing plant at night. They had to stop production for a few hours so there was huge time pressure. Crawling under machines to drain waste water tanks (onto yourself as the plug was of course on the bottom), slippery potato-mash everywhere, reaching into and almost running over all kinds of machines and conveyor belts.
All while wearing a rubber suit in the heat of warm water pressure washers and the smell of industrial cleaning agents.
So yes, I'll happily take that 15 minute useless "strategy meeting" with you.
Salmon canning in Alaska was many people's ticket/gateway to that romantic state. If you were a climbing or ski bum or fishing bum type of guy, it was a great but grueling gig that would grant you access to the best outdoors spots in the world.
Salmon canning was also a gateway to getting on board the crab boats (Deadliest Catch boats), which was another dream for many a young man, and where you could make real money (even more grueling) - and have plenty of leisure to boot.
> I generally suspect folks would be a lot happier if they'd had a few more crap jobs as a baseline!
One of my first coding jobs, was maintenance programming a 1970s-vintage FORTRAN IV codebase (primitive email system).
Over 100KLoC of slop-quality pasta. Not one single comment. No subroutines, 2-character variable names, etc. The original author was around, but it was worth your job to bother him. I was better off, using a Ouija board, to contact his father.
That informed my approach, for the rest of my life.
Semiconductor manufacturing takes this to the next level with techniques like statistical process control. Units of capacity or work can take themselves offline automatically if something falls too far out of range. Waiting for a human operator is not really feasible when the facility is mostly automated and the feature sizes are atomic scale. There are also concerns with wafers that must be processed at their next step promptly or they will become guaranteed scrap. Human judgement is too slow to sort all of this out on the inner loop.
The ability for one person to stop the entire factory on purpose is not desirable in these facilities. At least not without a quick meeting first. There isn't an actual "stop line" button anywhere. Investors and customers like Apple and Nvidia would never permit this. The best someone could do (without using a fire axe in the datacenter) is to manually whack the EMO button on all the tools in a certain area (e.g. photo) which would effectively stop the line. By the time the 2nd photo tool goes offline, half of manufacturing leads would have their pagers beeping and security would be well on their way. You could try doing this through the information systems, but they likely didn't grant you permissions to flip status flags on 400 million dollar EUV tools at your leisure.
Self-stopping is called autonomation (or jidoka) in lean/Toyota terms. It's born out of the realisation that automation is capable of producing mistakes at a very high rate, so anything automated needs a way to detect when it's operating outside of the narrow envelope for which it is designed, and shut itself down.
In theory a good escalation system. In practice there must be strong guarantees and trust that there are no repercussions for triggering. Otherwise management will tell you over and over to "pull the andon cord / escalate earle & often" but really no one does.
Not only must there be trust, but there must also be a resolve to make deeper fixes to problems surfaced through the andon system. It's a typical mistake to forget that part. If the response to a problem is patching over the immediate symptom, then the cord will keep being pulled for the same reasons so often no work will end up getting done.
Then the andon system is abandoned as "didn't work for our organisation".
Workers should pull the andon cord (and thus stop the entire production line!) when they need to go to the bathroom, for example. The solution is not to sternly tell the worker to hold it for longer, nor to have a replacement worker come over, but to review scheduling and include more appropriate bathroom breaks between shifts at the line. (Or, if the problem affects one worker disproportionately, figure out some alternative way for that worker to contribute.)
Yeah, at $WORK, the incident doc has a section for corrective actions. And they mostly get done, but having them struggle for priority with regular work is a problem.
Also, it occurs to me that we leaders should do a better job of saying, "look at that excellent job of attacking that corrective item that team just did!" so that it's clear to people that this is not clean-up work, but in fact the most essential kind of tech debt that's actively being paid down, which identified itself as such by causing an outage.
It's not just resolve, either. The org must recognize the problem in the first place, which isn't a given. Especially in smaller/govt orgs. Often, recognition of the problem is wrongly tied to how difficult the foremost suggestion is to implement.
And there are orgs for whom any suggestion can itself be so encumbered by uncontrolled red tape and social costs that relatively minor changes become a project.
It only works well in a "no blame" culture - just like software development should be. Sure, one person can really mess up, but there should be systems and processes in place that prevent that from happening or becoming a major issue. This includes limiting access to production, code reviews, CI, etc.
It feels like these best practices are often forgotten or skipped by people who just want to feel productive (be it through writing their own code or using AI). Which is fine to a point - for personal projects.
Maybe it works for small stuff like running out of bolts in a production line but not something high level like the owner is an idiot and made a massive mistake. I sometimes think about the high profile Amazon Fire Phone and why nobody involved in building the Amazon fire phone say at any point that this thing was destined to fail. I'm sure Amazon dot com had highly intelligent workers and managers who saw what was coming but never spoke up.
I have never ever heard from any Amazon employee I've met in person tell me of any instance where they told their manager or supervisor something and had the superior "disagree and commit". It always goes in one direction, down unlike what they say in their HR material.
I don't think this is a solved problem at all, short of making it very inexpensive to pull that proverbial cord as a worker AND making it very expensive to ignore such cord pulling as management. I don't know if it is possible to have that with our management system today. These two properties — cheap to pull and expensive to ignore — are intertwined. It means management giving up a lot, perhaps almost all, of its power to the workers. If you follow through with this, you also need to share more information because if the workers are actually empowered to decide, they should have the information necessary to make such decisions.
It requires someone with power to consistently and deliberately eschew this power which isn't sustainable because at some point in the management chain you will come across someone who will not.
Even at Toyota, I don't think you can pull the Andon cord on hydrogen fuel cell to switch to electric vehicles because at this point you are not just up against management, you are up against national energy policy.
One interesting thing you learn over the years is there's a million ways to improve process, and a million ways to avoid it. Virtually every improvement is unlikely; either it's difficult to understand, or difficult to get approval/consensus for, or difficult to implement. If it was easy you'd have done it already. Process won't improve until a motivation arrives that pushes past the difficulties.
I first heard about the Andon cord in this excellent This American Life episode about a GM factory that tried to learn and copy the Toyota method of production. It's still worth a listen and explains the concept well.
That's actually the wikipedia article that inspired me to name a repo-convention checker I made called Pokayoke[0]. Do find it fascinating (as with Andon) how many cool Japanese loan-words there are for process refinement.
Trader Joe's cashier bells have entered the chat.
Ironically, the "request assistance" button and accompanying blinking light on top of your stand at self-checkout are the "self-managing" version of that where you as the customer are partly an employee. oh well
I was at a Safeway last month, and I asked the overseer guy what the lamps meant on top of the self-check machines, and he said he honestly didn’t know.
where I live, they light up when the self check-out needs a member of staff to do an age verification or spot check. Or I suppose when there's an error, but that's not as common.
As a kid I used to substitute in an injection molding plant. We worked for four hours, pause, four hours. The plant ran three shifts, 7 days a week.
I worked in a big hall with two rows of molding machines. Each one had a pole with a yellow beacon light at the top.
The machines had fixed cycle times and the operator had to repeat the same steps every cycle. For example on one machine I had to put four small metal cylinders into the mould, close the door, wait, open the door and remove the finished parts. Cycle time was only a couple of seconds usually, with the shortest one I remember being 8 seconds. If you were too slow the machine stopped and the yellow light went on.
You also could stop the machine on purpose. There was a field with 3 by 6 buttons or something with different stopping reasons, toilet break being one of them.
So far, this probably doesn't sound too bad, but to complete picture you have to know two things:
1. Every restart meant throwing away the first one or two minutes of production
2. Foreman had to keep a quota.
That meant, yellow light, foreman came and shouted at you. The buttons were never used. You spent four hours straight doing the same routine every couple of seconds without skipping a beat.
Whenver I think my job is bad I remember that time and I'm glad for what I have.
> Foreman had to keep a quota.
I have worked on production lines and I’ve worked in the manufacturing and quality engineering side of things.
I know a lot of companies do have that old school production at all cost approach, but the better places I’ve worked at have put quality metrics ahead of productivity metrics.
Usually the metrics are safety, quality, delivery, and cost in that priority.
I know Goodhart’s law, “when a measure becomes a target, it ceases to become a good measure,” gets quoted a bunch on HN, but then nothing else is offered up. In my opinion, having these additional metrics with their own targets serves to balance out and constrain a single metric screwing things up. For example, if productivity is going up by producing lower quality parts, this will be shown in either the quality metrics (e.g., scrap rate, yield) or the cost metrics (e.g., material costs have increased, rework costs have increased).
The other component of it all is that the operators can only produce to the level of the systems they have. They can make some small, point improvements to help, but generally there need to be larger process or system level changes supported to make significant improvements.
> Whenver I think my job is bad I remember that time and I'm glad for what I have.
I generally suspect folks would be a lot happier if they'd had a few more crap jobs as a baseline!
I think my lowest point was vacuum packing smoked salmon. It wasn't dissimilar in some ways (repeat simple task endlessly), but the real crux was the absolute inability to evade the smell. It came home with you and lingered hard. The greasy aroma of kitchen fryers had nothing on this.
Yep.
* Unloading fish on a wharf during a violent snowstorm, at night.
* Doing the stop/go signs on roadworks for 12 hours in the rain.
* Carrying heavy boxes of government records up 5 flights of stairs from a basement for eight hours, and the next day, carrying them up 2 flights of stairs to their new home - blew out the medial meniscus in my right knee doing that
* Carrying what Americans call drywall, but like the kind with noise deadening material in it that makes it heavier, non-stop for three weeks during the construction of an art gallery. Ever had that thing where your body just says "no" because you've been pushing your muscles too hard and your fingers go nerveless and involuntarily release? I got a written warning for that because I dropped the expensive drywall. Only kind of drywall that was worse was the drywall used to line X-ray rooms in medical centres.
* Working the recycling trucks - the bins we emptied into the sorting tray had drainage holes in the bottom, and you had to lift them above your head to tip them into the sorting tray, so when you did, a lovely mixture of coagulating milk / wine / beer / whatever other fluids came from unclean recycling would run down your arm and until I learned to tape up the cuffs of my overalls, inside your overalls. And when we went to offload, rubbish dump dust would stick to your sweat / recycling juice, so every day ended up stinking and filthy.
* Kitchenhand in an airport kitchen so poorly managed I was the longest serving dishpig they'd ever had when I stayed there six months. I outlasted about 8 cooks, I suppose they had options.
* Case worker in social security - understaffed, pressured to achieve unrealistic numbers, and often directed by policy to not help people who needed help.
* Oh, and roguing for wild oats in wheat fields in near 100°F heat.
Those jobs really help me appreciate how privileged and lucky I am to be a developer.
I've done some odd jobs too; worked in an assembly plant for toilet freshener plastic things, most of the time was spent making sure the components were supplied to the machines and their workers and collecting, weighing, spot checking, stacking and wrapping the finished product boxes on pallets, but for a few weeks I was at one of the machines, putting a little foam insert rough-side-up on a plastic frame. Two hours on end, break, two hours, lunch, etc. It was earbuds in, brain off, blink, day finished, part of me misses it.
Other jobs were harvesting lettuce (on your knees with a knife behind a conveyor, my knees were fooked after a few days), building scaffolding in 30+ degree weather (lasted two weeks I think?), working at a DIY store (restocking, mixing paint, cleaning, it was a decent job, I was there for over a year working part-time), delivering newspapers, etc.
Each sounds like a process that could have been improved if andon had been implemented.
> I generally suspect folks would be a lot happier if they'd had a few more crap jobs as a baseline!
I'll go one further. Not only should everybody have to do a crap job or two, but they should have to work with the public. Imagine how much more chill people would be with service workers having had the experience of being one themselves.
Cheese warehouse: the feta buckets always leaked onto to wooden pallets, making an unholy stink.
My step dad's worst was shoveling iron shot used to grind cement out of the grinder, lifting them up over his head, next to the kiln on an already boiling hot Californian summer day.
> I generally suspect folks would be a lot happier if they'd had a few more crap jobs as a baseline!
Fully agree. I cleaned a potato processing plant at night. They had to stop production for a few hours so there was huge time pressure. Crawling under machines to drain waste water tanks (onto yourself as the plug was of course on the bottom), slippery potato-mash everywhere, reaching into and almost running over all kinds of machines and conveyor belts.
All while wearing a rubber suit in the heat of warm water pressure washers and the smell of industrial cleaning agents.
So yes, I'll happily take that 15 minute useless "strategy meeting" with you.
Salmon canning in Alaska was many people's ticket/gateway to that romantic state. If you were a climbing or ski bum or fishing bum type of guy, it was a great but grueling gig that would grant you access to the best outdoors spots in the world.
Salmon canning was also a gateway to getting on board the crab boats (Deadliest Catch boats), which was another dream for many a young man, and where you could make real money (even more grueling) - and have plenty of leisure to boot.
> I generally suspect folks would be a lot happier if they'd had a few more crap jobs as a baseline!
One of my first coding jobs, was maintenance programming a 1970s-vintage FORTRAN IV codebase (primitive email system).
Over 100KLoC of slop-quality pasta. Not one single comment. No subroutines, 2-character variable names, etc. The original author was around, but it was worth your job to bother him. I was better off, using a Ouija board, to contact his father.
That informed my approach, for the rest of my life.
Interesting manufacturing story that does sound pretty crappy.
To bring it across to Andon and Lean, Deming literally addressed this as one of his 14 points of quality management that Lean manufacturing follows:
Point 11. Eliminate numerical quotas for the workforce and numerical goals for management.
What’s important here is not that there are buttons in a factory but rather the culture that goes along with their usage.
Point 8. Drive out fear, seems relevant here also.
Sorry you experienced this.
Semiconductor manufacturing takes this to the next level with techniques like statistical process control. Units of capacity or work can take themselves offline automatically if something falls too far out of range. Waiting for a human operator is not really feasible when the facility is mostly automated and the feature sizes are atomic scale. There are also concerns with wafers that must be processed at their next step promptly or they will become guaranteed scrap. Human judgement is too slow to sort all of this out on the inner loop.
The ability for one person to stop the entire factory on purpose is not desirable in these facilities. At least not without a quick meeting first. There isn't an actual "stop line" button anywhere. Investors and customers like Apple and Nvidia would never permit this. The best someone could do (without using a fire axe in the datacenter) is to manually whack the EMO button on all the tools in a certain area (e.g. photo) which would effectively stop the line. By the time the 2nd photo tool goes offline, half of manufacturing leads would have their pagers beeping and security would be well on their way. You could try doing this through the information systems, but they likely didn't grant you permissions to flip status flags on 400 million dollar EUV tools at your leisure.
Self-stopping is called autonomation (or jidoka) in lean/Toyota terms. It's born out of the realisation that automation is capable of producing mistakes at a very high rate, so anything automated needs a way to detect when it's operating outside of the narrow envelope for which it is designed, and shut itself down.
In theory a good escalation system. In practice there must be strong guarantees and trust that there are no repercussions for triggering. Otherwise management will tell you over and over to "pull the andon cord / escalate earle & often" but really no one does.
Not only must there be trust, but there must also be a resolve to make deeper fixes to problems surfaced through the andon system. It's a typical mistake to forget that part. If the response to a problem is patching over the immediate symptom, then the cord will keep being pulled for the same reasons so often no work will end up getting done.
Then the andon system is abandoned as "didn't work for our organisation".
Workers should pull the andon cord (and thus stop the entire production line!) when they need to go to the bathroom, for example. The solution is not to sternly tell the worker to hold it for longer, nor to have a replacement worker come over, but to review scheduling and include more appropriate bathroom breaks between shifts at the line. (Or, if the problem affects one worker disproportionately, figure out some alternative way for that worker to contribute.)
Yeah, at $WORK, the incident doc has a section for corrective actions. And they mostly get done, but having them struggle for priority with regular work is a problem.
Also, it occurs to me that we leaders should do a better job of saying, "look at that excellent job of attacking that corrective item that team just did!" so that it's clear to people that this is not clean-up work, but in fact the most essential kind of tech debt that's actively being paid down, which identified itself as such by causing an outage.
It's not just resolve, either. The org must recognize the problem in the first place, which isn't a given. Especially in smaller/govt orgs. Often, recognition of the problem is wrongly tied to how difficult the foremost suggestion is to implement.
And there are orgs for whom any suggestion can itself be so encumbered by uncontrolled red tape and social costs that relatively minor changes become a project.
It only works well in a "no blame" culture - just like software development should be. Sure, one person can really mess up, but there should be systems and processes in place that prevent that from happening or becoming a major issue. This includes limiting access to production, code reviews, CI, etc.
It feels like these best practices are often forgotten or skipped by people who just want to feel productive (be it through writing their own code or using AI). Which is fine to a point - for personal projects.
Yes. There was an article somewhere about how in a Japanese factory it gets pulled thousands of times a year, but only 2 in an American one.
EDIT: Found it: https://davidoks.blog/p/why-japanese-companies-do-so-many
Search for “Ford plant”, second occurrence for that particular bit. The article made rounds on HN a couple months ago.
[flagged]
Maybe it works for small stuff like running out of bolts in a production line but not something high level like the owner is an idiot and made a massive mistake. I sometimes think about the high profile Amazon Fire Phone and why nobody involved in building the Amazon fire phone say at any point that this thing was destined to fail. I'm sure Amazon dot com had highly intelligent workers and managers who saw what was coming but never spoke up.
I have never ever heard from any Amazon employee I've met in person tell me of any instance where they told their manager or supervisor something and had the superior "disagree and commit". It always goes in one direction, down unlike what they say in their HR material.
I don't think this is a solved problem at all, short of making it very inexpensive to pull that proverbial cord as a worker AND making it very expensive to ignore such cord pulling as management. I don't know if it is possible to have that with our management system today. These two properties — cheap to pull and expensive to ignore — are intertwined. It means management giving up a lot, perhaps almost all, of its power to the workers. If you follow through with this, you also need to share more information because if the workers are actually empowered to decide, they should have the information necessary to make such decisions.
It requires someone with power to consistently and deliberately eschew this power which isn't sustainable because at some point in the management chain you will come across someone who will not.
Even at Toyota, I don't think you can pull the Andon cord on hydrogen fuel cell to switch to electric vehicles because at this point you are not just up against management, you are up against national energy policy.
One interesting thing you learn over the years is there's a million ways to improve process, and a million ways to avoid it. Virtually every improvement is unlikely; either it's difficult to understand, or difficult to get approval/consensus for, or difficult to implement. If it was easy you'd have done it already. Process won't improve until a motivation arrives that pushes past the difficulties.
I first heard about the Andon cord in this excellent This American Life episode about a GM factory that tried to learn and copy the Toyota method of production. It's still worth a listen and explains the concept well.
https://www.thisamericanlife.org/561/nummi-2015
Let's rewrite all software to use Andon instead if exception
Start with writing software that has https://en.wikipedia.org/wiki/Poka-yoke in mind
That's actually the wikipedia article that inspired me to name a repo-convention checker I made called Pokayoke[0]. Do find it fascinating (as with Andon) how many cool Japanese loan-words there are for process refinement.
[0] - https://pokayoke.codes
Was anyone else expecting an article about Andon Labs?
Trader Joe's cashier bells have entered the chat. Ironically, the "request assistance" button and accompanying blinking light on top of your stand at self-checkout are the "self-managing" version of that where you as the customer are partly an employee. oh well
I was at a Safeway last month, and I asked the overseer guy what the lamps meant on top of the self-check machines, and he said he honestly didn’t know.
where I live, they light up when the self check-out needs a member of staff to do an age verification or spot check. Or I suppose when there's an error, but that's not as common.
The original dashboard alert.
[flagged]