I started out in a "data centre" in 1998/1999 in the North West of England. Things are radically different now.
When I joined that ISP we had ~2000 subscribers, and one modem rack that could take 30 concurrent 33.6k dial-ins. The company had just been acquired by a small independent telco in Cheshire, headquartered in a small town that is most famous for a weekly farmers market and an annual steam rally.
We shared that headquarters at first with the founder's other business - a dating agency for rich people in later years - and slowly took over.
The building was a farmhouse of sorts. We'd turned into a rabbit warren of offices, and then, we decided to expand.
First we turned one of the offices into a "machine room". We didn't even bother with fire suppression that first time, we just ripped up the carpet and laid some laminate, put some 19" racks in and got the power upgraded. The air-con was insufficient in that room and the following summer I had to wheel in portable units and had a Cisco router in there report that the inlet fan had a temperature of ~80C. That was around the same time the chief of staff was asking us why we wore jeans and t-shirts and couldn't we wear suits? No, no we could not wear suits.
Bit by bit we took over more rooms through the building, we started working in a portacabin in the car park (where the neighbouring farmer's sheepdog would join us for lunch).
At some point we needed to install ~20k modems, (Nortel CVX1800s - 56kbps had arrived!), and the telephone switch to run it, so somebody bought a second hand DMS100 that went into another room we had converted.
By that point the dating agency had moved out.
We needed another ~100 servers as well to manage email, web hosting, and so on. Those came in parts (motherboards, cases, RAM sticks, hard drives - about £500k worth in 1999 money), and we had to build them and install BSD on them (first OpenBSD, then FreeBSD, long story). We also had to make our own ethernet cables, which is where I learned I can't wire ethernet cables if my life depended on it.
Within 12 months from when we started, we were handling 750,000 dial-in customers.
Around September of 1999 I left and that was when the company had heard about a custom built data centre that was up for sale in Milton Keynes, so decided to move everything there. Around that time they were pouring concrete under the floors of all those rooms we'd thrown racks in because the timber floor joists (!) were starting to give way under the weight of the kit.
That company got acquired and then went through a weird route, but basically it's still going as a major UK ISP. I'll keep the name private to protect the guilty. :)
2000 subscribers, and one modem rack to that and a business still operating 27 years later. A lot of sausage rolls, plenty of sticking plasters for the cuts from those shitty cases, and almost every day ending with a couple of pints in a pub somewhere.
One of my old colleagues and I often think back to that being one of the best years of work in our lives. But... data centre work is horrid, dirty, hot, and underpaid for what it is. But you can - sometimes - be part of something kind of cool and fun, and something very few people get to see or understand.
But there's a reason I've stuck to software ever since. :)
This made me smile, reminds me of my 90s, early 00s in web development & hosting. Where tiny offices above shops spread out as the business grew; racks were thrown in cupboards; the noise was constant and the heat was unbearable in the summer; the pigeons nesting on the shit covered Victorian window sill had the best seat in the office and nobody really knew what they were doing but we were all incredibly cocky about being wunderkids working on the coolest new stuff that nobody else understood. Good times!
I had to go to another data centre just outside London about 10 years ago with a colleague who had never been inside one - we had some colocation kit hosted there and were going to spend most of a day swapping some cards and checking things through the KVM. He looked at me really weirdly when I offered him ear defenders on the way there. He wasn't ready for the sensory overload - the noise, the heat, the movement of air, the artificial light, the slightly weird smell, it was all a lot. 3 hours in, he wanted out. Couldn't blame him, I wasn't far off myself.
Adjacent, I work in a lab so we have dozens of ovens and freezers running constantly, so I'm intimately familiar with the constant hum, even though my office is a couple of doors away from the oven room.
We had a full site shutdown last week, which involved all power being shut off, which meant no running ovens.
The lack of sound felt so strange, like I was somewhere I shouldn't be, or in some sort of liminal space. It was unsettling in the silliest way.
>We also had to make our own ethernet cables, which is where I learned I can't wire ethernet cables if my life depended on it.
Are you me? would always take me about 10 attempts to even get one where all the pins would light up on a tester. But not good enough for high-speed data.
Kids these days, don't know when they were born. They have push-through RJ45 now, piece of piss and you never have one shorter than the other.
I don't miss on-site/DC infrastructure anymore. No more having to squeeze into tiny spaces, balance a lot of expensive kit on your head, and blood donations to the server gods.
Compute is so ubiquitous now I only have like 2 nodes in my lab anyway
We had to make our own cable. The boss had his own special way for us to order the wires when we did it. Worked okay for 10Mb/s and 100Mb/s on a good day.
I used to travel doing optical networking deployments between data centers. Got to see a good 35+ states and even some foreign countries over 4 years or so but also completely burnt out. Telecom is a pretty miserable industry.
Most of the work in/around DCs is contract of some sort. Local structured cabling firms, electricians doing stints of various lengths, security, etc. Some of it good and lucrative, some of it not.
I did the same in europe and burnt out with my colleague after ~3 years.
Working and driving at night, barely enough sleep and overtime compensation during the time the body needs to recover was the worst.
I probably aged 10 years in that time.
Data center construction is consuming all of the tradespeople who usually do work in homes and causing price spikes for contracting similar to what we are seeing for hardware prices. I’m guessing those workers are doing at least OK for themselves ;)
Residential contractors are typically not the same as commercial contractors. Simply for the fact that commercial charges 2x what residential does is one of those reasons. Not saying there is zero overlap, but the pools are not totally shared.
Yea, commercial has far stricter inspection and material requirements then residential. You're playing with a whole lot more angry pixies in a pipe.
Most of the overlap I see is when there is a commercial building collapse (economically, not literally) and you end up seeing master electricians working on residential. Much more rare to see the opposite movement unless it's a permanent change.
> commercial has far stricter inspection and material requirements then residential. You're playing with a whole lot more angry pixies in a pipe.
Not only that, but the trades stick to their trades.
You don't get, for example, datacentre electricians rocking up pretenting they can install network cable to spec and running away having (if you are lucky) only done a continuity test on the cable they just installed.
Meanwhile every man and his dog, or even just the dog, will very happily turn up at your house and tell you they can install your CAT6 to spec for you when they've never seen a Fluke in their entire life.
> You don't get, for example, datacentre electricians rocking up pretenting they can install network cable to spec and running away having (if you are lucky) only done a continuity test on the cable they just installed.
I frequently have my electricians pull and terminate network cable if it’s only one or a few (daisy-chained VAV controllers or a network drop for a BAS panel), if the customer needs a formal TIA-568 test then I’ll sub that out. My electricians are already pulling and terminating building automation cable, CAT 6 isn’t gonna stop em :)
Oh awesome! I’m glad there are others in the industry around here :)
I’d like to learn more about PLCs and SCADA, but that scope of work usually falls on another contractor, we mostly provide installation for BAS systems that control HVAC and chilled/hot water plants.
I do some water treatment plant projects and there’s a local contractor that handles the PLC and SCADA stuff.
In a bunch of recent residential building I've seen the electricians "attempt" to install networking cables as it's a value add they can install for very little while the walls are still open.
Needless to say I've re-pulled a lot of those cables due to so many failures.
I think some of this comes from everyone "moving up a notch". If the "top" electricians are going to data centers, the second rate are covering their old jobs, and on down until residential electricians are doing light industrial and commercial work (can probably charge more, and open up a new avenue of business).
A residential electrician would be completely lost on a data center site. Commercial electricians travel around to the various data center projects. Most of the trades are similar.
> Data center construction is consuming all of the tradespeople who usually do work in homes
I want to know what country you live in that you can find domestic tradespeople who will do work to the same standard as datacentres, and do it at domestic prices. ;)
lol. may as well put a sign around your neck that says that you don’t know what you’re talking about. this sounds like a social media rumour cooked up to add fuel to the anti data center fire.
At my first job in a bank there was an old-timer whose job had been to fly to every trading desk the bank had in other countries to install software updates. He would do a round the world trip 1-2 times a year with multiple duplicate boxes of floppies during 3-4 weeks.
Don't know about US, but I work within European Datacenter where we intend to have excess heat to be used for residential building heating [1]. You must be in proximity of the consumer for this to be possible. Not necessarily center is needed, but even then, DCs can be pretty :)
For example our office is heated using heatpump that has >22C of water heated by equipment as a source.
> excess heat to be used for residential building heating
In the late 90s I worked at an insurance company in leafy Surrey[1]. I was told that they used to use their IBM mainframe to heat the swimming pool[2] effectively for free; when they upgraded they had to buy and run a heating system for the pool as the new mainframe didn't pump out enough heat.
[1] In England, south west suburbs of London to be precise.
[2] yes it had an indoor pool, badminton courts, outdoor tennis courts and other facilities - a great place to work.
Thousands of people Google "why is my building cold" causing the AI DC to wake up and pump out heat. AI sees the heat and reports "heating is on, you're fine" then shuts down.
It is crazy how you see so many datacenters getting built in places of high land value such as say northern virginia. Why not just build them in west virginia and build twice as many for the same spend? They'd probably even let you burn coal to power them.
I suspect it's driven by availability of existing grid infrastructure (so they don't have to wait for upgrades) and ease of permitting; the contents of the building and indeed running costs are so high that the land value is much less relevant.
Power is the biggest one here in Texas, so much so even this laissez-faire state has stopped new permits for data centers. All the inner connect space for grids is used up and the wait for hardware and installs is massive.
So much so we're seeing private grids being setup for data centers.
I'm guessing it's to do with regulation. There's probably some policy document that says a certain org needs to be able to access the physical servers within a certain amount of time, and so they need to be nearby, even though nobody ever does the accessing.
DC would be the kind of place with a lot of government contractors that would have this kind of stipulation, is my guess.
West Virginia also has a lot of coal. I wonder why the current Admin doesn't encourage DCs to be built next to coal. Coal communities don't care much for any regulations, DC development should be perfectly fine. Hyperscalers can brag about 100% coal powered AI.
You would think, but I live in a tiny pop state that is barely thought about, and we have coal, but the hyperscale datacenter project did get pushback. Almost entirely political. No one had an opinion in January. But around summer something flipped. I also work with an existing datacenter here, and there was a similar thing. People were going on about how cool it was, etc, for years, and then suddenly, wham, sentiment changed.
>Texas has 623 tracked data center facilities and projects: 349 operational, 101 under construction, 156 planned, 2 paused, and 15 inactive or historical. Reported or announced power totals 96.1 GW, including 85.4 GW tied to planned or under-construction projects.
This is for just one state.
>4,998 facilities and projects nationwide, with 374.7 GW in reported or announced power figures. Of that, 318.6 GW is tied to planned or under-construction projects. Climate-risk and water-stress labels summarize current records with those fields. The directory retains 150 inactive or historical records. States are ordered by current reported or announced power.
Nearby skilled labor, police/fire infrastructure willing to handle it, and ease of demo/tours/sales for clients to install their racks. Also much more stable power.
There is a lot of electrical supply and demand is high plus incentives from Virginia itself and proximity to DC. It’s already a data center dense market
Same is true for software "engineers". We don't take a professional exam or adhere to any professional licensing. We aren't legally engineers in some states/nations.
You'll often find a lot of engineers, and in my experience, a lot of former navy nuke personnel, overseeing the power and facilities at large data centers. There also tend to be a number of others with engineering backgrounds working in data center operations even if a lot of their work may not be what would commonly fall under "engineering". Technicians who perform some of the break/fix and routine maintenance do tend to be lower skilled but, again, it's not uncommon to find advanced degrees. I knew someone with a Phd in Chemistry working as an entry level data center technician because they were simply looking for something different outside of academia and it was a career that had rapid advancement potential.
Yes, I worked for a bit on site at a datacenter in Fort Worth and was surprised by how many ex-nuclear-submarine engineers there were. I guess it's similar, you have a lot of high tech equipment in an environment where the climate is closely controlled.
In Canada, members of the Royal Canadian Engineers have just as much right to the title of engineer as professional engineers do, and the operators of locomotives are also referred to as engineers.
You're using engineer in a highly context specific way; yes, in some contexts, engineering is a licensed field, but the general meaning of the word is:
> Engineering is the practice of systematically applying natural science and mathematics to design and improve systems, devices, or processes that solve problems under constraints.
Which is a word-for-word description of the sort of work (…some…) software engineers do.
(Perhaps less and less "systematically applying science" these days with the slop fest out there … but the field should be doing engineering.)
Most software "engineers" never got a degree in engineering, and even fewer ever bothered to do what is necessary to enter the profession where it is regulated.
TL;DR: you walk around smelling and listening for issues. Most of the staff on site are doing the same thing; the rest are private security hired by customers housing within the DC. You make between $66K and $120K/annual.
There, saved you from reading a story that isn't really informative and only was written because of the politics surrounding DCs are keeping them in the news.
TBH it was informative for me because I've got family who applied for data center jobs. But I'll admit that you captured it all in your blurb. The article has a lot of fluff.
IYAM the VOCs in most data centers I've visited would be contributing to early death. I would never work in one every day. I get a headache just being in ours for more than a couple of hours. And then I can smell it in my clothes and hair for the rest of the day.
Speaking of DCs, is there evidence that they are sucking up water? What about deteriorating air quality around the buildings? Could they be causing major environmental issues? It seems they mostly have closed loop systems a la CPU coolers and they consume a lot of electricity that's already been earmarked for them. What about air quality and air humidity?
I worked at Google datacenters for over a decade. It was a huge ordeal to request running generators because our entire ECS (environment control services) team was geared towards NOT having to run them. I know AWS and Microsoft both have a similar situation. The referenced article just seems like senseless fear mongering.
There have been suggestions that the DC operators themselves have been astroturfing articles about water use to steer discussions away from the other issues.
The overwhelming, #1 issue should be the fossil fuel use. So maybe I can understand them promoting the water issue as a limited hangout (a sacrificial topic to act as a lightning rod to hide the other topic)
That sounds backwards to me. Fossil fuel use being the main issue is also mighty hypocritical for a society tarrifing cheap electric cars and solar panels.
Most data centers are closed-loop, like your air conditioner. Does your air conditioner use water?
Presumably there are some data centers that use water cooling. I think google and amazon's big ones do. But even then, they're not "polluting", they're using them as heat sinks.
There was a lame rolling stone article that claimed amazon was "polluting" in Eastern Oregon, but what Amazon was doing was taking brined water from an aquifer and putting more concentrated brined water back into the water table. Air conditioning doesn't really add brine, so even the premise is ridiculous. I know that article has been referenced a lot by the "Data Centers Pollute The Water" folks.
I'm not convinced that most DC use closed loop cooling. The recent discussions had a lot of water and power permitting topics.
Secondly drawing brine water from the aquifer to then putting the stronger brine back is essentially water consumption. I would also say it's externalising environmental cost of the stronger brine impact.
Yes, it does. As far as water is concerned, air conditioners are open loop. Air conditioners make water from condensation forming on the inside (cool) coils. Modern window air conditioners then splash that water on the outside hot coils, to benefit from evaporative cooling. I haven't looked into the details about data center water usage, but I would guess the cited usage is from evaporative cooling.
What research have you done so far? I assume you’re being downvoted because those questions are fairly easily answered, there is a lot of reports and data on DCs resource usage
https://archive.vn/Myhqh
I started out in a "data centre" in 1998/1999 in the North West of England. Things are radically different now.
When I joined that ISP we had ~2000 subscribers, and one modem rack that could take 30 concurrent 33.6k dial-ins. The company had just been acquired by a small independent telco in Cheshire, headquartered in a small town that is most famous for a weekly farmers market and an annual steam rally.
We shared that headquarters at first with the founder's other business - a dating agency for rich people in later years - and slowly took over.
The building was a farmhouse of sorts. We'd turned into a rabbit warren of offices, and then, we decided to expand.
First we turned one of the offices into a "machine room". We didn't even bother with fire suppression that first time, we just ripped up the carpet and laid some laminate, put some 19" racks in and got the power upgraded. The air-con was insufficient in that room and the following summer I had to wheel in portable units and had a Cisco router in there report that the inlet fan had a temperature of ~80C. That was around the same time the chief of staff was asking us why we wore jeans and t-shirts and couldn't we wear suits? No, no we could not wear suits.
Bit by bit we took over more rooms through the building, we started working in a portacabin in the car park (where the neighbouring farmer's sheepdog would join us for lunch).
At some point we needed to install ~20k modems, (Nortel CVX1800s - 56kbps had arrived!), and the telephone switch to run it, so somebody bought a second hand DMS100 that went into another room we had converted.
By that point the dating agency had moved out.
We needed another ~100 servers as well to manage email, web hosting, and so on. Those came in parts (motherboards, cases, RAM sticks, hard drives - about £500k worth in 1999 money), and we had to build them and install BSD on them (first OpenBSD, then FreeBSD, long story). We also had to make our own ethernet cables, which is where I learned I can't wire ethernet cables if my life depended on it.
Within 12 months from when we started, we were handling 750,000 dial-in customers.
Around September of 1999 I left and that was when the company had heard about a custom built data centre that was up for sale in Milton Keynes, so decided to move everything there. Around that time they were pouring concrete under the floors of all those rooms we'd thrown racks in because the timber floor joists (!) were starting to give way under the weight of the kit.
That company got acquired and then went through a weird route, but basically it's still going as a major UK ISP. I'll keep the name private to protect the guilty. :)
2000 subscribers, and one modem rack to that and a business still operating 27 years later. A lot of sausage rolls, plenty of sticking plasters for the cuts from those shitty cases, and almost every day ending with a couple of pints in a pub somewhere.
One of my old colleagues and I often think back to that being one of the best years of work in our lives. But... data centre work is horrid, dirty, hot, and underpaid for what it is. But you can - sometimes - be part of something kind of cool and fun, and something very few people get to see or understand.
But there's a reason I've stuck to software ever since. :)
This made me smile, reminds me of my 90s, early 00s in web development & hosting. Where tiny offices above shops spread out as the business grew; racks were thrown in cupboards; the noise was constant and the heat was unbearable in the summer; the pigeons nesting on the shit covered Victorian window sill had the best seat in the office and nobody really knew what they were doing but we were all incredibly cocky about being wunderkids working on the coolest new stuff that nobody else understood. Good times!
>But... data centre work is horrid, dirty, hot, and underpaid for what it is
You left out "loud" haha. Just the nonstop buzz; it's like you're working at an airport tarmac or something.
I had to go to another data centre just outside London about 10 years ago with a colleague who had never been inside one - we had some colocation kit hosted there and were going to spend most of a day swapping some cards and checking things through the KVM. He looked at me really weirdly when I offered him ear defenders on the way there. He wasn't ready for the sensory overload - the noise, the heat, the movement of air, the artificial light, the slightly weird smell, it was all a lot. 3 hours in, he wanted out. Couldn't blame him, I wasn't far off myself.
Adjacent, I work in a lab so we have dozens of ovens and freezers running constantly, so I'm intimately familiar with the constant hum, even though my office is a couple of doors away from the oven room.
We had a full site shutdown last week, which involved all power being shut off, which meant no running ovens.
The lack of sound felt so strange, like I was somewhere I shouldn't be, or in some sort of liminal space. It was unsettling in the silliest way.
The empty is what I call it. It's not a buzz in your ears, it permeates your entire body. And when it stops it's like part of you is missing.
Which is really odd, because just leaving the building doesn't have the same effect.
>We also had to make our own ethernet cables, which is where I learned I can't wire ethernet cables if my life depended on it.
Are you me? would always take me about 10 attempts to even get one where all the pins would light up on a tester. But not good enough for high-speed data.
Kids these days, don't know when they were born. They have push-through RJ45 now, piece of piss and you never have one shorter than the other.
I don't miss on-site/DC infrastructure anymore. No more having to squeeze into tiny spaces, balance a lot of expensive kit on your head, and blood donations to the server gods.
Compute is so ubiquitous now I only have like 2 nodes in my lab anyway
We had to make our own cable. The boss had his own special way for us to order the wires when we did it. Worked okay for 10Mb/s and 100Mb/s on a good day.
> blood donations to the server gods
It really bothered me that for how much enterprise gear costs, the manufacturers couldn't be bothered to deburr the edges of rails/servers/etc.
Hey man, saving 10 cents a rack is 10 cents directly in my pocket -- rack CEO.
I used to travel doing optical networking deployments between data centers. Got to see a good 35+ states and even some foreign countries over 4 years or so but also completely burnt out. Telecom is a pretty miserable industry.
Most of the work in/around DCs is contract of some sort. Local structured cabling firms, electricians doing stints of various lengths, security, etc. Some of it good and lucrative, some of it not.
I did the same in europe and burnt out with my colleague after ~3 years. Working and driving at night, barely enough sleep and overtime compensation during the time the body needs to recover was the worst. I probably aged 10 years in that time.
A bit like this? https://www.youtube.com/watch?v=K8KXXJH8Zy4
Data center construction is consuming all of the tradespeople who usually do work in homes and causing price spikes for contracting similar to what we are seeing for hardware prices. I’m guessing those workers are doing at least OK for themselves ;)
Residential contractors are typically not the same as commercial contractors. Simply for the fact that commercial charges 2x what residential does is one of those reasons. Not saying there is zero overlap, but the pools are not totally shared.
Yea, commercial has far stricter inspection and material requirements then residential. You're playing with a whole lot more angry pixies in a pipe.
Most of the overlap I see is when there is a commercial building collapse (economically, not literally) and you end up seeing master electricians working on residential. Much more rare to see the opposite movement unless it's a permanent change.
> commercial has far stricter inspection and material requirements then residential. You're playing with a whole lot more angry pixies in a pipe.
Not only that, but the trades stick to their trades.
You don't get, for example, datacentre electricians rocking up pretenting they can install network cable to spec and running away having (if you are lucky) only done a continuity test on the cable they just installed.
Meanwhile every man and his dog, or even just the dog, will very happily turn up at your house and tell you they can install your CAT6 to spec for you when they've never seen a Fluke in their entire life.
> You don't get, for example, datacentre electricians rocking up pretenting they can install network cable to spec and running away having (if you are lucky) only done a continuity test on the cable they just installed.
I frequently have my electricians pull and terminate network cable if it’s only one or a few (daisy-chained VAV controllers or a network drop for a BAS panel), if the customer needs a formal TIA-568 test then I’ll sub that out. My electricians are already pulling and terminating building automation cable, CAT 6 isn’t gonna stop em :)
Hey hot stuff. Same here. Hope you have a good day.
Oh awesome! I’m glad there are others in the industry around here :)
I’d like to learn more about PLCs and SCADA, but that scope of work usually falls on another contractor, we mostly provide installation for BAS systems that control HVAC and chilled/hot water plants.
I do some water treatment plant projects and there’s a local contractor that handles the PLC and SCADA stuff.
In a bunch of recent residential building I've seen the electricians "attempt" to install networking cables as it's a value add they can install for very little while the walls are still open.
Needless to say I've re-pulled a lot of those cables due to so many failures.
No instead in my experience its end clients being cheap bastards and having the electricians also pull fire alarm cable.
We had to redo and re-pull so many circuits in the last datacenter project.
I think some of this comes from everyone "moving up a notch". If the "top" electricians are going to data centers, the second rate are covering their old jobs, and on down until residential electricians are doing light industrial and commercial work (can probably charge more, and open up a new avenue of business).
A residential electrician would be completely lost on a data center site. Commercial electricians travel around to the various data center projects. Most of the trades are similar.
> Data center construction is consuming all of the tradespeople who usually do work in homes
I want to know what country you live in that you can find domestic tradespeople who will do work to the same standard as datacentres, and do it at domestic prices. ;)
Looks like AI induced inflation will eat a lot of any such gains ?
lol. may as well put a sign around your neck that says that you don’t know what you’re talking about. this sounds like a social media rumour cooked up to add fuel to the anti data center fire.
At my first job in a bank there was an old-timer whose job had been to fly to every trading desk the bank had in other countries to install software updates. He would do a round the world trip 1-2 times a year with multiple duplicate boxes of floppies during 3-4 weeks.
It's bizarre that Digital Realty wants to put a DC in a town centre. Surely the latency disadvantage from being outside city limits is neligible.
Don't know about US, but I work within European Datacenter where we intend to have excess heat to be used for residential building heating [1]. You must be in proximity of the consumer for this to be possible. Not necessarily center is needed, but even then, DCs can be pretty :)
For example our office is heated using heatpump that has >22C of water heated by equipment as a source.
https://delska.com/about/news-resources/delska-newsroom/pres...
> excess heat to be used for residential building heating
In the late 90s I worked at an insurance company in leafy Surrey[1]. I was told that they used to use their IBM mainframe to heat the swimming pool[2] effectively for free; when they upgraded they had to buy and run a heating system for the pool as the new mainframe didn't pump out enough heat.
[1] In England, south west suburbs of London to be precise.
[2] yes it had an indoor pool, badminton courts, outdoor tennis courts and other facilities - a great place to work.
Well we can hope that AI will become power efficient :) May be a headache for DCs for sure, but what can you do.
And the way things are structured is that heating is amended and not exactly a direct connection to residential buildings at the mercy of DC load.
> residential buildings at the mercy of DC load.
Thousands of people Google "why is my building cold" causing the AI DC to wake up and pump out heat. AI sees the heat and reports "heating is on, you're fine" then shuts down.
They are now stuck in a cycle.
It is crazy how you see so many datacenters getting built in places of high land value such as say northern virginia. Why not just build them in west virginia and build twice as many for the same spend? They'd probably even let you burn coal to power them.
I suspect it's driven by availability of existing grid infrastructure (so they don't have to wait for upgrades) and ease of permitting; the contents of the building and indeed running costs are so high that the land value is much less relevant.
Power, fiber, people.
Power is the biggest one here in Texas, so much so even this laissez-faire state has stopped new permits for data centers. All the inner connect space for grids is used up and the wait for hardware and installs is massive.
So much so we're seeing private grids being setup for data centers.
I'm guessing it's to do with regulation. There's probably some policy document that says a certain org needs to be able to access the physical servers within a certain amount of time, and so they need to be nearby, even though nobody ever does the accessing.
DC would be the kind of place with a lot of government contractors that would have this kind of stipulation, is my guess.
West Virginia also has a lot of coal. I wonder why the current Admin doesn't encourage DCs to be built next to coal. Coal communities don't care much for any regulations, DC development should be perfectly fine. Hyperscalers can brag about 100% coal powered AI.
You would think, but I live in a tiny pop state that is barely thought about, and we have coal, but the hyperscale datacenter project did get pushback. Almost entirely political. No one had an opinion in January. But around summer something flipped. I also work with an existing datacenter here, and there was a similar thing. People were going on about how cool it was, etc, for years, and then suddenly, wham, sentiment changed.
1 datacenter is neat. 1000, maybe not so much.
To add, the numbers are actually this crazy.
https://dcmap.us/states/texas/
>Texas has 623 tracked data center facilities and projects: 349 operational, 101 under construction, 156 planned, 2 paused, and 15 inactive or historical. Reported or announced power totals 96.1 GW, including 85.4 GW tied to planned or under-construction projects.
This is for just one state.
>4,998 facilities and projects nationwide, with 374.7 GW in reported or announced power figures. Of that, 318.6 GW is tied to planned or under-construction projects. Climate-risk and water-stress labels summarize current records with those fields. The directory retains 150 inactive or historical records. States are ordered by current reported or announced power.
Nearby skilled labor, police/fire infrastructure willing to handle it, and ease of demo/tours/sales for clients to install their racks. Also much more stable power.
Low-latency fiber interconnects to other nearby data centers are a core feature of datacenters.
When it all goes bust, at least you'll have some real estate.
There is a lot of electrical supply and demand is high plus incentives from Virginia itself and proximity to DC. It’s already a data center dense market
You'd have to pay folks to move to the middle of nowhere.
Engineers, really? Maybe technicians ?
Same is true for software "engineers". We don't take a professional exam or adhere to any professional licensing. We aren't legally engineers in some states/nations.
Licensed engineers are a subset, in a mathematical way. All licensed engineers are engineers, not all engineers are licensed engineers.
The licencing of some engineering is simply a relatively recent regulatory capture but that does not limit the types of work that engineering covers.
https://www.etymonline.com/word/engineer
https://en.wiktionary.org/wiki/engineer
Years ago I used to get a neat (50%) tax discount in Poland for creative / artistic work as a SWE.
Wiping tears with my Portuguese tax return this year.
You'll often find a lot of engineers, and in my experience, a lot of former navy nuke personnel, overseeing the power and facilities at large data centers. There also tend to be a number of others with engineering backgrounds working in data center operations even if a lot of their work may not be what would commonly fall under "engineering". Technicians who perform some of the break/fix and routine maintenance do tend to be lower skilled but, again, it's not uncommon to find advanced degrees. I knew someone with a Phd in Chemistry working as an entry level data center technician because they were simply looking for something different outside of academia and it was a career that had rapid advancement potential.
Yes, I worked for a bit on site at a datacenter in Fort Worth and was surprised by how many ex-nuclear-submarine engineers there were. I guess it's similar, you have a lot of high tech equipment in an environment where the climate is closely controlled.
next you're going to tell me that the guys who drive locomotives aren't engineers
Unlike the USA where anyone can be an "engineer", there are countries where words mean something and terms are protected: https://engineerscanada.ca/become-an-engineer/use-of-profess...
In Canada, members of the Royal Canadian Engineers have just as much right to the title of engineer as professional engineers do, and the operators of locomotives are also referred to as engineers.
> there are countries where words mean something
You're using engineer in a highly context specific way; yes, in some contexts, engineering is a licensed field, but the general meaning of the word is:
> Engineering is the practice of systematically applying natural science and mathematics to design and improve systems, devices, or processes that solve problems under constraints.
Which is a word-for-word description of the sort of work (…some…) software engineers do.
(Perhaps less and less "systematically applying science" these days with the slop fest out there … but the field should be doing engineering.)
Most software "engineers" never got a degree in engineering, and even fewer ever bothered to do what is necessary to enter the profession where it is regulated.
That's why I've always described myself as an engineering graduate
TL;DR: you walk around smelling and listening for issues. Most of the staff on site are doing the same thing; the rest are private security hired by customers housing within the DC. You make between $66K and $120K/annual.
There, saved you from reading a story that isn't really informative and only was written because of the politics surrounding DCs are keeping them in the news.
TBH it was informative for me because I've got family who applied for data center jobs. But I'll admit that you captured it all in your blurb. The article has a lot of fluff.
IYAM the VOCs in most data centers I've visited would be contributing to early death. I would never work in one every day. I get a headache just being in ours for more than a couple of hours. And then I can smell it in my clothes and hair for the rest of the day.
Sounds like AC and airflow does not remove it?
The smell is nostalgic.
Speaking of DCs, is there evidence that they are sucking up water? What about deteriorating air quality around the buildings? Could they be causing major environmental issues? It seems they mostly have closed loop systems a la CPU coolers and they consume a lot of electricity that's already been earmarked for them. What about air quality and air humidity?
Yes, there's a lot of evidence on all of those things:
https://news.vcu.edu/article/northern-virginia-data-center-a...
I worked at Google datacenters for over a decade. It was a huge ordeal to request running generators because our entire ECS (environment control services) team was geared towards NOT having to run them. I know AWS and Microsoft both have a similar situation. The referenced article just seems like senseless fear mongering.
There have been suggestions that the DC operators themselves have been astroturfing articles about water use to steer discussions away from the other issues.
The overwhelming, #1 issue should be the fossil fuel use. So maybe I can understand them promoting the water issue as a limited hangout (a sacrificial topic to act as a lightning rod to hide the other topic)
That sounds backwards to me. Fossil fuel use being the main issue is also mighty hypocritical for a society tarrifing cheap electric cars and solar panels.
Most data centers are closed-loop, like your air conditioner. Does your air conditioner use water?
Presumably there are some data centers that use water cooling. I think google and amazon's big ones do. But even then, they're not "polluting", they're using them as heat sinks.
There was a lame rolling stone article that claimed amazon was "polluting" in Eastern Oregon, but what Amazon was doing was taking brined water from an aquifer and putting more concentrated brined water back into the water table. Air conditioning doesn't really add brine, so even the premise is ridiculous. I know that article has been referenced a lot by the "Data Centers Pollute The Water" folks.
I'm not convinced that most DC use closed loop cooling. The recent discussions had a lot of water and power permitting topics.
Secondly drawing brine water from the aquifer to then putting the stronger brine back is essentially water consumption. I would also say it's externalising environmental cost of the stronger brine impact.
> Does your air conditioner use water?
Yes, it does. As far as water is concerned, air conditioners are open loop. Air conditioners make water from condensation forming on the inside (cool) coils. Modern window air conditioners then splash that water on the outside hot coils, to benefit from evaporative cooling. I haven't looked into the details about data center water usage, but I would guess the cited usage is from evaporative cooling.
You don't have to downvote me. I am genuinely interested in the impact. I just want to learn.
What research have you done so far? I assume you’re being downvoted because those questions are fairly easily answered, there is a lot of reports and data on DCs resource usage
[dead]