Powering Datacenters

On Sep 20, 2007, at 3:21 PM, chuck goolsbee wrote:

>> I suspect no one has written an inside-the-data-center view of this
>> in any depth -- I can't find one.
>
> Not from the outside of the business looking in, at least not with
> much accuracy. Within this miniature industry though, there has been
> reams of electrons wasted on it. ;)
>
> It usually goes under the guise (misnomer if you ask me) of "Green
> Datacenter" blather, but look a little and you'll find it Glenn.

I'm mindful of the fact that this thread has abandoned the "ads on
the Internet" subject but I have a question related to the de facto
topic.

[I should have started a new thread earlier, but I've taken the liberty of doing so now. -Joe]

I recall hearing about ideas to use DC power supplies in datacenter
hardware. Each normal device's power supply converts AC to DC for use
within the device. I think the idea was it would be more efficient if
that conversion was done outside the device, once, and then have DC
delivered to each device, if not to all the servers in a datacenter
then to a smaller collection like one conversion per rack. Am I
remembering this correctly and if so, what happened with that idea?
My impression is that it hasn't been adopted.

At 3:06 AM -0700 9/22/07, Curtis Wilcox wrote:

I recall hearing about ideas to use DC power supplies in datacenter hardware. Each normal device's power supply converts AC to DC for use within the device. I think the idea was it would be more efficient if that conversion was done outside the device, once, and then have DC delivered to each device, if not to all the servers in a datacenter then to a smaller collection like one conversion per rack. Am I remembering this correctly and if so, what happened with that idea? My impression is that it hasn't been adopted.

48 Volt Direct Current power is common in telephone company datacenters to this day. It just has not been adopted for computers with any great success. There is a datacenter on another floor of our building which is mostly DC power. Our building was built to be the headquarters of a major fiber-optic network provider back in the "boom days" and this company spent many millions of Venture Capital funding on the facility before going Tango Uniform in October of 2001. <http://seattlepi.nwsource.com/venture/layoff.asp?id=351>

(Our datacenter, on a different floor, was built by a completely different venture-funded .com startup... who also went bankrupt, in December 2001... the best way to build a datacenter on a budget is move into one that somebody else spent big bucks to build! But I digress.)

Anyway, this DC-power facility sat empty until about a year ago when finally somebody who had the need for DC power on a large scale moved in. The tenant is from southern California and they hired us to provide on-site care and feeding of their equipment since we are in the same building. I've learned more about DC power than I ever wanted to know through this process.

It may be "more efficient" at some level but to be honest, I don't see how or where. At least not at this scale. The UPS gear is HUGE. Glenn remarked about the size of our units, but they are dwarfed by the mass of wet-cell batteries & banks of rectifiers used to buffer & transform the DC power system here. They literally take up a quarter of the floor. The power is not transmitted by cable around the facility, but by HUGE copper bus bars. Given how expensive it is to buy copper these days I imagine that wire is much cheaper than these giant planks of copper that measure many inches thick and extend all over the facility. Unlike AC power, where you just grab a cable and plug it into an outlet, DC requires heavy copper cable, sheathed in zinc and woven wire harness. It is attached to the equipment by stripping wire and threaded through a lug. DC power servers are fed through dual power busses, each with heavy cable. No easy plug/unplug options.

In an AC powered datacenter, it is common for 480V 3-phase power to be routed through the UPS then sent to a PDU (Power Distribution Unit) for stepping down to single-phase power at whatever voltage is required on a per-circuit basis. The PDUs are large, but you only need a few of them.

DC power has to transformed from the AC grid by large rectifiers then distributed to devices similar to PDUs where it is subdivided into individual circuits. These devices are about the size of a rack, but in order to service a datacenter you need a LOT of them.

Our facility had one of them when we moved in, here is a what it looked like: <http://chuck.forest.net/images/DC-Gear/DC-Gear-Pages/Image6.html> (front) <http://chuck.forest.net/images/DC-Gear/DC-Gear-Pages/Image7.html> (back)

The other datacenter in our building has to or three of those in every row of racks to distribute power. (I'd love to show their example, but they forbid photographs of their facilty, sorry) So you lose a *lot* of floor space to power distribution infrastructure. Datacenter efficiency is measured by how much stuff (computers, servers, watts/sq', etc) you can cram into a space. From what I can see, with DC you may gain some efficiency at the server itself, but lose it everywhere else in the facility.

Hope that helps.

--chuck

On Sep 23, 2007, at 4:27 AM, chuck goolsbee wrote:

> 48 Volt Direct Current power is common in telephone company
> datacenters to this day. It just has not been adopted for computers
> with any great success. There is a datacenter on another floor of
> our building which is mostly DC power. Our building was built to be
> the headquarters of a major fiber-optic network provider back in
> the "boom days" and this company spent many millions of Venture
> Capital funding on the facility before going Tango Uniform in
> October of 2001. <http://seattlepi.nwsource.com/venture/layoff.asp?
> id=351>

In the several years after DSL became of interest and before Qwest
noticed Port Townsend, we operated a DSLAM and ran some DSL around
Port Townsend on dry copper. (Some of our clients were in an area
that Qwest couldn't reach with DSL from the central office, two
blocks from us when they did start DSL). The DSLAM needed 48 VDC (as
one would expect based on its telco nature). So we also operated an
AC to DC converter for it. Not on the scale of Chuck's neighbor--
about half a rack total for both boxes.

When I left San Diego in 1989, SDG&E either had just stopped or was
trying to stop shipping DC around downtown San Diego to power some
legacy elevators. (They had gotten out of the steam business not
long before as I recall.)

   --John

On 23-Sep-2007, at 05:27, dr wrote:
> The biggest issue is 120 years of engineering has gone into making
> AC power for small equipment reliable, safe, and economical to set up.

Thank you Mr Tesla for standing up to that putz T. Alva Edison.

:)

It isn't catching on in datacenters, as others have pointed out, but one
area DC distribution is catching on is Power Over Ethernet, driven mainly
by corporate VOIP needs. (Note: Power over Ethernet is the opposite of the
Ethernet over power lines article I wrote for TidBITS. This is
distributing DC power over twisted pair ethernet cable. My article was on
using in-wall power wiring for networking.)

<http://en.wikipedia.org/wiki/Power_over_Ethernet>

I'm not sure if the PoE standard was developed because of VOIP or VOIP
decided to take advantage of the existing standard. Basically companies
implementing VOIP didn't want phones that plugged into the wall and into
the network. So the phone's power is provided over the ethernet cable.

I don't believe anyone makes iPod or phone chargers that work off PoE. They
would still need transformers of some sort as PoE provides 48 Vdc. AC
transformers are more efficient than a so I'm not sure this is actually a
green solution. I think the proposed laws limiting vampire power draws
might be a better solution (or putting transformers on secondary
powerstrips that actually get turned off).

<http://en.wikipedia.org/wiki/Vampire_power>

On Sep 24, 2007, at 2:35 AM, Kevin van Haaren wrote:

> I'm not sure if the PoE standard was developed because of VOIP or VOIP
> decided to take advantage of the existing standard. Basically
> companies
> implementing VOIP didn't want phones that plugged into the wall and
> into
> the network. So the phone's power is provided over the ethernet cable.

PoE is also very handy for WiFi access points, not so much for homes
but for organizations hanging a lot of them high on walls or above
hung ceilings. You don't want to intall AC power outlets up there.

> I don't believe anyone makes iPod or phone chargers that work off
> PoE. They
> would still need transformers of some sort as PoE provides 48 Vdc. AC
> transformers are more efficient than a so I'm not sure this is
> actually a
> green solution. I think the proposed laws limiting vampire power draws
> might be a better solution (or putting transformers on secondary
> powerstrips that actually get turned off).
>
> <http://en.wikipedia.org/wiki/Vampire_power>

"Vampire power?" Somebody's been reading GOP consultant Frank Luntz's
playbook.
Estate Tax -> Death Tax
Standby Power -> Vampire Power

I don't equate "off" with "dead," I would have gone with something
more parasitic like "leech power."

Curtis Wilcox wrote:
> On Sep 24, 2007, at 2:35 AM, Kevin van Haaren wrote:
>
>> I'm not sure if the PoE standard was developed because of VOIP or VOIP
>> decided to take advantage of the existing standard. Basically
>> companies
>> implementing VOIP didn't want phones that plugged into the wall and
>> into
>> the network. So the phone's power is provided over the ethernet cable.
>
> PoE is also very handy for WiFi access points, not so much for homes
> but for organizations hanging a lot of them high on walls or above
> hung ceilings. You don't want to install AC power outlets up there.

Another thing is when you want to isolate the data systems from the rest of the junk. Copiers, coffee pots, etc... Rules and color coded outlets get ignored. And in a small shop of high end designers the boss isn't going to fire someone who plugs a space heater into the power reserved for data things only. POE just eliminates a lot of situations where this can happen.

I've found that small businesses where we isolate the data "things" run much more smoothly than other offices.

David

On 24/9/07 07:35, Kevin van Haaren wrote:
> I don't believe anyone makes iPod or phone chargers that work off PoE. They
> would still need transformers of some sort as PoE provides 48 Vdc.

FWIW, Apple's Airport Extreme base station is (or at least was) available in a
variant supporting PoE.


[I'm pretty sure this is a past thing. -Adam]


> I think the proposed laws limiting vampire power draws
> might be a better solution (or putting transformers on secondary
> powerstrips that actually get turned off).

Getting somewhat OT, but I love this concept:

http://www.jackgodfreywood.co.uk/switch.htm

- Ben

On Sep 25, 2007, at 6:47 AM, David Ross wrote:

> And in a small shop of high end designers the boss isn't going to
> fire someone who plugs a space heater into the power reserved for
> data things only.

You mean like the off-duty sergeants on contract cleaning task who
took Fort Huachuca off the military command teletype network by
plugging a floor buffer into the wrong place. Pentagon was NOT happy
when the base vanished during the Viet Nam war.

   --John

johnbaxterlistsmac.com wrote:
>
> On Sep 25, 2007, at 6:47 AM, David Ross wrote:
>
>> And in a small shop of high end designers the boss isn't going to
>> fire someone who plugs a space heater into the power reserved for
>> data things only.
>
> You mean like the off-duty sergeants on contract cleaning task who
> took Fort Huachuca off the military command teletype network by
> plugging a floor buffer into the wrong place. Pentagon was NOT happy
> when the base vanished during the Viet Nam war.
>
It IS easier to fire the cleaning crew. It is hard to fire your right hand man. They tend to just pay for the damage and move on. Some yelling here and there but ...