engineering pornography

This is one of my favorite geek stories of all time. I know I've mentioned it to a few of you recently, and you hadn't seen it, so here it is for posterity.


Date: Wed, 27 Sep 89 10:27 PDT
From: "Thomas L. Mc Mahon" <tlm@riverside.scrc.symbolics.com>
Subject: Cuts and jumpers (on a different scale)
To: Hardware@riverside.scrc.symbolics.com

[Not quite the right mailing list but close. If you don't care about megawatts, bus bars bigger than your wrist, things that cause ground loops out to Hawaii, or big hairy construction projects hit D now.]

---

Several days ago a very large number of trucks and men from the Los Angeles Department of Water and Power descended on my neighborhood. They removed large sections of Pershing drive to a depth of 15 feet or so over a stretch of about a city block. I assumed they had a problem with a water main or something.

When they started building semi-permanent structures over the holes I knew something really big was up. When the large trucks full of strange power tools, mega-welding machines, breathing equipment, and racks of test equipment came I started wondering. Driving by a couple nights ago (11 PM), I noticed that the pace hadn't slowed - they were at it 24 hours a day.

My curiosity got the best of me yesterday when they brought in the giant tanks full of liquid nitrogen. LN-2 for the DWP? I parked my car and played the lookie loo.

<lj-cut text=" --More--( 9%) ">

It turns out they have a problem with an underground wire. Not just any wire but a 230 KV, many-hundred-amp, 10 mile long coax cable. It shorted out. (Lotta watts!) It feeds (fed) power from the Scattergood Steam Plant in El Segundo to a distribution center near Bundy and S.M. Blvd.

To complicate matters the cable consists of a copper center conductor living inside a 16 inch diameter pipe filled with a pressurized oil dielectric. Hundreds of thousands of gallons live in the entire length of pipe. Finding the fault was hard enough. But having found it they still have a serious problem. They can't afford to drain the whole pipeline - the old oil (contaminated by temporary storage) would have to be disposed of and replaced with new (pure) stuff which they claim takes months to order (in that volume). The cost of oil replacement would be gigantic given that it is special stuff. They also claimed the down time is costing the costing LA $13,000 per hour. How to fix it and fast?

That's where the LN-2 comes in. An elegant solution if you ask me. They dig holes on both sides (20-30 feet each way) of the fault, wrap the pipe with giant (asbestos-looking) blankets filled with all kind of tubes and wires, feed LN-2 through the tubes, and *freeze* the oil. Viola! Programmable plugs! The faulty section is drained, sliced, the bad stuff removed, replaced, welded back together, topped off, and the plugs are thawed. I was amazed.

------

The next day:

Last night the DWP held a curbside chat to allay the neighborhood's fears that they were going to accidentally blow us all up. Apparently all the vapor clouds from all the LN-2 blowoff had caused a great deal of concern.

Interesting bits:

The feeder was laid 17 years ago and was designed to have an MTBF of 60 years. There are other similar feeders in use around California, in the Pacific North West, and some on the east coast. This was the first failure in the western US. No one out here had any idea how to fix it so they brought in experts from the east. (NYC has had some faults.)

This link is a very critical part of the LA power grid. Last night the city engineers verified the $13,000 per hour power cost figure quoted the day before. (I guess that means they are being forced to buy power off the grid somewhere else.)

There are actually three center conductors (they had a cross sectional model to show us). Each is about 3" in diameter with a one inch solid copper core. Each is wrapped with hundreds of layers of a special paper. That, in turn, is sheathed with copper and then each one is spiral-wrapped its entire segment-length with a 1/4 inch bronze "wire". The three conductors are then twisted together during the pulling process. The bronze spiral wraps form a kind of linear bushing with minimal contact area with the inside of the pipe so it's "easy" to pull each segment. Ha.

Each of the three legs in the feeder carrys 600-800 Amps (depending on demand) of 230KV three phase power. The ground return is the Santa Monica Bay. Down at the Scattergood Steam Plant and up in Santa Monica they have a giant copper anchors out in the bay.

They lay these things in 2000 foot segments. 2000' is the longest segment they can pull through the steel pipe. The pipe is laid first and then the internal cable(s) are pulled through. Tensile forces must be enormous. At each segment joint (splice) there is a very large and expensive ($100K) underground vault. Future technology may allow them to go 3000 feet, reducing the number of vaults needed per run, thereby saving money.

After the feeder was originally built (and the cable pulled) it was thoroughly evacuated to both leak test and remove any contaminants. It was flushed with dry nitrogen and then reevacuated. Golden Bear High Tension Oil was then slowly added while still maintaining a vacuum so as to "pull" any residual gas contaminants out of the oil and the cables in the pipe. The pipe, full of oil, is then pressurized to about 200 PSI for some period of time before it gets powered up. 200 PSI is maintained during operation to keep any bubbles from forming and to drive insulating oil into the paper.

At both ends of the pipeline they have 6000 gallon tanks of Golden Bear lightly pressurized under a blanket of dry nitrogen. There are pumps at both ends. There is about 100K gallons in the entire pipeline, not including the 6K gal tanks. Every six hours they reverse the pumps so the oil oscillates back and forth in the pipe. The pumps only run at 3 gallons per minute but that is enough, over 6 hours, to get the oil in each 2000 foot segment to go at least a segment or two length in either direction. This eliminates hot spots in the copper conductors and spreads the heat out over several thousand feet. A little competitive pressure is always maintained between the pumps to get the 200 PSI.

They learned the hard way that you simply don't reverse the pumps lest you get the Golden Bear equivalent of water hammer. The last hour of every 6 hour cycle is spent slowly reducing the oil velocity down to zero before you reverse it and then slowly ramp back up in the other direction.

In between segments, in the vaults, are temperature sensors embedded in the pipe. These monitor the oil temperature. These are wired to a computer downtown. Because the oil oscillates, the DWP can track the temperature gradient along the pipe and get an early indication of the location of any hot spot problems. They have regularly spaced flow rate and pressure monitors for the same purpose - detecting and isolating faults.

Every vault also has a nipple which allows sampling of the pipe oil. They said you withdraw the oil through a thick membrane with a syringe (?). This happens monthly on all feeders in the LA area. The samples are analyzed downtown by a staff of chemists who can relate the presence of things like acetylene, butane, and benzene in the oil to arcing, coronas, and so forth. Apparently the oil chemistry is a very good indicator of the health of the segments.

One of their worst fears, after they open up the pipe, is having a blowout of the freeze plugs. If they ever run out of nitrogen during the repair process they'll lose one side of the pipe (or both). Right now they've got the pipe on each side of the fault dropped down to 80 PSI. They are afraid that if they go any lower in oil pressure any gas in the oil will come out of solution and cause an explosive expansion. Not only that, but since there is so much oil embedded in the paper insulation, any sort of gas bubbling (oil foaming) would shred the insulation, rendering the entire feeder useless. They say it could take months to safely let the pressure off to zero. (That is the other reason ($13k/hr) they cannot afford to drain the whole pipe.)

Even at 80 PSI, if they lose a freeze plug they will have a really big mess outside the pipeline. The holes they've dug cannot hold 100K gallons and they're operating on a hill near the beach anyway... (Big pollution threat for LA basin.) Potentially fatal for anyone around. Right now they have LN-2 companies on call from San Diego to San Francisco with contingency plans of all sorts in case there is a major traffic problem with trucks getting in.

They say the repair could take weeks or more, depending on what they find when they get inside. They believe the cause of the fault was the inner conductors slipping downhill inside the pipe and shorting against a metal flange. Even if that's true they wonder where it slipped to, and hence, where it may be bunched up down hill.

Finding the fault was a problem in itself. Since this was all new to them they really didn't know how to start. They tried time-domain reflectometry equipment but got inconclusive information. They tried ultrasound and radar but that didn't work. Then they got a thing called a "thumper" shipped in which got them pretty close. The thumper sends mondo-amp pulses into one end of the cable. The electromotive force tends to cause physical displacement of the conductors which you can hear from the street level. The place where the clicking stops is where the problem usually is. This got them to the defective segment.

What pinpointed the problem in the end was a bunch of car batteries and some millivoltmeters. (From one technology extreme to the other.) They hooked up car batteries to both ends, tapped the cable at several points (maybe there are taps in the vaults?) and, knowing the drops and resistance of the cable, got within a few feet of the fault. (I used to use the exact same technique on memory boards.)

Next came the X-Ray equipment. Sure enough, they can see the cable shorting against the steel wall of the pipe.

Once all of the repair is done they still have to close it up. How do you weld a steel pipe with paper insulation inside? Slowly. They have special heliarc welding equipment and "certified operators" who take 8 hours to weld around one cross section of pipe. They are required to keep their hand on the pipe no more than 3 inches from the tip of the welder. If it is too hot for their hand they stop and let it cool. After all, they can't afford another failure.

------

Oct 20, 1989 Update

I am getting all of these bits from a guy named Jim who is the project manager. He looks kinda like a red neck RWK (Jesus in a hard hat with a Harley belt buckle). [RWK is Bob Kerns, an ex-Symbolics person, 6'7" tall, skinny, bearded. -- DLW] He is a really great guy. Jim was one of the splicers on the project 17 years ago when he was working his way through school. He is a now professional electrical and mechanical engineer. After having worked his way up through the ranks at the DWP he is now The Big Boss. He claims to be having the time of his life - back in the field with one of the biggest challenges of his career. If we ever recruit a VP of engineering I would hope its someone like him.

So, what went wrong? Varying load conditions in the three legs of the 3-phase circuit caused tremendously strong and dynamic magnetic field changes. The electromagnetic forces between the three conductors and the steel pipe (gack!) cause the conduit to wiggle around inside the pipe. Over many years (and under the influence of gravity) the thing slipped and wiggled every which way. Also, due to very slight diametric temperature gradients, the differential thermal expansion of a cable that big across causes bending and warping forces. Nobody ever thought of any of this.

Wiggle alone may not have cause the problem, however. The spices between cable segments are much larger in diameter than the cable itself. The steel pipe at these points is much larger than the main run. So the whole affair get fat and then shrinks down every 2000 feet or so. What really screwed them was failure to put any sort of clamp at the splices to keep the fat splice from getting pulled into the narrower main runs. This is what cause the fault.

Jim says the fault lasted 20 milliseconds before breakers tripped. (The breakers for a wire like this are pretty amazing in their own right. They use high pressure gas to blow out the arc as the circuit begins to open. Anything that can cut off this number of megawatts in 20 ms gets my respect.) It blew carbonized oil about 3000 feet down the pipe to either side of the fault. (Compute velocity...)

They will be removing a long length of cable from the faulted area for analysis. The entire length will be dissected. Jim says the insulation they have inspected at so far looks like shredded cauliflower due to the explosion from the fault and the gas bubbling in it. (BTW - The insulation consists of 118 layers of paper tape.)

Based on X ray imaging they are going to have to open up 14 of the 23 splices along the 10 mile run. They'll have to drain the pipe to do so. It will take them 2 months to take the pipeline down (depressurize and drain). (The oil will be recycled - see below.) At each of the opened splices they are going to install special aluminum (non magnetic) collars around the conductors to keep the splices from getting pulled into the necked-down section of the pipe. These collars are being specially fabricated now and will be ready in about a month.

At each splice they have to build a semi clean room to keep dirt, moisture, worker sweat, and any other contaminants out of the joint before closing. After all, we're talking a quarter megavolt! They have special air conditioning and filter units for the vaults. Each joint will take two months of work. They will get some degree of parallelism in the phase of the project.

After repairing and replacing the faulted section of cable, stabilizing all of the splices, and buttoning it all up comes the job of putting the oil back in. First the pipe is evacuated and then back-filled with nitrogen etc as I described earlier. After extensive filtering, the oil is heated to about 230 degrees farenheit. It gets injected into a vacuum chamber at the temperature thorough hundreds of spray nozzles. This gets the maximum possible surface area so all the crap in it boils out into the chamber. The good stuff that's left is collected and pumped immediately into one end of the pipeline.

Then they power it up and see if it works. If not, they start over again. I'll keep you posted.

----

12/11/89

The pipeline is now completely drained and filled with nitrogen at atmospheric pressure. They are still in the process of opening up many of the splices to install the collars. They originally planned on doing 14 of them but now the number is 17.

The splice where the short happened, and the cable for about 40 feet to either side of it, has been completely replaced. Engineering prototypes of the aluminum collars are now installed there and the whole thing is all welded back together. They built new vaults under the street where the new splices are. The trucks, men, and their equipment are all gone and the roadbed has been repaved. They consider the segment where the short happened "fixed". We'll see...

Now that they verified the assembly of a splice with the new collars in it (to make sure they had the details right) they are going ahead with the fab run for the 17 other joints. They won't be done installing them until mid April. Then they'll put the oil in as described above and power it up slowly. I don't expect to have much to report until then and I promise to send an update.

Assuming 8 month down time the cost of the electricity alone will approach $75 million!

------

[after many queries of "So how did it all work out?"]

... I am sure that you will all get this final message in due time.

You see, I found out yesterday that even the Department of Water and Power repair crew got copies! It rattled across electronic bulletin boards around the country and eventually ended up in the hands of a DWP manager who promptly showed it to the team. Small world. Every time you turn over a rock you find that we are all connected.

So, here's the scoop:

They are back on line. The failure was attributed to "TMB", short for Thermal Mechanical Bending. There have been several similar failures on the east coast but this was the first out here. TMB causes the cable to wiggle in place due to load surges. This eventually causes insulation failure due to abrasion against the pipe and separation of the many layers of paper tape. They repaired the short, put aluminum collars in most of the joints to hold the splices in place, and have added a load management scheme to reduce the current peaks. They powered it up and the darned thing works. Amazing.

I'll impulse them again in a few months to see if there is any news.

Tags: , , ,

9 Responses:

  1. mcgroarty says:

    The notion of the "thumper" (and the initial variances which worried the insulation) sending enough power through to physically displace something that size (and designed with high conductivity in mind in the first place) blows my mind. I want to use "Akiras" as a unit of power here.

    the fault lasted 20 milliseconds before breakers tripped. (The breakers for a wire like this are pretty amazing in their own right. They use high pressure gas to blow out the arc as the circuit begins to open. Anything that can cut off this number of megawatts in 20 ms gets my respect.) It blew carbonized oil about 3000 feet down the pipe to either side of the fault.

    These are two different things? The breaker, I get. Slam that fucker off, and kill the arc as fast as you can.

    But the carbonized oil -- this is at the point of the fault, not the breaker? In the 20ms spent heating due to the short, it cooked enough oil/insulation to throw contamination half a mile in each direction through pressurized oil?!

    Man am I glad these things are well below ground.

  2. See, and here I was thinking I understood, at least on a basic level, most neato engineering feats. I had absolutely no idea that this sort of thing even existed, or what you did with it when it broke. This is *awesome*.

    Is there a list for this kind of exposition of amazing stuff, that you know of? Maybe there should be...

  3. Thank, it's a sweet story.
    I've read about the cables before, in a book called "Underneath New York". Those mentioned a slightly different technology :
    "132,000 volt cable, each carried in a separate lead sheath, are hollow copper tubes, filled with oil at a pressure of twenty pounds. If you are wondering how electricity can be sent through a hollow tube, the answer, I am afraid, will be most mystifying. The electrons do not travel through a conductor. They are in too much of a hurry. They shoot from atom to atom near the outside surface of the conductor! Actually they constitute a field around the conductor. The conductor seems merely to hold them in a directed path."

    The book is a reprint, the original was written in 1947.

    I dog eared that page because I wanted to write about that cable in my journal (and the book was in a pile of books which I wanted to mention.)

    • unabomber says:

      At least twice since I've moved to New York (1999), I've seen large tanks of Liquid Nitrogen sitting on sidewalks with hoses running underground. I always assumed it was either for something like this, or some kind of high-test welding going on underground.

      What really amazes me is that these big tanks could sit around without getting stolen or broken...

      -J

      • Actually, now that you mention it, yeah, there is a tank like that near my office. I am not sure that it's liquid nitrogen. I'll chat up a tech to find out.

        There is something like thatmentioned on usenet.

        I especially like this explanation:

        It has to do with the alligators in the sewers who sometimes get a bit
        adventurous and wander out of their normal havens...

  4. shadowblue says:

    That's incredibly, incredibly cool. And completely insane. :)

    I have to reset my difficulty meter for engineering problems now.

  5. jruske says:

    It's interesting that NYC is mentioned several times as an area of expertise for this. A year or so back, the Canadian power companies were reviewing a plan to export power directly into NYC (high demand market provides higher revenue opportunities). I don't think the proposal has died out, but it definitely met with some serioous resistance from several sectors as the junction interface into the city would require a couple of city blocks, the path would have potentially massive environmental impacts, and never mind the potentially massive single point of failure at the receiving substation...

    The general plan was to run similar high power density cables directly from the east coast of Canada, into the Atlantic, and then down to NYC where it would resurface into a massive power substation. The junction point into either Long Island or all the way to Manhattan (there were competing plans and northern NJ was also a contender) would involve several very long runs of this "power pipeline."

    Since the most efficient way of transporting the power was in a more or less 'raw' format, it would have been very high voltage and high amperage much like and no doubt greater than the conduit described in California. Add the underwater fun, digging through the seabed for good measure, and then having to surface that and convert all that power into a standard transmission format, and it would have been some seriously cool engineering with a huge cost for maintenance should anything go wrong.

    Ironically the first concern was how laying the pipeline might upset the fisheries. It took the environmentalists a bit to grasp that these pipes would be full of lots and lots of oil under high pressure. (snicker) I imagine if no one had explained it to them in detail they'd still be picturing an overhead wire 2 meters in diameter thinking it was just coiled wire and paper...

    • unabomber says:

      You say "A year or so back, the Canadian power companies were reviewing a plan to export power directly into NYC (high demand market provides higher revenue opportunities). I don't think the proposal has died out, but it definitely met with some serious resistance"

      There are so many bad engineering puns to be made about serious resistance and a thousand miles of cable, so I'll just shut up.

      -J

  6. kragen says:

    http://www.cs.auckland.ac.nz/~pgut001/misc/mercury.txt was where I first heard about these oil-filled cables; Auckland had a weeks-long power failure in 1998 when all three of their power cables failed. Rereading it, it looks like bits of it were copied from the above. (It couldn't have been the other way around because the above text is from 1989, and Gutmann's text is from 1998.)