Monday, April 23, 2012

GIGO at SanO: Steam Generator Replacement: A major failure for computer simulations?

Monday, April 23rd, 2012

Dear Readers,

It now appears that the months-long shutdown of the two reactors as the San Onofre Nuclear (Waste) Generating Station might be the result of an over-reliance on computer-aided design software, as well as a massive failure of engineering analysis in general.

Meanwhile, a 3.9 earthquake this morning struck about as far north of the plant as I am south of it. The quake was unusually sharp, and reportedly "felt like Mike Tyson punching the building." Maybe it's a REALLY good thing the plant is closed right now!

This $680-million failed replacement steam generator project appears to be on par with the failure of the hanging walkways at the Hyatt Regency Kansas City Hotel in 1981 (which killed 114 people), the "Galloping Gertie" bridge disaster across the Tacoma Narrows in 1940 (which killed... a dog) and, of course, the "unsinkable" Titanic, which went down one night in April 100 years ago, killing 1,514 people. Her sinking has been blamed variously on bad rivets, too much speed, the lookout's inattention and a number of other possible contributing factors. The large number of deaths is mainly blamed on the lack of lifeboats, which is almost exactly like the Nuclear Regulatory Commission's ridiculous 10-mile evacuation zone around our nuclear power plants. Beyond that, virtually NO planning is done: you will sink or swim on your own (and you will find out if you should leave on your own, too!). After Fukushima in Japan, our own state department recommended evacuating (Americans) from a 50-mile area around THOSE plants! Why not here?

Too costly? Yes, that too, but mainly, IT SIMPLY CAN'T BE DONE.

Last January, about a year after installation, one of the thousands of thin tubes inside one of the four new steam generators that were replaced at the two reactors at San Onofre ruptured. Some radiation was released. Does it compare to "Galloping Gertie"?

That bridge cost $6,000,000 to build (in 1940). San Onofre's steam generators cost about $680 million to replace, and doing so again will probably cost even more (they'll charge it to the ratepayers, just like last time).

When the tube inside the steam generator ruptured, primary coolant escaped into the secondary coolant loop. Primary coolant is highly pressurized, highly radioactive, fast-moving, very hot water which is also full of toxic chemicals. The super-heated primary coolant spewed into the secondary coolant loop, flashing to steam as it came out at about 1000 psi differential between the two loops.

The reactor was safely shut down, but not before the radioactive coolant that flashed to steam was condensed back to liquid at atmospheric pressure, and radioactive gases that had previously been entrapped in the highly pressurized primary coolant loop's fluid were released to the atmosphere -- and thus, to the public on I-5, which runs nearby, and at the nearby surfing beach.

Months later, both reactors remain closed. Subsequent testing and checking has just ended (for now), and has undoubtedly revealed ... who-knows-what problems to the operators of the plant? They aren't telling the public what they've found, but perhaps they've shared it with the Nuclear Regulatory Commission. Perhaps not.

It could have been poor fabrication techniques but that now appears to be the less likely culprit. However, those techniques are often proprietary, which means... we're not allowed to know what they are. But they might involve -- depending on the alloy, which depends on which part of the steam generator we're talking about -- quenching, annealing, bathing in a chemical solution to dissolve precipitates, welding, grinding, molding... and each of these processes have specific conditions such as hydrogen or nitrogen atmospheres, high (or low) temperatures and pressures and lengths of times.

Testing for proper manufacturing might include photoelasticity analysis, eddy-current testing, sampling, x-ray imaging, and maybe banging on things with a little rubber hammer to hear how they vibrate, for all I know. Maybe they skipped that step.

Below are some quotes from an article about the San Onofre steam generator replacement project itself. The article was published in October 2006 by the American Nuclear Society in a special section on Nuclear Power Plant Maintenance. It was written by an employee of Southern California Edison. [Note: "PLM" stand for "Product Lifecycle Management"]:

"The shutdowns will cost SONGS $1 million each day just for replacement power (at 4 cents per kilowatt), and so any overlooked steps or unanticipated challenges could send costs soaring... PLM [is a] business strategy that includes the use of a sophisticated suite of 3-D design, modeling, and simulation software and processes. But SONGS engineers are the first to apply this strategy to the nuclear industry, using PLM to plan and validate major operations involved in replacing the steam generators... The PLM process has already been applied to the steam generator project at SONGS. Mockup capabilities were used to verify the equipment designs developed by Mitsubishi through the use of more traditional, paper-based methods. Most contractors in the industry still use these traditional methods, and PLM allows SONGS to verify contractors' proposals quickly and affordably."

But did they verify them CORRECTLY? It appears not! They let the software do the work:

"Software also manages the relationships among various disciplines of design data, applying restraints and standards to alert design engineers to clashes or inadequate clearances and ensuring compliance with the company's design standards, as well as Nuclear Regulatory Commission regulations."

We've learned that many of the tubes were degrading because they were banging into each other. Were THOSE clearances adequately analyzed? I think not!

The software allowed Southern California Edison to: "simulate the stresses on key pieces of equipment." The tube are also clanging into their support structures, and wearing excessively because of this movement. So they evidently didn't do this step correctly. Were the tube-to-tube clearances redesigned too close together? They added hundreds of tubes, so that's a distinct possibility.

Are they running them too hot, to generate more steam, and/or with too high a flow rate or pressure? Reactor operators constantly want to "uprate" their reactors for greater output (to make more money). Are there very uneven flow patterns from tube to tube? Are there unexpected swirling circulation patterns of steam on the steam side of the tubes, causing vibration which is the root cause of both types of wear they are seeing?

Is there a "galloping Gertie" type of phenomenon going on inside the steam generators? What kind of sensors are they using? How many of them are there?

It could be a complex combination of many factors that are causing the unusual degradation. Activists would certainly like to know what the problem actually is, although the main question remains unanswered: WHY are we generating electrical power by such a dangerous method in the first place?

The glowing article in ANS continues, telling us:

"The plant refuels every 20 months, and that is enough time for workers to forget lessons learned on the previous refueling."

Kinda scary when you think about it. But it goes on to assure us that: "With the simulation software, SONGS can capture those lessons from one outage and use them again during the next outage. Time and motion studies performed in virtual 3-D on standard outage tasks also allow SONGS to identify the most efficient procedures possible, saving time and money. Officials at SONGS see PLM as a critical strategic process that enhances the plant's ability to perform one-of-a-kind projects efficiently, cost-effectively, and, most important of all, safely."

Safely? Fortunately only one steam generator tube failed while the reactor was operating (seven more failed in subsequent testing). A cascade of tube failures could have been catastrophic for all of Southern California.

The steam generator replacement project was the second use of PLM at the plant. They first led the nuclear industry by using the software to simulate a "thimble rod" repair project which required hundreds of hours of radiation exposure to divers at the plant, who went into the reactor itself. Subcontractors are always used for these sorts of things because long-term plant worker's exposure limits would be exceeded. Divers are usually young ex-Navy guys of course, who are always in plentiful supply, and who are eager to make some extra spending money for a few hour's work. They are assured their exposure will be carefully measured and limited. Many will be smokers whose lifetime polonium-210 intake from the tobacco will, indeed, way exceed their radiation exposure during their work as "thimble rod cutters" at San Onofre. And in any event, cancers would not occur for many years afterwards, and the divers would be unable to prove in court that SanO was to blame. They might not even think of it.

San Onofre's operators complained that the old manual process is: "slow and time-consuming and requires countless checks to ensure that you have covered all your bases. PLM is undoubtedly a better process..."

Perhaps PLM IS better.

When it's used right.

But when it isn't, it's worthless.

Let's keep San Onofre closed FOREVER.


Ace Hoffman
Carlsbad, CA

The author, a computer software developer, has spent more than 40 years watching the nuclear industry wreck itself and the rest of us.


The above quotes are from the following American Nuclear Society article:


Galloping Gertie:


** Ace Hoffman, Owner & Chief Programmer, The Animated Software Co.
** POB 1936, Carlsbad CA 92018
** U.S. & Canada (800) 551-2726; elsewhere: (760) 720-7261
** home page:
** email: