Boeing Built Deadly Assumptions Into 737 Max, Blind to a Late Design Change

 

 

 There is nothing like two essentially identical crash events to prove the existence of a design failure.  

 

It is also possible that because of departmentalization, that no single decision is too blame.  This is a really good reason to dump the doctrine of compartmentalization and replace it rigorously with the Rule of Twelve.

 

Yet departmentalization happens to be the cheap way of securing your data, but at the expense of technical security.

 

What is clear though is that they knew enough after the first crash to look at the sensor system where they immediately discovered the lack of redundancy and the need for grounding.  They had the information.  And the pilot is no backup when a plane has crashed..  .

 

 

Boeing Built Deadly Assumptions Into 737 Max, Blind to a Late Design Change

 
After Boeing removed one of the sensors from an automated flight system on its 737 Max, the jet’s designers and regulators still proceeded as if there would be two


By Jack Nicas, Natalie Kitroeff, David Gelles and James Glanz

 
June 1, 2019

https://www.nytimes.com/2019/06/01/business/boeing-737-max-crash.html?

SEATTLE — The fatal flaws with Boeing’s 737 Max can be traced to a breakdown late in the plane’s development, when test pilots, engineers and regulators were left in the dark about a fundamental overhaul to an automated system that would ultimately play a role in two crashes.

A year before the plane was finished, Boeing made the system more aggressive and riskier. While the original version relied on data from at least two types of sensors, the final version used just one, leaving the system without a critical safeguard. In both doomed flights, pilots struggled as a single damaged sensor sent the planes into irrecoverable nose-dives within minutes, killing 346 people and prompting regulators around the world to ground the Max.

But many people involved in building, testing and approving the system, known as MCAS, said they hadn’t fully understood the changes. Current and former employees at Boeing and the Federal Aviation Administration who spoke with The New York Times said they had assumed the system relied on more sensors and would rarely, if ever, activate. Based on those misguided assumptions, many made critical decisions, affecting design, certification and training.

“It doesn’t make any sense,” said a former test pilot who worked on the Max. “I wish I had the full story.”

While prosecutors and lawmakers try to piece together what went wrong, the current and former employees point to the single, fateful decision to change the system, which led to a series of design mistakes and regulatory oversights. As Boeing rushed to get the plane done, many of the employees say, they didn’t recognize the importance of the decision. They described a compartmentalized approach, each of them focusing on a small part of the plane. The process left them without a complete view of a critical and ultimately dangerous system.

The company also played down the scope of the system to regulators. Boeing never disclosed the revamp of MCAS to Federal Aviation Administration officials involved in determining pilot training needs, according to three agency officials. When Boeing asked to remove the description of the system from the pilot’s manual, the F.A.A. agreed. As a result, most Max pilots did not know about the software until after the first crash, in October.

“Boeing has no higher priority than the safety of the flying public,” a company spokesman, Gordon Johndroe, said in a statement.

He added that Boeing and regulators had followed standard procedures. “The F.A.A. considered the final configuration and operating parameters of MCAS during Max certification, and concluded that it met all certification and regulatory requirements,” Mr. Johndroe said.

At first, MCAS — Maneuvering Characteristics Augmentation System — wasn’t a very risky piece of software. The system would trigger only in rare conditions, nudging down the nose of the plane to make the Max handle more smoothly during high-speed moves. And it relied on data from multiple sensors measuring the plane’s acceleration and its angle to the wind, helping to ensure that the software didn’t activate erroneously.



Then Boeing engineers reconceived the system, expanding its role to avoid stalls in all types of situations. They allowed the software to operate throughout much more of the flight. They enabled it to aggressively push down the nose of the plane. And they used only data about the plane’s angle, removing some of the safeguards.

 

[  And they tought that this did not demand a complete workup after. - arclein ]

 


Ray Craig, shown in a 2003 Boeing magazine, was the chief test pilot when he put the Max through maneuvers in a flight simulator in 2012.Creditvia Boeing's Aero Magazine


The disasters might have been avoided, if employees and regulators had a better understanding of MCAS.

A test pilot who originally advocated for the expansion of the system didn’t understand how the changes affected its safety.   [ That was not his job - arclein ] Safety analysts said they would have acted differently if they had known it used just one sensor. Regulators didn’t conduct a formal safety assessment of the new version of MCAS.

The current and former employees, many of whom spoke on the condition of anonymity because of the continuing investigations, said that after the first crash, they were stunned to discover MCAS relied on a single sensor.

“That’s nuts,” said an engineer who helped design MCAS.

“I’m shocked,” said a safety analyst who scrutinized it.

“To me, it seems like somebody didn’t understand what they were doing,” said an engineer who assessed the system’s sensors.

[  They still did notground the aircraft then and there - arclein ]

MCAS Is Born
In 2012, the chief test pilot for the Max had a problem.

During the early development of the 737 Max, the pilot, Ray Craig, a silver-haired retired Navy airman, was trying out high-speed situations on a flight simulator, like maneuvers to avoid an obstacle or to escape a powerful vortex from another plane. While such moves might never be necessary for the pilot of a passenger plane, the F.A.A. requires that a jet handle well in those situations.

But the plane wasn’t flying smoothly, partly because of the Max’s bigger engines. To fix the issue, Boeing decided to use a piece of software. The system was meant to work in the background, so pilots effectively wouldn’t know it was there.

Mr. Craig, who had been with Boeing since 1988, didn’t like it, according to one person involved in the testing. An old-school pilot, he eschewed systems that take control from pilots and would have preferred an aerodynamic fix such as vortex generators, thin fins on the wings. But engineers who tested the Max design in a wind tunnel weren’t convinced they would work, the person said.

Mr. Craig relented. Such high-speed situations were so rare that he figured the software would never actually kick in. [ so far so good ]

To ensure it didn’t misfire, engineers initially designed MCAS to trigger when the plane exceeded at least two separate thresholds, according to three people who worked on the 737 Max. One involved the plane’s angle to the wind, and the other involved so-called G-force, or the force on the plane that typically comes from accelerating.




A Boeing 737-800 flight simulator. When Mr. Craig simulated high-speed maneuvers for the Max, it didn’t fly smoothly, so Boeing settled on MCAS for a fix.

The Max would need to hit an exceedingly high G-force that passenger planes would probably never experience. For the jet’s angle, the system took data from the angle-of-attack sensor. The sensor, several inches long, is essentially a small wind vane affixed to the jet’s fuselage.

Adding More Power

On a rainy day in late January 2016, thousands of Boeing employees gathered at a runway next to the 737 factory in Renton, Wash. They cheered as the first Max, nicknamed the Spirit of Renton, lifted off for its maiden test flight.

“The flight was a success,” Ed Wilson, the new chief test pilot for the Max, said in a news release at the time. Mr. Wilson, who had tested Boeing fighter jets, had replaced Mr. Craig the previous year.

“The 737 Max just felt right in flight, giving us complete confidence that this airplane will meet our customers’ expectations,” he said.

But a few weeks later, Mr. Wilson and his co-pilot began noticing that something was off, according to a person with direct knowledge of the flights. The Max wasn’t handling well when nearing stalls at low speeds.

In a meeting at Boeing Field in Seattle, Mr. Wilson told engineers that the issue would need to be fixed. He and his co-pilot proposed MCAS, the person said.

The change didn’t elicit much debate in the group, which included just a handful of people. It was considered “a run-of-the-mill adjustment,” according to the person. Instead, the group mostly discussed the logistics of how MCAS would be used in the new scenarios.

“I don’t recall ever having any real debates over whether it was a good idea or not,” the person said.


The change proved pivotal. Expanding the use of MCAS to lower-speed situations required removing the G-force threshold. MCAS now needed to work at low speeds so G-force didn’t apply.

The change meant that a single angle-of-attack sensor was the lone guard against a misfire. Although modern 737 jets have two angle-of-attack sensors, the final version of MCAS took data from just one.




Ed Wilson, right, with his co-pilot, Craig Bomben, after the first Max test flight in 2016


Using MCAS at lower speeds also required increasing the power of the system. When a plane is flying slowly, flight controls are less sensitive, and far more movement is needed to steer. Think of turning a car’s steering wheel at 20 miles an hour versus 70.

The original version of MCAS could move the stabilizer — the part of the tail that controls the vertical direction of the jet — a maximum of about 0.6 degrees in about 10 seconds. The new version could move the stabilizer up to 2.5 degrees in 10 seconds.

Test pilots aren’t responsible for dealing with the ramifications of such changes. Their job is to ensure the plane handles smoothly. Other colleagues are responsible for making the changes, and still others for assessing their impact on safety.

Boeing declined to say whether the changes had prompted a new internal safety analysis.

While the F.A.A. officials in charge of training didn’t know about the changes, another arm of the agency involved in certification did. But it did not conduct a safety analysis on the changes.


The F.A.A. had already approved the previous version of MCAS. And the agency’s rules didn’t require it to take a second look because the changes didn’t affect how the plane operated in extreme situations.

“The F.A.A. was aware of Boeing’s MCAS design during the certification of the 737 Max,” the agency said in a statement. “Consistent with regulatory requirements, the agency evaluated data and conducted flight tests within the normal flight envelope that included MCAS activation in low-speed stall and other flight conditions.”

 
‘External Events’

After engineers installed the second version of MCAS, Mr. Wilson and his co-pilot took the 737 Max for a spin.

The flights were uneventful. They tested two potential failures of MCAS: a high-speed maneuver in which the system doesn’t trigger, and a low-speed stall when it activates but then freezes. In both cases, the pilots were able to easily fly the jet, according to a person with knowledge of the flights.

In those flights, they did not test what would happen if MCAS activated as a result of a faulty angle-of-attack sensor — a problem in the two crashes.

Boeing engineers did consider such a possibility in their safety analysis of the original MCAS. They classified the event as “hazardous,” one rung below the most serious designation of catastrophic, according to two people. In regulatory-speak, it meant that MCAS could trigger erroneously less often than once in 10 million flight hours.

 
 

 



Boeing Max fuselages on their way to an assembly plant. The company declined to say whether it had conducted a new safety analysis of the revised MCAS.

That probability may have underestimated the risk of so-called external events that have damaged sensors in the past, such as collisions with birds, bumps from ramp stairs or mechanics stepping on them. While part of the assessment considers such incidents, they are not included in the probability. Investigators suspect the angle-of-attack sensor was hit on the doomed Ethiopian Airlines flight in March.


Bird strikes on angle-of-attack sensors are relatively common.

A Times review of two F.A.A. databases found hundreds of reports of bent, cracked, sheared-off, poorly installed or otherwise malfunctioning angle-of-attack sensors on commercial aircraft over three decades.

Since 1990, one database has recorded 1,172 instances when birds — meadowlarks, geese, sandpipers, pelicans and turkey vultures, among others — damaged sensors of various kinds, with 122 strikes on angle-of-attack vanes. The other database showed 85 problems with angle-of-attack sensors on Boeing aircraft, including 38 on 737s since 1995.

And the public databases don’t necessarily capture the extent of incidents involving angle-of-attack sensors, since the F.A.A. has additional information. “I feel confidence in saying that there’s a lot more that were struck,” said Richard Dolbeer, a wildlife specialist who has spent over 20 years studying the issue at the United States Department of Agriculture, which tracks the issue for the F.A.A.

 
A Simple Request

On March 30, 2016, Mark Forkner, the Max’s chief technical pilot, sent an email to senior F.A.A. officials with a seemingly innocuous request: Would it be O.K. to remove MCAS from the pilot’s manual?

The officials, who helped determine pilot training needs, had been briefed on the original version of MCAS months earlier. Mr. Forkner and Boeing never mentioned to them that MCAS was in the midst of an overhaul, according to the three F.A.A. officials.

Under the impression that the system was relatively benign and rarely used, the F.A.A. eventually approved Mr. Forkner’s request, the three officials said.


Boeing wanted to limit changes to the Max, from previous versions of the 737. Anything major could have required airlines to spend millions of dollars on additional training. Boeing, facing competitive pressure from Airbus, tried to avoid that.

Mr. Forkner, a former F.A.A. employee, was at the front lines of this effort. As the chief technical pilot, he was the primary liaison with the F.A.A. on training and worked on the pilot’s manual.

“The pressure on us,” said Rick Ludtke, a cockpit designer on the Max, “was huge.”

“And that all got funneled through Mark,” Mr. Ludtke added. “And the pushback and resistance from the F.A.A. got funneled through Mark.”

 
Federal Aviation Administration officials said Boeing’s request to remove MCAS from the pilot’s manual didn’t mention that the system was being overhauled.CreditJason Redmond/Agence France-Presse — Getty Images

 

Like others, Mr. Forkner may have had an imperfect understanding of MCAS.

Technical pilots at Boeing like him previously flew planes regularly, two former employees said. “Then the company made a strategic change where they decided tech pilots would no longer be active pilots,” Mr. Ludtke said.

Mr. Forkner largely worked on flight simulators, which didn’t fully mimic MCAS.

It is unclear whether Mr. Forkner, now a pilot for Southwest Airlines, was aware of the changes to the system.

Mr. Forkner’s attorney, David Gerger, said his client did not mislead the F.A.A. “Mark is an Air Force veteran who put safety first and was transparent in his work,” Mr. Gerger said.


“In thousands of tests, nothing like this had ever happened,” he said. “Based on what he was told and what he knew, he never dreamed that it could.”

The F.A.A. group that worked with Mr. Forkner made some decisions based on an incomplete view of the system. It never tested a malfunctioning sensor, according to the three officials. It didn’t require additional training.

William Schubbe, a senior F.A.A. official who worked with the training group, told pilots and airlines in an April meeting in Washington, D.C., that Boeing had underplayed MCAS, according to a recording reviewed by The Times.

“The way the system was presented to the F.A.A.,” Mr. Schubbe said, “the Boeing Corporation said this thing is so transparent to the pilot that there’s no need to demonstrate any kind of failing.”

The F.A.A. officials involved in training weren’t the only ones operating with outdated information.

An April 2017 maintenance manual that Boeing provided to airlines refers to the original version of MCAS. By that point, Boeing had started delivering the planes. The current manual is updated.

Boeing continued to defend MCAS and its reliance on a single sensor after the first crash, involving Indonesia’s Lion Air.


At a tense meeting with the pilots’ union at American Airlines in November, Boeing executives dismissed concerns. “It’s been reported that it’s a single point failure, but it is not considered by design or certification a single point,” said Mike Sinnett, a Boeing vice president, according to a recording of the meeting.

His reasoning? The pilots were the backup.

“Because the function and the trained pilot work side by side and are part of the system,” he said.

Four months later, a second 737 Max crashed in Ethiopia. Within days, the Max was grounded around the world.

As part of the fix, Boeing has reworked MCAS to more closely resemble the first version. It will be less aggressive, and it will rely on two sensors.