More Software Bugs with Boeing's Starliner

[ThePointblank]

From Ars Technica and Space News:

 

https://arstechnica.com/science/2020/02/starliner-faced-catastrophic-failure-before-software-bug-found/

 

Quote

During its quarterly meeting on Thursday, NASA's Aersopace Safety Advisory Panel dropped some significant news about a critical commercial crew test flight. The panel revealed that Boeing's Starliner may have been lost during a December mission had a software error not been found and fixed while the vehicle was in orbit.

 

The software issue was identified during testing on the ground after Starliner's launch, said panel member Paul Hill, a former flight director and former director of mission operations at Johnson Space Center in Houston. The problem would have interfered with the service module's (SM) separation from the Starliner capsule.

 

"While this anomaly was corrected in flight, if it had gone uncorrected it would have led to erroneous thruster firing and uncontrolled motion during SM separation for deorbit, with the potential for catastrophic spacecraft failure," Hill said during the meeting.

 

https://spacenews.com/nasa-safety-panel-calls-for-reviews-after-second-starliner-software-problem/

 

Quote

MOUNTAIN VIEW, Calif. — A NASA safety panel is recommending a review of Boeing’s software verification processes after revealing there was a second software problem during a CST-100 Starliner test flight that could have led to a “catastrophic” failure.

 

That new software problem, not previously discussed by NASA or Boeing, was discussed during a Feb. 6 meeting of NASA’s Aerospace Safety Advisory Panel that examined the December uncrewed test flight of Starliner that was cut short by a timer error.

 

That anomaly was discovered during ground testing while the spacecraft was in orbit, panel member Paul Hill said. “While this anomaly was corrected in flight, if it had gone uncorrected, it would have led to erroneous thruster firings and uncontrolled motion during [service module] separation for deorbit, with the potential for a catastrophic spacecraft failure,” he said.

 

Basically, during the December's widely publicized failure of Boeing's Starliner in reaching the ISS due to a software glitch that caused the clocks on the Starliner to be set incorrectly, there was also a second, much more serious software glitch.

 

During today's NASA's Aersopace Safety Advisory Panel (ASAP) meeting, it was revealed by the panel that there was a second software glitch, that nearly lead to the loss of Starliner. The issue would have caused excessive thruster firing and uncontrolled motion during the separation process from the main spacecraft from the service module during the deorbiting procedure. Had the issue not been noticed and quickly patched, it would have lead to the loss of the spacecraft during reentry.

 

The bug was only discovered during ground testing when the spacecraft was still in orbit, and was quickly patched two hours before the spacecraft was due to reenter Earth's atmosphere.

 

It appears NASA isn't impressed; the ASAP members are all recommending a review of not only the root cause of the software glitches, but both a review of Boeing's software verification processes, AND a comprehensive assessment of Boeing's Systems Engineering and Integration processes. The chair of the ASAP committee, Patricia Sanders, noted yet another ongoing evaluation of Boeing, stating "Given the potential for systemic issues at Boeing, I would also note that NASA has decided to proceed with an organizational safety assessment with Boeing as they previously conducted with SpaceX."

 

Boeing is taking a $410 million dollar charge against its earnings in part to cover the costs of a second uncrewed flight, and the decision if NASA is going to demand a second flight will come by the end of February, and it appears that NASA is very likely leaning towards making Boeing re-do the entire mission again because of these two errors.

 

On the other hand, the NASA ASAP committee seemed fairly upbeat about SpaceX's Crewed Dragon; they made remarks that seemed to indicate they are the point where it is no longer a matter of IF they will be flying crew, but a matter of when.

 

[circeseye]

so not only do they cheap out on sensors on their planes, their software people are incompetent

[GDRRiley]

well I could have figured after Boeing let go of lots of staff 5-7 years ago.

wonderful work guys

[ARikozuM]

Why are we funding them again?

[GDRRiley]

23 minutes ago, ARikozuM said:

Why are we funding them again?

because they old aerospace and for a long time did a good job.

[Tech_Dreamer]

3 hours ago, ThePointblank said:

fixed while the vehicle was in orbit.

damn! that's some mission impossible stuff right there.

[Arika]

Jesus christ. What happened to Boeing... Oh right, they bought McDonald Douglas out of financial ruin for the military contracts and put the people in charge of MD in high positions in Boeing.. 

[TechyBen]

This. This is why I hate Windows 10 updates and most "games as a service" and sometimes alpha/beta (public paid for) releases. In that, the release/software is not dangerous as such, the programmers/companies "fix it later" attitude it breeds is!

 

For a person rated transport/spacecraft, you don't want to need any fixes at any time your bum is in the seat.

[TechyBen]

1 hour ago, Tech_Dreamer said:

damn! that's some mission impossible stuff right there.

Been quite common for unmanned flights, and manned flights early in the projects. But as for aircraft and current flight proven space travel (which is generally just Soyus IIRC, until Boeing/SpaceX complete their projects) it's generally safe and stable. At least it should be!

[ThePointblank]

Initial findings report from NASA is out:

 

https://blogs.nasa.gov/commercialcrew/2020/02/07/nasa-shares-initial-findings-from-boeing-starliner-orbital-flight-test-investigation/

 

There was also a media teleconference with both NASA and Boeing present to discuss the findings.

 

It ain't pretty. There's now three issues with Starliner that was discovered during the test flight. As quoted by NASA:

 

Quote

1. An error with the Mission Elapsed Timer (MET), which incorrectly polled time from the Atlas V booster nearly 11 hours prior to launch.
2. A software issue within the Service Module (SM) Disposal Sequence, which incorrectly translated the SM disposal sequence into the SM Integrated Propulsion Controller (IPC).
3. An Intermittent Space-to-Ground (S/G) forward link issue, which impeded the Flight Control team’s ability to command and control the vehicle.

 

NASASpaceFlight.com and Ars Technica both have articles as well:

 

https://www.nasaspaceflight.com/2020/02/boeing-nasa-admit-multiple-anomalies-starliner-mission/

 

https://arstechnica.com/science/2020/02/boeings-starliner-problems-may-be-worse-than-we-thought/

 

In short, the problem definitely seems much worst than initially thought. The first two errors could have easily lead to the loss of the spacecraft during the mission had ground controllers not intervened. More worryingly, the second issue related to the Service Module disposal sequence would have never been found had the first issue with the Timer not happened.

 

The third issue related to the ability communicate with Starliner minutes after launch is currently being ascribed to potential interference from cell phone towers.

 

Boeing is going to have to re-verify all of the code on Starliner, which is about 1 million lines of code. That will easily take months, and with NASA breathing down Boeing's neck and getting involved, it could easily be a year or more until this process is complete, assuming everything goes right and nothing more is found.

 

Reflecting on this, the key and shocking point to me is just how basic the critical failures during OFT were. They were not subtle, complex, or unexpected conditions but fundamental normal parts of any flight. These issues should have been caught long before Starliner went on the pad and lifted off, and the only way these problems managed to sneak through is through some very serious and systematic issues with Boeing’s processes. This needs to be addressed before a second flight is even attempted, and even then, I suspect there is going to be a need to repeat the unmanned flight test to demonstrate flawless operation in order to restore confidence and trust here. With process failures this bad we have to assume the worst case scenario in that more bugs will be found that are currently unknown. Fixing bugs is not without risk of introducing unintended consequences, and these will also need to be tested to ensure complete reliability.