This post- What Engineers Do During The Test Phase, is part 5 of the ‘Role of an Engineer’ series.
If you missed parts 1 through 4 of this series you definitely should read those first since these phases build on one another. You can find the Welcome post here, the Define phase post here, the Design phase post here, and the Develop phase post here.
This series is based on my experience as an engineer. Although the New Product Development Process is used in many industries, the exact tasks may vary.
The five phases of the New Product Development (NPD) Process include:
- Define
- Design
- Develop
- Test
- Launch
This post will focus on the Test phase and dive into what an engineer does during this phase.
Test Phase
Benjamin Franklin wisely explained, “The bitterness of poor quality remains long after the sweetness of meeting the schedule has been forgotten.” This defines the test phase perfectly. A lot of an engineer’s time is dedicated to testing and addressing failures.
This is the last phase before launch. It cannot be rushed, even though you are so close to launching the product. An engineer’s job is to work through this phase diligently, which is critical to the success of the project.
If you rush through testing to meet a schedule and miss something, the impact will be much worse than delaying your schedule to address the proper testing and any failures that occurred. The result of missing a failure mode in the test phase could be devastating and may cause harm to your customers. Engineers will work closely with quality during this phase.
The key tasks an engineer is responsible for during the Test phase include:
- Executing all test methods and recording the data
- Performing statistical analysis on the data
- Documenting any failures that occur and performing an investigation to determine what happened
- Finding solutions to failures, potentially re-build and re-testing when necessary
Learning Curve
As a new engineer, I did not know a lot about this phase. When I had to take my first project as an engineer through the Test phase, there was a lot of on the job training.
I utilized test protocols that were used on other products. This helped me understand all of the testing we would have to consider. Then, I used other test reports to learn the type of data analysis conducted. The Test phase was not really covered in college. We had labs, but we mainly just collected data and did some analysis. For example, mean, median, standard deviation were calculated. However, there was no context given to these results.
There was no discussion over what the product specification was, why we were testing, or what to do if there is a failure. I gained some of this knowledge through my last internship.
Therefore, make sure you are looking for this information if you do have an internship. If someone tells you to go test something, ask what is the testing being used for?
What is the specification?
How are they analyzing the data after it is collected?
What decisions will be made after the analysis?
Understanding the relationship between product specifications, testing, data analysis, and results will better prepare you for your job as an engineer. It will impress employers if you can speak to this during your interviews.
Market Specification Testing
The final market and product specification testing can take place once the manufacturing site completes the final product builds. The customer (the physician in my case) is involved in the market specification testing. This usually includes simulating how the product is used and assessing if it performs to their expectations.
As an introvert, interacting with the physicians was the most anxiety inducing part of the Test phase. I had worked with them for months, gathering their feedback on different designs. Therefore, it was really stressful to show them the final design.
Engineers are responsible for creating the surveys used during the evaluations. Trying to make the surveys as objective as possible is one of the challenges. Every customer may use the product slightly differently and have different expectations. This introduces some subjectivity.
For example, some people just prefer Apple products. Therefore, it doesn’t matter if you give them a better technology. They will always rank Apple higher because of their inherent bias. This is something to keep in mind during the customer evaluations.
Product Specification Testing
The product specification testing takes place at the company using the test methods the team developed. All of the operators completing the testing will be trained prior to executing the test method. This is to ensure the testing was executed properly. You do not want the way the operators are executing the test to impact your results.
The purpose of the test phase is to ensure the product will perform as expected by the user when it gets launched. Therefore, conditioning the product prior to testing is common. You want the product to be exposed to the same elements it will see before the customer gets it. This could include simulating the shipping and storage conditions.
Other testing can include shelf life testing, if there is an expiration date on the product. The purpose of shelf life testing is to demonstrate the stability of the product over an extended period of time. First, the product is exposed to the suggested storage temperature and humidity. Then, it is tested to show it still meets the product specifications.
As in every phase, documentation is very important. Test results are recorded carefully on raw data sheets. Then, engineers compile all of the results in a report to document all of the testing.
Defining Passing Criteria
What defines if a product passes a test? The product met the product specification so we can launch it, right? Not usually. Unless the specification has been deemed a low risk. Meaning, if the product did not meet this specification it would not cause harm to the user.
For example, if there is a handle on the product and the requirement is the handle must be blue. Maybe this was a user preference but had nothing to do with the safety of the product. Then, just passing is probably acceptable.
Surprising right? Don’t worry, this was the biggest learning curve for me as a new engineer as well. I want to let you know about it because no one let me know about this until I started my first job.
If meeting a specification is not enough, what do you as an engineer have to do to launch a product?
This is where quality comes in, to help define acceptance criteria for what is considered passing. The reason only meeting a specification is not enough is because it does not take into account the process variation. It also does not ensure the product will remain within specification over time.
When you are testing, you are only testing a subset of parts. You are not testing every part that is ever made. Especially if your test is destructive (requires the product to be broken apart), such as a tensile test. Therefore, you need to account for the process variation during testing. Then, you can use this to determine if your product passes or not.
A lot goes into establishing acceptance criteria. Such as the risk level associated with the product not meeting the specification and the confidence interval. The Process Capability Analysis for Six Sigma is a great resource if you are interested in learning more about capability analysis.
Test Phase Case Study
The hoverboard is an example of a potentially great product that missed some testing.
Do you remember the hoverboard? What a disaster that was. It was launched in 2015 and seemed like the hottest Christmas gift of the year. Until they started catching on fire or exploding!
In 2015, multiple airlines banned the hoverboards because of this risk. Also, eight brands were recalled in 2017 because the lithium-ion batteries in the devices caught fire. This ultimately led to the hoverboard’s demise.
The hoverboard was a brand new product. Everyone wanted to get their version launched as fast as possible before the competitors. This met selecting cheap components (i.e. the batteries). The companies also did not take the time to complete the due diligence and put the product through more rigorous testing. Everyone wanted to be first to market, so schedule triumphed over quality.
If the hoverboards had undergone more testing, maybe this failure could have been caught before the product reached the consumer. The testing phase can be frustrating when the product fails, especially when you are so close to launching. However, it is much better for the product to fail in the lab than when it is out in the real world.
Summary
To summarize, the engineer’s role in the Test phase includes:
- Executing all test methods and recording the data
- Performing statistical analysis on the data
- Documenting any failures that occur and performing an investigation to determine what happened
- Finding solutions to failures, potentially re-build and re-testing when necessary
As an engineer, there was a lot of on the job training to help me understand this phase. I took statistical courses offered by my company which helped.
I was lucky that I worked for a larger company with a lot of training opportunities. When you are looking for a company, this is one of the pros for choosing a larger company. They usually have more formalized trainings for their engineers because they have more resources.
Through the training, I learned more about capability and I met the statistician at the company. This was an excellent connection to have and I went to him often for help. Again, you are not going to know how to do everything you are asked to as an engineer. You have to be comfortable with making connections and asking questions. Resourcefulness is the only way I got through this phase in my first project.
Learn more about what engineers do during the final phase of the NPD process, the Launch Phase!