In my last post I wrote about system reliability versus system robustness. I briefly explained my definition of the two, and suggested some design process shifts to help improve both. Sometimes the required process change is small; sometimes it is substantial, almost like an entire design paradigm shift. But the reward, whether measured in improved product reliability or robustness, is usually worth the investment. A recent experience brought this fact into focus.
My church runs a food cannery just a couple of miles from my home. As food production plants go, it is a small to medium-sized facility. Permanent staff probably numbers less than ten people; food production relies heavily on church and community volunteers. Despite its modest size, however, this cannery packages food to help feed folks in need throughout the United States.
I usually work one or two half-day cannery shifts each year. If you have never visited a food production facility, but are fascinated by machines that whir and spin and rattle and shake and sputter to produce a product, you should put a production plant visit on your Bucket List. Such plants are an interconnected, entertaining mix of technologies. Depending on the product in production, you will find mechanical, electrical, electronic, optical, hydraulic, pneumatic, software, and even chemical elements in the plant system mix.
Improvements in process technology often automate manual steps in many production flows. Human-in-the-loop inconsistencies are frequently replaced with mechanized precision. The upgrade process, however, usually takes place over time. One automation upgrade paves the way for another, and another, and another. During my years working at the cannery, I have watched a gradual transition from a labor intensive process with volunteers working at stations throughout the plant, to a handful of folks placed at key locations to monitor (and occasionally help) the automated process, while most of the volunteers now work at a near-end-of-process station doing what machines currently cannot: visually inspect processed fruit for blemishes that might lower quality or reduce consumer appeal.
During canning season, fruit is moved from one side of the plant to the other along several conveyor belts. The final step in fruit preparation, just before canning, is inspection. The fruit inspection area is a long conveyor belt that runs one-half the length of the production floor. A typical crew of thirty workers, divided into two teams of fifteen, can stand comfortably spaced along either side of the belt. Fruit fresh from the peeler enters at one end of the inspection belt, and is conveyed along in front of the workers who inspect and trim the pieces before they are dumped into a water bath prior to being stuffed and sealed inside a can. On my most recent visit to the cannery, I was the anchor on my side of the inspection belt — the last person to check the fruit pieces before canning. Fortunately the inspection process is a team effort, so one position on the belt is not much busier than any other. As I worked away trying to keep up with my inspection duties, I noticed the flow of fruit slowed a bit, and finally just stopped altogether. “Great!” I thought. “A short break!” So I waited. Then I waited some more. After staring at an empty belt for a short while, I figured there had to be equipment trouble somewhere in the plant. Then a fellow volunteer pointed to the problem: a failed motor, which the permanent plant staff was trying to replace. The motor was maybe ten inches in diameter and perhaps sixteen inches from end-to-end, so not really very big as electric motors go. But that single, small motor’s failure brought the entire cannery to a standstill. Volunteers waited while plant mechanics replaced the failed motor with a spare. Even though replacing the motor only took twenty minutes or so, the breakdown reduced production volume, and could easily have cost money in terms of non-productive workers — luckily we were all volunteers.
So I was reminded that day that failures have a very real cost beyond just the price of the repair. The failure of an inexpensive part, in this case a $200 motor, can cost many times its value in wasted resources and reduced production. It is not uncommon for even smaller, cheaper part failures to cost much, much more in production losses. System components, in this case a simple motor, fail — an engineering fact of life, and a reminder that no matter how diligent our design process, systems are only as reliable as the weakest component. Choosing system components for their reliability and robustness metrics is just as important as making sure we account and compensate for system variability in our design process. Component selection, and designing reliable and robust systems, naturally go hand-in-hand.