Failures

Is it just me or does it seem like things have a tendency to break on the weekend when the parts and service repair companies are closed? This past weekend, as temperatures outside nudged up above 100°F, a strange electrical odor started filling the house. By evening, it was starting to get warm inside.  I did a quick check of the air conditioning system.  The condenser unit outside was running, but the inside blower was not working.  Oh great!  Why is the blower not working?  Turning “fan only” mode on didn’t fix it.  I did some quick investigation on our system and discovered we have a ECM blower motor.  ECM ,which stands for electronically commutated motor, means that it is a microprocessor controlled motor that is designed to optimize efficiency. It is very cool!  I have to admit, by this point I was distracted by the tech and not focused on the restoration of service.  With the warming room and questions from the rest of my family, I was reminded to go back into incident management mode and focus on addressing the outage.

I took apart the unit and began testing the control circuit and all the voltages.  The main 110V feed was going to the motor. The ECM 24V control lines that program the motor speed were also good.  The air handler control system seemed solid, so that meant it had to be the motor itself, and very likely the microcontroller that runs the motor. Of course, some quick online searches revealed that this wasn’t going to be an easy replacement with our current supply chain issues and chip shortages. I reached out to several service companies and received several, “We will get back with you on Monday” responses.

Now the funny part of the recent adventure is that we just had the entire system inspected and serviced earlier this year.  We didn’t want to end up in the heat of the summer and unable to find service or parts to fix any issues. Yet, here we are. This is a great, if not warm, reminder that failures will occur despite all our efforts.  

Things fail.  Reliability engineering teaches us that it is not “if”, but “when”.  Because we know things will fail, we design systems and processes to mitigate those failures, restoring service as fast as possible.  Done well, failures are addressed quickly, and sometimes even automatically, helping us continue to meet our service level objectives for the particular system.  With complex systems, it can be challenging to plan for all the failure modes that can occur.  How do you know what can fail?  One way to achieve this is by probing the weakness and safety boundaries of a system through Chaos Engineering.

We often laugh that we don’t need Chaos Engineering because the apps we run inject their own chaos!  But, Chaos Engineering is not about testing known broken parts of the system. If we already know something is broken (like a motor or an app), we should prioritize and fix it. Chaos Engineering is about discovering otherwise unknown weaknesses and limits of a working system. By introducing various degrees of planned failures (chaos) into our system, we learn new ways our service levels (SLOs) can be impacted. This gives us the opportunity to learn more about the system and improve it for faster recovery when it does fail.

Learning is key.  Failures, chaos engineering and experimentation are all teachers who can impart wisdom to all who seek their advice.  And as I just experienced this past weekend, life is full of lessons.  The only true failure is the failure to learn.  We are surrounded by a laboratory of learning that can instruct us and make us better.  Let’s make sure we maximize the opportunities that come our way and implement improvements that make the systems we run, better.  Oh, and depending on your heat tolerance level, you may want to stock a spare EMC blower motor on the shelf. 

Who Is Your Favorite?

“Please Ms. Smith, tell us, who is your favorite?”  A group of kids crowded the teacher’s desk, each with eager, pleading faces. 

“Is it Emma?” asked Mason, resting his arm on the desk with his hands under his chin. 

“I bet it is Isabella,” Mai said, “or Aiden.”  

Jamal chimed in, “No, it has to be Diego, she always calls on Diego.” 

Lily had a mischievous smile, “I think we all know who it isn’t.” The kids giggled and some glanced to the back of the room to see Alexander hunched over his desk with his arms folded. Alexander glared back at them with a snort.  His wild uncombed hair and scowling face seemed to ignite with anger.  The group crowded Ms. Smith even more and started to laugh.  

Alexander was a troublemaker.  He had a standing reservation in the timeout corner and would stir terror in anyone who even looked at him. “That’s enough of that,” Ms. Smith said sternly, “I told you, no favorites. Now, get back to your seats.”  

“But, please, Ms. Smith, tell us!!” the kids sang in a non-stop chorus.

“All right, all right, I tell you what I’m going to do. I’m not going to tell you now, but if you will all quietly go back to your seats, I will show you tomorrow.”  With a flurry of soft “oohs” and squeals, the children all excitedly scrambled back to their seats to finish their lesson for the day.

The next morning came and there was great anticipation in the room.  Ms. Smith arrived with a box.  She placed it carefully on a chair in the front of the room. “Okay class, I told you I would show you who my favorite is. I have put the answer in this box. I am going to allow all of you to come up and take a look, one by one. But you must keep it a secret. This person is extremely important to me. They are worth more than all the money in the world.  They are full of incredible potential and can do anything that they set their mind to do. I know that with their unique talents, they will even be able to change the world.”

“Mason, let’s start with you.  Please come take a look,” Ms. Smith said.  Mason nearly leaped out of his seat and then suddenly blushed, realizing the whole class was watching for his reaction.  Several kids giggled.  He couldn’t wait to see the answer.  When he looked in the box his mouth dropped opened and a huge grin exploded across his face. He laughed and smiled at Ms. Smith and nearly danced back to his seat.  Jamal was next and he too was stunned and then delighted when he looked into the box. A huge smile and arms waving wildly accompanied him back to his seat as if to celebrate being in on the secret.  Next, Lily paraded up to the front and like those before her, she grinned and giggle and put her hands over her mouth in surprise. Then came Isabella, Aiden, Mai, Emma, and Diego.  One by one, every child in the class came to the front to see inside the box, each returning to their seats delighted, excited and giddy as if they had all received a secret prize.  

A hush came over the class.  It was Alexander’s turn.  They all stared at him.  His bitter face and folded arms seemed to suggest he wouldn’t be participating.  But to everyone’s surprise he stood up.  Gasps were heard as he slowly marched up to the front, still with arms folded and an enormous frown.  Then, he looked down into the box.  

Stunned!  His frown split and his mouth opened up wide. Peering down into the black box he saw two wide dark brown eyes staring back at him. He gazed into those eyes, unbelieving for a minute before realizing what the teacher had done.  It was a mirror!  A chuckle formed in his throat and he looked up at the teacher who was beaming a full face smile back at him.  She nodded as if to say, Yes, it is you, Alexander!  The chuckle turned to a lump in his throat.  He turned to see the class, all eyes beaming back at him with smiles of joy and acceptance as if they had all joined a secret club.  For the first time in a long time, Alexander felt included.  In that moment, he belonged. He smiled back at them.  

“Clap, clap, clap,” a round of cheering and applause began to fill the little classroom.  Alexander couldn’t take it any longer.  Big tears welled up in his eyes and started streaming down his dirty face.  He tried to hide them at first and even batted them away but they kept coming.  He sheepishly grinned and looked up at Ms. Smith.

“Class, now you know,” Ms. Smith said smiling, dabbing away at the tears in her own eyes too, “You are all important.  You are all worth more than all the money in the world.  Each one of you is unique and full of possibility. Your future is limited only by what you saw in the box. That’s right, YOU! Give yourself permission to unlock your potential, and go change the world!”

Mothers

Early on Saturday morning I make my way down to our kitchen. I fire up the coffee pot and begin dealing pans out across the stove.  I pull out the breakfast ingredients and start dicing, chopping and cooking. The clanging of the cookware and breakfast making can be heard throughout the house.  Soon the odor of the simmering morning meal will fill our home.  

I hear some rustling.  The sleepy home begins to wake.  For the past several years, the first to the table would be my mother-in-law.  She came to watch the show, but more importantly, she came to get some goodies.  I always prepared some peanut butter toast and fruit for her, mainly to help curb the “Is it ready yet?” endless loop.  However that never deterred her back-seat-cooking instructions that I would receive.  She loved food and loved cooking.  When she was no longer able to manage the task herself, she would live vicariously through others, including me.  As I cooked, she would dream into the past and tell me stories.  She would relay wisdom from her many years and reflect on current events. We discussed the changing seasons, family, vocations, life and of course, the menu.  She is gone now, but I will forever laugh about and cherish those moments. 

My mom was a teacher.  She taught 3rd grade most of her career, which meant that like it or not, I got really good at addition, subtraction, multiplication, division, reading comprehension, writing, social studies, the scientific method, maps and graphs.  My mom believed that the third grade curriculum built a wide foundation for the rest of your education.  Most kids left their teacher at school, but not me.  I got to be public schooled and home schooled at the same time. To be fair, I needed it.  I would much rather be goofing off, building tree houses, riding bikes and catching tadpoles in the local stream than studying some book, diagraming sentences or doing my math assignment.  In hindsight, it was an incredible blessing to have a teacher for a mom.  But she wasn’t just a teacher.  She was an encourager.  “You can be anything you want to be, I’m incredibly proud of you.”  Words like that are fuel for life.  She gave me confidence to try and courage to fail.  She was my champion and my defender as well as my teacher and my mom.

Good mothers shape the future.  They make an incredible impact on our lives.  It may not even be our biological parent that has the biggest impact. Many of us were adopted by other moms.  I recall so many who spoke into my life with some encouragement, kindness and yes, even some discipline, that all helped shape who I am today.  We are all the sum of the many investments made in us by others.  I’m sure many of you have similar stories.  This coming Sunday, as we arrive at another Mother’s Day in the US, take this opportunity to reflect on those investments our moms have made in us and be thankful.

To all you moms out there, you make a difference!  Thank you, mom!