Let’s talk about Bruno! Wait, what? We don’t talk about Bruno? For those of you who haven’t seen Walt Disney Animation Studios’ Encanto , you are missing out. I won’t post any spoilers, but let’s say it is definitely one of those multi-play movies, especially at the Cox home, and likely for any of you with kids at home. I hear the soundtrack playing again right now.
Over the holidays, I had the pleasure of going to the theater to see Encanto with my entire family, including my son who moved away. That’s something we haven’t been able to do together for several years now. In typical Disney fashion, the story unfolds a powerful moral narrative, celebrates family decorated with spectacular magical moments, music, characters and beautiful vibrant colors. We talked about it as a family for hours afterwards. Great memories!
It’s a joy and a privilege to work at Disney. We make moments that last a lifetime. But I think we can get so caught up in the day to day, log4j or sprint of the moment that we forget the incredible impact that our work, our company and our fellow cast members have on our human family. We make magic. The artistry, the technical innovation, the story telling, and the signature quality of our products all power our ability to deliver enchanted experiences. Those interactions create moments and memories that last forever. They help craft and elevate personal narratives and family stories that pass down from one generation to the next. Like many of my fellow cast members, I often hear family and friends excitedly convey their personal or families’ first experiences with Disney, perhaps a Mickey Mouse short, a character, a movie or a trip to Disneyland. A smile crosses their face and they close their eyes as they reminisce and cherish those wonderful moments.
Can I suggest something? This isn’t unique to working at Disney. All of our actions, our work and our jobs impact others. Take a few minutes today and look up from your laptop screen, walk outside and gaze out into our world. Look at the horizon of the fabulous Earthship we are riding. Across our world is a magnificent variety of amazing people, families and connections. Imagine the infinite experiences happening to our human family across the globe right now. The work we do matters to them. We can make a positive difference.
Now, let’s go make some of that magic…. Oh, and let’s talk about Bruno!
“On every world, wherever people are, in the deepest part of the winter, at the exact mid-point, everybody stops and turns and hugs. As if to say, ‘Well done. Well done, everyone! We’re halfway out of the dark.’ Back on Earth we call this Christmas.” – Doctor Who
I confess, I’m addicted to data. I love building graphs and colorized charts to visualize that data. When we installed our Tesla Energy solar panels back in September, I was delighted to have our house powered by that ancient and reliable thermonuclear fusion reactor that shows up each day in our sky. But, I wanted to see the data! I soon discovered a wealth of information available from our new Tesla Energy Powerwall system. I began mining it and even created a simple python library, pyPowerwall, to make it easy to manipulate. Using that and Telegraf, InfluxDB, and Grafana I had an observability platform and dashboard to feed my data and visualization appetite. Naturally, the graphs were in full color.
It’s getting dark. You don’t need a graph to tell you that. Our days get shorter in the winter. We all observe the blanket of darkness that gradually pulls over us, stretching deeper into our mornings and evenings. But what you may not know is how dramatic that change can be in terms of energy. Our solar energy system went online at the beginning of October, with 35kWh of energy captured from the sun every day. It has been on a steady ramp down since that first day. As we tick inescapably towards our winter solstice, in just two months the energy we get from our heavenly benefactor has dropped in half to 17kWh per day. The day-to-day graph shows the flattening of the curve as it bends down ominously towards the winter solar abyss of December 21st. It’s victory over us is felt in the cold wind, the fading colors and the melancholy shadows that scrape against our souls. Darkness is here.
Light a candle. Our human family has witnessed this solar dance since our beginning. We measure the sky and plot the stars to know where we are. We embrace the rhythm of the year by decorating it with celebrations and traditions. In the darkness of the winter, we light our lights. We illuminate our winter journey with candles. We adorn our homes with fragrant greens, twinkling lights, cheerful ornaments and glowing fires. All the while, we know, the light is coming again. The glory of the sun will return! But for now, we celebrate.
It’s almost here. The dead of winter has arrived. Cuddle up with a warm cup of coffee or tea, your loved ones and a glowing fire. The light will return. We are halfway out of the dark! Celebrate it.
“I am confident the need for great storytelling will endure for generations to come, enhanced by new technologies that will bring these tales to life and deliver them to people around the globe in even more extraordinary ways… we will always gravitate to stories that bind us together—tales of adventure, love, friendship, and heroism, tales that enable us to escape, that comfort and inspire us, that give us hope and reason to be optimistic, tales that inform us…. and, of course, touch our hearts.” – Bob Iger, Farewell
I woke up this morning and glanced out the window to see the sun peaking over the mountain tops. The beams of light struck the beautiful and brilliant red and golden leaves of the trees behind our house. The backlit spectacular was incredible! As I soaked it in, I was struck by the temporal nature of this artistry. The occasional leaf would detach and glide to the ground. Soon, this majestic presentation would be gone and all that would be left will be the branches. It occurred to me, how tragic it would have been if I had missed it. It’s a great reminder that things change. New scenes are appearing all the time. The lesson is, enjoy the moment, those golden nuggets of time that grace your path. Drink them in. Be grateful for the show that was made, just for you.
I noticed another changing of seasons today. Disney’s former CEO and Chairman, Bob Iger’s farewell email landed in my inbox this morning. As with the beautiful sunrise, it definitely has me nostalgic, a bit misty eyed and grateful.
A few months after I started at Disney (now sixteen and a half years ago!) a new CEO was announced, Bob Iger. Disney was a strong brand, but creativity and storytelling had become stagnant and mundane. Bob came in with a simple yet powerful plan. He identified three strategic pillars that would propel Disney forward:
Creativity – We would focus on creating compelling content and telling stories that delight our guests in new and powerful ways.
Technology – Second only to creativity would be the adoption and innovation of new technologies to help us create and deliver those storytelling experiences. It would help us amplify the magic, better connect with our guests and allow our guests to better connect with us.
Globalization – Disney would expand into new markets around the world. As part of that effort, our content, products and experiences would strive to reflect our diverse human family around the globe, both on screen as well as behind the scenes.
The result of Bob’s strategy was a return to epic storytelling, investments and innovation in technology and the expansion of our impact around the globe. It resulted in us welcoming PIXAR (2006), Marvel (2009), Lucasfilm (2012) and most recently 21st Century Fox (2019) to the Disney family. Adding those creative engines, brilliant teams and storytellers, along with new innovative technologies, amplified our ability to entertain, inform and inspire people around the globe.
Like many of my fellow cast members, I feel like I have had the privilege of sitting in the front row watching the incredible transformation Bob Iger unleashed for Disney. And like many others, I am incredibly grateful to have been able to participate and be part of that “ride of a lifetime” story with Bob at the wheel.
I am sad to see Bob Iger leave, but I’m incredibly optimistic about the future. The work Bob started is growing and shaping Disney into a modern force for good. I’m convinced it will continue. Sure, we will make missteps, try and fail at times, but our mission to deliver magic, entertainment and inspiration to the world is still key. It is an honor to be part of an organization that is about elevating the human experience across the planet, especially when it includes unlocking the potential of technology to do so.
Have a great week! And don’t forget… enjoy the ride.
“As human beings, our job in life is to help people realize how rare and valuable each one of us really is, that each of us has something that no one else has – or ever will have – something inside that is unique to all time. It’s our job to encourage each other to discover that uniqueness and to provide ways of developing its expression.” – Fred Rogers
Did any of you brave it out on Friday to see the bustling crowds at stores or the mall? We did. I’m a glutton for punishment I suppose, but I do enjoy the excitement, festive mood and the occasional good deals. To be fair, there was much less this year. Especially the crowds and the good deals, but the festive mood was still there. Regardless, it was a fun date with my sweetie to parade around the town and spot a good deal or two.
We started putting up Christmas decorations even before Thanksgiving, but this past weekend was the final sprint. Trees, lights, candles and greenery now adorn every room in our house and even flow outside. Present boxes, sweets, music and greeting cards begin their annual appearance. And, as some of you know, it was the first candle lighting of Hanukkah last night. As the days grow darker and a bit gloomy, the warm glow of the candles, fireplace and the Christmas lights are healing salves to the soul.
I’m clumsy. I love helping set up the trees, string the lights and set out the decorations. But in the process, I manage to bump things and there is the evitable breakage. I was setting up a small tree next to our dining room and added a small string of LED lights. It was a low-cost variety and had that irritating 60Hz strobing. I decided to solder in a rectifier, which did the trick, but in the process, I knocked over one of those big LED-lit candles we have. It fell to our unforgiving tile floor and the wax broke into bits.
“Oh great,” I said to myself and then heard the expected, “What did you break now?” My poor motor skills are well known to my family so they came into the dining room to see the show. They managed to piece the candle back together. To our surprise, the candle still worked. The glow of the light in the candle warmed up the entire cylinder and to our surprise, created a marvelous new display, illuminating the cracks and casting new shades on the surface. It was beautiful! It looked like a landscape wrapped around the candle, with dramatic shadows to create depth. The result was a natural and unique work of art that now sits on the shelf. The broken candle created something new, giving inspiration and perspective that didn’t exist before.
We are all broken. We all have cracks. We experience hurts, failures, disappointments, physical limitations, mental struggles and other self or externally judged imperfections. As my youngest often reminds me, especially when my obsessive compulsiveness kicks in, “Nobody is perfect.” She is right! The reality is that part of our immeasurable value and enduring purpose is the incredible uniqueness of our seemingly broken bits. Where we are fractured, the light shines through us best. Glow! We are creating something new by embracing who we are, striving yet shining. Our uniqueness gives inspiration and perspective that didn’t exist before. Don’t hold back. Be the best “you” you can be and shine!
Did you notice it? We often wish for more time in the day to get things done. Yesterday we got it! Sunday was 25 hours long. I don’t know about you, but the only part I noticed was the extra hour of sleep. Of course, to pay off this luxury, we will suffer a 23-hour Sunday in the Spring. That is, of course, unless we suddenly manage to retire Daylight Savings Time.
The stream of time keeps going on. It reminds me of my childhood when we would go river rafting. We would find a mountain stream in the Ozarks and launch our rafts at the headwaters. It would be early in the morning as the sun was just coming out. There would be dew on the leaves and the air would be cool and crisp. As we would float downstream we encountered areas that moved quickly. We would hit the churning water of the rapids with loud and heart pounding excitement. Frantic and vigorously we would paddle to steer clear of the approaching boulders, tree limbs and perilous drops.
There were times when the stream would slow to a crawl and the water would become like glass. We would hear the birds and wind gently whistling through the trees. It was like time and the stream just nearly stopped. We would still paddle to stay on course, navigate the river bends and slowly make progress. Occasionally we would just simply glide along, letting the energy of the stream carry us forward. Or we might steer the raft to the shore to rest, warm up in the sun, eat a snack or just relax. Those are some of my favorite memories. They were beautiful opportunities to soak in the views and drink in the sounds.
Life throws a lot of variety at us. There are twists and turns. There are times of rest. There are moments of chaos. Sometimes we face the rapids head on, quickly navigating and strongly reacting to the dramatically unfolding challenges before us. At other times we get to glide and enjoy the serene and beautiful moments. In all those moments, we have a choice. We can actively participate in the journey, paddling our best to navigate the adventure. And at other times, we can pull up our oars and enjoy the scenic serenity of living in the moment, frozen in time. Make sure you do both.
Boo! Trick-or-treat was a blast this year. I love seeing the neighborhood kids, and sometimes their parents, all dressed up for the occasion. While some safety measures were still in place, it was great to see some return to normalcy even if it was a street full of super heroes, princesses, goblins and ghosts.
I don’t know about you, but I start buying candy way back in September, most of which is trick-or-treated even before the first costumed characters arrive at our door. In fact, so much so that I usually have to make several subsequent trips to the store to get more. I figure it is all good UAT (user acceptance testing) so we deliver a quality experience on Halloween night. Based on the pounds of candy invested, we nailed it! Of course, the weight scales have a different opinion about the outcome.
Bring on the fall decorations! Christmas and holiday decorations have been on display in most retail stores for the past several weeks. But before we light the candles and trim the trees, we must past through the season of thanksgiving. I love this time of year! The next four weeks are a great time for reflection and gratefulness. Yes, it is a wonderful time of family, friends and food, but it is also a time of remembering. This past year has been a challenging time for all of us. Major life events and minor inconveniences added up to a sum of substantial challenges for many of us. Yet through the journey there has been incredible moments of celebration, accomplishments and learning.
Life is like that. It is a composition of major and minor chords. Sometimes you are soaring on the high notes, ringing warm with the delight and optimism that the major chords bring. Then suddenly, tragedy or setbacks strike and the strings ring a cold, somber and sad march. Yet in it all, there is a beauty. The shadows of the gloomy mood are overtaken by the next soaring melody. We overcome. We sing new life into the darkness and transition our way to the next mountaintop anthem. Looking back, we see the journey and the score that was laid down behind us. Remember it. It is part of our experience, a bit of who we are and the personal soundtrack of our life’s story. Change it. As the conductor, we have the opportunity to set the next measure. Write it. Enjoy every section that makes up our life. We journey this way but once. Don’t miss the concert!
“Yesterday’s the past, tomorrow’s the future, but today is a gift. That’s why it’s called the present.” – Bil Keane
Halloween is this Sunday. Christmas is just two months from today. Can you believe it? As I stare out my dining room window, I notice the trees are beginning to paint themselves with the beautiful colors of fall. The brilliant reds, gilded yellows and warming browns highlight against the faded greens. Nature’s parade of time strikes a chime, reminding us that the year’s last act is on the stage.
I love this time of year. The cool crisp autumn breeze and ever-increasing autumn nights seem to pull a thick dark relaxing blanket over the mind. Melancholy and hope dance. There is rising optimism and wistful reflection, thinking back and looking forward. We remember and we plan.
Reflecting back on the year, I know it has been challenging for all of us. There has been a lot of change. We have seen wonderful successes, painful losses, personal victories and sometimes even discouraging defeats. So much has changed since the start of the pandemic. Yet, so much is still the same. This time of the year we get to relive the delightful nostalgia of our past while reaching forward with hope to grab the future.
Can I suggest something? Stop for a few moments today and disconnect. Take a walk, maybe even in the rain, or stare out the window. Reflect on this past year, dream about the coming holidays, but don’t forget to soak in the present. The act is on stage right now. It’s beautiful. Savor it.
Thanks for all the great memories and intoxicating hope for the future.
“The more deeply one drinks of this spring, the more securely one becomes in one’s own abundant happiness, the more bountiful it becomes to relish the joy of other people.”
Have you ever entered a room where you didn’t know anyone but where everyone was excited, happy and enthusiastic and suddenly found yourself feeling the same way? Or perhaps you happened to witness a heated debate or argument and suddenly felt angry, frustrated and anxious? Maybe you are watching a movie, a show or a theatrical presentation and suddenly feel connected to the story in a visceral way, with fear, anger, disgust, sadness and joy. If so, you can thank your mirror neurons for those fun experiences. Researchers in cognitive neuroscience have been trying to figure out why we have mirror neurons and their specific benefits to us. It appears that at least for one thing, the mirror neurons are the neural basis of the human capacity for emotions such as empathy. I don’t know about you, but I’m immensely grateful to have these, and especially grateful that you have them too!
Mudita. In Spanish that word means “mute”, but in Pāli and Sanskrit it has a completely different meaning. In fact, you will find an equivalent word in many languages around the world sometimes even represented by a single character. That is, all but English. We don’t have an equivalent word. For us, it would translate to something like, “sympathetic joy”. It is the overwhelming feeling of pleasure that comes only from a vicarious delight in the well-being of others. In other words, it is feeling immense happiness for the happiness of others.
Some adversaries of mudita are jealousy, envy and derision. Mudita can heal these poisons and untap a reservoir of happiness. Many of our world religions teach the virtues of drinking in this joy. They encourage celebrating the delight and achievement of others, even when we are facing tragedy ourselves. “Rejoice with those who rejoice.” The real challenge is to cultivate mudita even in a world often full of misery.
We are wired for mudita, with our mirror neurons helping us, but do we really experience it as we should? Or do we find that we live in more of a manic state, seeking and reaping things for ourselves while missing the real treasure of joy that comes from celebrating with others? I’m definitely guilty of that. We should be in the business of the happiness of others that we may be happy too.
Mudita is flowering all around us. Let’s not miss it. Grasp the moments as they come. Absorb the incredible joy that can be ours when we step out of ourselves and rejoice at the blessings of others. I suggest we all take time this week, stop and smell the fragrance of those mudita blooms.
“I was a fixer, a builder – an inventor – ever since I can remember.” – Tom Scholz
I love this time of year. The mornings are nice and cool (11C) but can still warm up in the afternoon (32C). Our neighbors have decorated their homes with Fall and Halloween decorations. Target has a mile-long line of candy aisles and we do our best to replicate that in our home.
I confess, I have an addiction. No, not the candy, I can meter that fairly well. My real addiction is my obsession to fix things. I can’t stand it when something is broken or a system is down. It drives me crazy. I am compelled to get it working. This past weekend, my obsession was our new Tesla Solar system. It all began with our decision to get solar a year ago…
After research and talking with several solar companies, we decided on a small 8.5kW Tesla Solar plus Powerwall system. The main reason for our decision was the low cost, handsomely framed panels and the whole house backup capability. There were several good offers out there, but we were delighted when Tesla came in with the lowest price. I’m very cheap, I admit, I love a good deal. What we didn’t realize is that the way Tesla beat the market was to remove some of their overhead, mainly in the area of customer service. That translated into a lot of waiting and frustration.
Our system was finally installed last month and during commissioning, the first signs of trouble started showing up. The installers downloaded the latest software updates but were unable to get the Solar Panels to work correctly. The solar assembly was only producing 160W in full sun. They tried for hours, upgrading, rebooting and calling. They eventually gave up after showing me that the Powerwall could power our house if they cut the mains. They explained that Tesla would send out a software patch to fix the Solar panels, most likely.
I contacted our Tesla Advisor to report the problem and to see if they had an update. After several days of emailing, texting and calling, I received a note from the advisor that he would contact the electricians about the problem. I spent another week requesting updates but my Advisor had gone radio silent. It turns out that this is a common experience with Tesla. Assuming best intentions, I can only imagine that the Tesla teams are understaffed and overwhelmed. Regardless, it all results in a very frustrating experience for the consumer. I escalated and was finally told that they could schedule a service appointment on Dec. 14th. Seriously?!
We had all this gear in place. We had these beautiful solar panels, high tech Powerwalls and all the gear to drive it. But none of it was working. Tesla’s mistake was bringing all of this into my home. I can’t help myself. My addiction set in. I have to fix it. So, I did. I began poking around, measuring voltages, sniffing the system’s communication links, and researching all of the modules that make up the system.
I finally found the issue. I discovered a WiFi based metering system Tesla had used was defective. I reprogrammed and recommissioned the system. It came online and for the first time ever, we had full solar energy powering our house and charging our Powerwalls. Our home had gone green and we were 100% powered by the sun! I love making things work.
Reflecting back on this experience it occurred to me that this is a lot of what we do. In technology, we are often the Sherpas that help our business partners reach their destination. Sometimes it requires research, investigation, designing, hacking, and even reprogramming to arrive at a reliable and operative solution. All along the way, our partners are depending on us to be the experts to creatively solve problems, fix what is broken and deliver a working solution that helps them, our businesses and organizations deliver the best outcomes as we can.
Thank you all for being the fixers. You substantially make a difference to the world!
If you want the gory details on my investigation and fix you can read them here.
After research and talking with several solar companies, we decided on an 8.5kW Tesla Solar plus Powerwall+ system with their new high efficiency 425W panels. The main reason for our decision was the low cost, handsomely framed panels and the whole house backup capability. Other solar companies had good backups systems but we did not find any who would provide whole house backup. And, more importantly, we were delightfully surprised to see that Tesla came in with the best price. Having said that, we would soon discover that they seemed to have significantly reduced overhead by mostly eliminating customer service.
The Install
25 September 2021
Tesla Crew
After ordering, reviewing designs, applying for HOA and City approval, we were finally ready to get the system installed. Two different crews arrived over a span of a week to get the system installed. The panels went on first. The panel install crew was professional and friendly. However, when they left I noticed that the handsome skirts (frames) we were so excited about were only installed on the front of the house (street facing roof). It looked great but I had expected to get them on the back as well. I reported it and in just a few days a technician came out and added the skirts to the back roof panels. He explained that they usually only install the skirts on the front. So, please note, if you want skirts on all your panels, make sure you let them know in advance. Also, the skirts are only put on the left, right and bottom. There are no skirts on the top to allow heat to escape from the panels during the hot summer.
Panel Mount – Torx T30
While installing the skirts for the back, I noticed one of the panel edges was sticking up about 1/2″ higher than the rest. The technician tried to fix it but he didn’t have all the tools. He only had what was needed to install the skirts. He asked me if I had a Torx T30 driver. I didn’t but he explained how I could adjust the panels myself. I picked up a T30 at our Newhall True Value store. I climbed up on the roof and found the adjustment area. I used a vice grip on the screwdriver to get enough leverage (mostly because I’m pretty weak especially when I’m up on the roof). I was able to lower the panel 1/2″ so it was flush. It looked beautiful.
Front
Back
A week later, another crew showed up to wire it in and commission the system. This meant a day without power, but we were prepared for that. I tried not to be a nuisance, but couldn’t help but watch and ask questions. I made sure all of the crew had plenty of bottled water, Gatorade and snacks, including ice cream candy bars since it was so hot. They installed the Powerwalls in our garage and wired in the breaker panels and Tesla Gateway by the utility meter. After everything was installed, they powered it up and began the commissioning process.
The Problem
During commissioning, the first signs of trouble started showing up. The installers downloaded the latest software updates but were unable to get the Solar Panels to work correctly. The solar assembly was only producing 160W in full sun which doesn’t even show up in the app. They tried for hours, upgrading, rebooting, calling. They eventually gave up after showing me that the Powerwall could power our house if they cut the mains (based on 22% charge from the factory). They explained that Tesla would send out a software patch to fix the Solar panels, most likely.
I contacted our Tesla Advisor to report the problem and to see if they had an update. After several days of emailing, texting and calling, I received a note from the advisor that our inspection would be scheduled in 3-4 weeks and he would contact the electrician about the problem. I spent another week requesting updates but my Advisor had gone radio silent. It turns out that this is a common experience with Tesla. Assuming best intentions, I can only imagine that the advisors are understaffed and overwhelmed. Regardless, it all results in a very frustrating experience for the customer.
I did manage to finally get an update and a promise to further investigate the issue. While I waited, I decided to do some more research on the system to see if I could find the problem myself…
HIGH VOLTAGE WARNING: I need to stop here and remind everyone that these systems contain extremely high voltages and are dangerous. Hopefully it goes without saying, but please be careful if you poke around inside these electrical boxes. High voltage can be fatal.
The Investigation
You can connect to the Tesla Gateway by scanning the QR code inside the box. It will have your phone connect to the Gateway’s access point. You will need to stay close to the gateway if you connect this way (and browse to https://192.168.91.1). However, keep in mind that it is also connected to your home network and if you know how to find the IP address, you can point your browser to that IP and login as the Installer to see more details about the system. Your browser will likely require that you ignore the security certificate warning (more on this in the observations section below) and you will need to toggle the power switch to one of your Powerwalls but it will let you in. That is essentially their 2nd factor system to ensure you are authorized. Here is what my system looked like after installation on the main screen and on the “System” screen :
The System screen also shows details about the solar generation, Powerwalls and power usage:
Below the above list was a section for “Remote Meter” that would occasionally appear. This was particularly interesting:
Remote Meter (Vxxxxxxxxxxxxxxx) CT 1 (Solar): —W
That seemed odd. Also when I clicked on the solar panel icon on the main screen, it would say “Stale Meter Data” – that had me wondering if the solar meter was the real issue.
The Fix?
I first disabled the Solar Assembly by clicking “Disable” on the System screen.
I opened up the Inverter, the box above the first Powerall. There is a small latch on the bottom that will unlock and let the panel swing up. I found a wooden dowel to prop it open so it would bang on my head the whole time I was investigating.
I noticed that there was a box on the right that had a “n” LED flashing. The code on the box was the same code that was listed as the “Remote meter” in the system’s display (the one showing no power). Some more research and I discovered that this module is a Neurio W2-Tesla WiFi based current reader that sends the solar power data to the Gateway.
Neurio was recently purchased by Generac but you can still find manuals and some models for sale online. This particular model, W2, has been customized for Tesla. It is designed to connect to the access point of the Tesla Gateway and send the solar power data.
CT-1 Amp Probe Wire
The Neurio has a wire plugged in to the top in the CT-1 (current transformer) port. I traced it over to the solar inverter where a clamp is wrapped around the solar inverter output AC line to measure the amperage. I re-seated that connector.
Antenna
I then noticed that there was an antenna jammed below it that was tucked to the left, under the massive metal inverter shield. I turned the antenna to the right, in the open unshield space.
As soon as I did these two things, the LED “n” on the box began to change and a tune started coming out of the box. It sounded like “I am connected now”. The flashing “n” became a solid blue light.
Eureka!
I went back to the System screen and re-activated the Solar Assembly by clicking “Enabled”. This takes several minutes and you will see the system go through and activate the solar arrays, test relays and impedance before the assembly comes online.
As soon as the Solar Assembly came online, I started seeing kW of power show up on the Systems screen. 5800W of power was coming in, fully powering the house and charging the Powerwalls!
Not so fast…
Sadly, just two hours later my elation was destroyed. The solar energy dropped back to zero.
I checked the inverter. Sure enough, the Neurio was flashing again. I attempt the above process again, several times, but no joy this time. It would chime and go green, but then started flashing again. Based on my research, the Neurio connects to the Tesla Gateway WiFi only. The beeps and flashes indicate that it is unable to connect to the Gateway WiFi.
One thought I had was to reach out to Neurio (which is now owned by Generac) to see if they could provide the API, pinout or schematics for the W2 device so I could troubleshoot at the firmware and component level. When I contacted them, Generac replied that the serial number for my device contains proprietary firmware by Tesla that they cannot support. They recommend that I contact Tesla at: 888-518-3752. Oh well, it was worth a shot.
The Workaround
I love a challenge. In fact, when something isn’t working, it is almost an addiction to me. I have to figure it out and fix it! So, I had two thoughts at this point. First, I wanted to see what the Neurio was actually doing. I thought about setting up an ESP8266 to be an WiFi access point to intercept the Neurio’s communication attempts with the Gateway. But before I do that, it occured to me, I wonder what would happen if the system didn’t have a solar meter at all. In my investigation, I discovered that the solar power meter feature is often an add-on or post-install enhancement. Maybe this was more of an add-on feature than a requirement?
At the minimum, I wanted to see if there are alternatives to the Neurio in the Tesla configuration. Unfortunately, there isn’t an easy way to edit this data. I discovered that settings could only be set during the initial setup time. That would require running the setup wizard again. I decided to be bold and fire up the installation Wizard. At the bottom of the system portal is the “Run Wizard” link. Of course, I clicked it.
WARNING: I’m fairly confident that you can completely break your Tesla Solar setup using the Wizard, maybe even disable power to your house permanently. It is intended for installers. I’m taking the risk, but you should consider this first and be cautious about proceeding. I’m also fairly confident I’m going to void something in the process, but if you put something in my house, fair game, I must hack.
The wizard is straightforward. It requires you to Stop the system, but the settings are mostly intuitive. When I arrived at the Meter screen, it had 3 different meters displayed. I apologize, I did not take screenshots but will update this blog if I capture them in the future but the screens are very basic.
Two of the sensors were for the built-in CT’s used to measure the power in the Tesla Backup Gateway (you can see them on the main bus if you open the Gateway panel – which I did).
Tesla Backup Gateway DANGER: High voltage – Seriously!
These tested extremely fast (subsecond) were working correctly and tested “good”. The third meter, a WiFi meter, was the Solar Meter (Neurio) and it’s status was Error, unable to connect. I clicked the connect button which reported it would take 3 minutes to configure the WiFi sensor. No shock, it didn’t work. I tried it 3 more times. The “Advanced” drop down allows you to add MAC address and IP, but this didn’t help. There was a “Delete” button. I thought it might be worth a try to delete and re-add. At the bottom were options to add “WiFi” or “Wired” CTs. I tried to add the Neurio (WiFi) again, multiple times, rebooting the Neurio occasionally to see if that would help. Nothing.
Here is where something interesting happened. The Wizard would NOT let me advance because the WiFi sensor was not healthy (connected). Hum… Well, I figured I would just have to delete it to see what other screens I could find in the Wizard. I deleted the Neurio. I advanced to the next screen and was presented with a “Warning – you do not have a solar sensor selected.” Naturally, I ignored that and continued.
Commissioned! I completed the Wizard setup and the system came back online. Surprisingly, the system screen looked basically the same but the dynamic flow diagram was actually working. There were no sensor errors or warnings. Power was flowing from the Grid to the House. It was the middle of the night so I signed off and went to bed.
I know what you are thinking. This is dangerous, right? I mean, we seemed to have removed solar power observability from the platform. Will the Gateway and Inverter sill know what to do? Well, it turns out… it does!
The Power of the Sun
Next morning, I woke to discover solar generation was charging the powerwalls and our house was completely powered by the sun! I still want Telsa to fix the Neurio or, better yet, provide some hardwire CT to monitor Solar power generation. I’m assuming that the display below means that the Gateway is computing the the solar generation based on other CTs. In any case, my workaround is in place and we now have a working system again.
As I’m looking at my phone, I realize… I’m holding the power of the sun in the palm of my hand. Yes, that is a geeky Doc Ock reference. We are now powering our home with an ancient but reliable and self-regulating, thermonuclear fusion reactor… our sun.
The Return of Tesla
1 November 2021
I gave up on waiting on Tesla to respond to me about the Neurio. I figured it didn’t matter since I had a working system. A month after the install and I still didn’t have an inspection date. Then it happened. I received a text message and email from Tesla that my inspection was scheduled. There was NO DATE or TIME given. Instead of asking, I figured it didn’t matter. We would see what would happen.
The day of inspection had arrived. A surprise knock on the door and there was Ishmael from Tesla. He explained he was there to meet with the City inspector for the final inspection. I showed him the gear, the Powerwalls, the gateway and the breaker panels. He looked at me and asked, “Did the install crew not put on the warning labels?” Nope.
This was something I had noticed after the installers left. In the Tesla plans are specific instructions on where to place the red warning labels on all of the gear. It includes a helpful diagram for anyone wanting to know how to kill all power in case of emergency. I had raised this issue with my project advisor a few times, but as usual, told me he would look into it and of course, nothing happened. I explained this to Ishmael who rolled his eyes and expressed apologies and said he would need to call to get the labels or it would not pass inspection. He would wait for the delivery and get them installed and ready for the City.
Shortly after meeting Ishmael, another Tesla vehicle pulled up. I figured it was the inspection stickers, but instead, it was Rocío, a Quality Assurance technician. She told me that her job was to make sure everything was installed correctly and running. I almost hugged her! I expressed my delight and appreciation that she would check on us. I explained everything that happened including how the installers said it must have been a Tesla software bug and gave up after trying for hours to get it work. I also told her about the Neurio hack I had done to get it working. She was shocked, sympathetic and determined to fix the issue.
Naturally, Rocío attempted to reset the Neurio and discovered the same thing that I did, with the exception that she was able to get the Neurio to work if she held the connector, pressing on it in a certain way. “There is clearly something wrong with the hardware and it needs to be replaced,” she concluded. I hate to be cynical, but I was definitely thinking this new chapter in my Tesla adventure would turn into an multi-week RMA, repair order and a return visit that may get scheduled sometime next year, if I’m lucky.
To my delight, Rocío looked straight at me and said, “And we’re going to get this fixed today!” She was right! She made a phone call and 30 minutes later another Tesla van showed up with the replacement Neurio!
Rocío got it working. Less than 30 minutes later she had the entire system back online and working correctly. “That’s amazing!” I told her. She clearly saw my astonishment and said, “I used to be an installer, I know what’s needed.” Well, that was completely accurate. She didn’t stop there. She examined all the gear and climbed up on the roof to ensure all the panels were in good order.
Shortly after the good news, the warning labels arrived and were attached to the new gear, ready for the official inspection. I started passing out my sincere appreciation, candy bars, water and Gatorade to these brilliant Tesla soldiers that had come to save the day. After bidding farewell to our new friends, Rocío drove off on her shiney white stallion… uh, I mean Tesla van.
About 30 minutes later, the City Inspector arrived and after a quick survey of the installed gear with Ishmael, signed his approval. Now we are on to the Permission to Operate (PTO) by Southern California Edison.
Power Gremlins
3 November 2021
I should definitely learn to be more guarded in my optimism about this Tesla adventure. After two days of having the new Neurio re-installed, I started noticing something odd. After solar production when I would expect the Powerwalls to kick in and power the house, I would see grid power start to show up and the Powerwalls drop to zero. It would only last for a few minutes then return to normal operations. Looked at the Powerwall Monitor I set up and can even see the grid power spiking during the day when solar production was more than enough to power the house.
Powerwall Monitor: Notice the purple (grid power) spikes throughout the day. These just started showing up after the new Neurio was installed.
The grid power spikes did not exist before the new Neurio. I went out to look at the inverter. The Neurio’s purple light was mostly solid but would “flicker” blue. It was random, like a candle flame not like the error condition of the previous Neurio. It was happening constantly as I watched. When the flickering would get bad, I would see the powerwall drop to zero and grid power surge. There seemed to be a correlation. At any rate, I wasn’t going to let the flickering continue.
HIGH VOLTAGE WARNING: I need to stop here again and remind everyone that these systems contain extremely high voltages and are dangerous. Hopefully it goes without saying, but please be careful if you poke around inside these electrical boxes. High voltage can be fatal.
I powered off the Neurio by unplugging the power next to the antenna at the bottom. I noticed the antenna was once again tucked under the massive shield. I guess that was the typical install. I changed it so it was pointing away from the inverter shield and reworked the cables to plug it back in. The Neurio went through the startup (flashing, then solid green, then blue and then purple). I watched it for a while and noticed it stayed solid purple, no flickering.
I don’t know if this was a fix or a sign of things to come. Other people have reported similar problems with the Neurio, including a YouTube video on how to reset it the way I did. It is rather shocking how unreliable this little box is. I understand it is a “revenue grade” meter which is likely why Tesla is using it, allowing them to report “Solar Renewable Energy Credits” (SRECs). The Inverter itself seems to have a decent meter without the Neurio which is why my workaround hack worked while waiting for the Neurio replacement. If the reset doesn’t work, I will likely revisit my “fix.”
5 November 2021
So it has been a few days after the power cycle, antenna adjustment and rewiring. The system has been stable. There are no more unexpected grid spikes in the solar or powerwall powering phases. Fingers crossed!
24 Hours – No Mystery Grid Spikes!
I’ll keep posting updates here.
Conclusion
The Tesla Solar system has been an adventure for us. I don’t regret going with Tesla even though they have improvement areas, especially related to consumer experience. We love the look of the panels and the equipment. In typical Tesla fashion, the design is stunning and feels like quality. If you do select Tesla Solar, my advice is to plan on being the project manager. Stay on top of the details to keep things moving and make sure items are not dropped. And, if you are lucky, you might even get a visit from Rocío too!
While a similar adventure may not be for everyone, if you are in the market for a Solar system, I still highly recommend checking out Tesla’s options. Use this link and you can save $300 if you do order and I get a reward too: http://ts.la/jason50054
Fun Extras
Monitoring – The Tesla App and web based portal present great animations showing the solar generation and usage. Of course, I needed a scripted way to pull this information for my own use (trending, graphing and automation). Here is a simple python module pyPowerwall to pull data from the Powerwall Gateway using your “customer” credentials: https://github.com/jasonacox/pypowerwall
Powerwall Display – I wanted to see the current solar generation and state of the Powerwall on a simple LED display. I 3D printed a case and installed the display and an ESP8266 WeMos controller. Instructions are here: https://github.com/jasonacox/Powerwall-Display
I took note of several areas of concern and improvement during my investigation and problem solving. I have recorded them here.
WPA TKIP Command Access Point – The Tesla Gateway uses this weaker method to host its WiFi access point. As I discovered the Neurio uses this same access point to send Solar Power data (if it works). WPA TKIP has been dropped due for security reasons and more modern access points use WPA2 and AES encryption (WPA2-AES).
HTTPS Security Certificate – The HTTPS certificate the Gateway uses will create a browser warning (or error) when you go to the system control portal, either via your home network or via the access point at https://192.168.91.1.
Second Factor – For setup, the user is required to toggle the switch on a Powerwall as a 2nd factor to prove authorization, which is a good thing. That works well for me since my Powerwalls are locked in my garage, but if your Powerwalls are outside next to the Gateway, an attacker on-location could easily join and toggle without you even knowing.
IoT Sensors – The main problem on my system was the Neurio W2 WiFi based sensor. This IoT device sends back power data it measure to the Gateway controller. Generally, this is an elegant way to handle transmitting sensor data between systems without having to wire things. The irony is that the Gateway and Inverter already have several wires and control signal between them. Why not add another wire and avoid any WiFi communication outages? Hopefully I will be able to replace my Neurio with a wired solution.
Solar System Plan – I asked the Tesla Advisor to provide me with the design plans developed for the City Permit. They do not provide this without asking. I am glad I asked. The plans have all the schematics for the wiring as well as the layout. I discovered several things that I wanted changed and was able to get them to update before they came onsite. If you wait until they come onsite, they may not have the materials to make the adjustment and, worse, could charge you for any changes.
I found the following github projects, references and diagrams during my investigation into my Tesla Solar Adventure. I’m pasting them all here to be helpful for anyone else experiencing the same problems. The information may not be directly related but could provide a clue.
