Fixers

“I was a fixer, a builder – an inventor – ever since I can remember.” – Tom Scholz

I love this time of year.  The mornings are nice and cool (11C) but can still warm up in the afternoon (32C).  Our neighbors have decorated their homes with Fall and Halloween decorations.  Target has a mile-long line of candy aisles and we do our best to replicate that in our home.

I confess, I have an addiction. No, not the candy, I can meter that fairly well.  My real addiction is my obsession to fix things.  I can’t stand it when something is broken or a system is down. It drives me crazy. I am compelled to get it working.  This past weekend, my obsession was our new Tesla Solar system.  It all began with our decision to get solar a year ago… 

After research and talking with several solar companies, we decided on a small 8.5kW Tesla Solar plus Powerwall system. The main reason for our decision was the low cost, handsomely framed panels and the whole house backup capability. There were several good offers out there, but we were delighted when Tesla came in with the lowest price.  I’m very cheap, I admit, I love a good deal.  What we didn’t realize is that the way Tesla beat the market was to remove some of their overhead, mainly in the area of customer service.  That translated into a lot of waiting and frustration. 

Our system was finally installed last month and during commissioning, the first signs of trouble started showing up. The installers downloaded the latest software updates but were unable to get the Solar Panels to work correctly. The solar assembly was only producing 160W in full sun. They tried for hours, upgrading, rebooting and calling. They eventually gave up after showing me that the Powerwall could power our house if they cut the mains.  They explained that Tesla would send out a software patch to fix the Solar panels, most likely.  

I contacted our Tesla Advisor to report the problem and to see if they had an update. After several days of emailing, texting and calling, I received a note from the advisor that he would contact the electricians about the problem. I spent another week requesting updates but my Advisor had gone radio silent. It turns out that this is a common experience with Tesla. Assuming best intentions, I can only imagine that the Tesla teams are understaffed and overwhelmed. Regardless, it all results in a very frustrating experience for the consumer.  I escalated and was finally told that they could schedule a service appointment on Dec. 14th.  Seriously?!

We had all this gear in place.  We had these beautiful solar panels, high tech Powerwalls and all the gear to drive it.  But none of it was working.  Tesla’s mistake was bringing all of this into my home.  I can’t help myself.  My addiction set in.  I have to fix it.  So, I did.  I began poking around, measuring voltages, sniffing the system’s communication links, and researching all of the modules that make up the system.

I finally found the issue.  I discovered a WiFi based metering system Tesla had used was defective.  I reprogrammed and recommissioned the system.  It came online and for the first time ever, we had full solar energy powering our house and charging our Powerwalls.  Our home had gone green and we were 100% powered by the sun!  I love making things work.

Reflecting back on this experience it occurred to me that this is a lot of what we do.  In technology, we are often the Sherpas that help our business partners reach their destination.  Sometimes it requires research, investigation, designing, hacking, and even reprogramming to arrive at a reliable and operative solution.  All along the way, our partners are depending on us to be the experts to creatively solve problems, fix what is broken and deliver a working solution that helps them, our businesses and organizations deliver the best outcomes as we can.

Thank you all for being the fixers.  You substantially make a difference to the world!

If you want the gory details on my investigation and fix you can read them here

My Tesla Solar Adventure

The Install

After research and talking with several solar companies, we decided on an 8.5kW Tesla Solar plus Powerwall+ system with their new high efficiency 425W panels. The main reason for our decision was the low cost, handsomely framed panels and the whole house backup capability. Other solar companies had good backups systems but we did not find any who would provide whole house backup. And, more importantly, we were delightfully surprised to see that Tesla came in with the best price. Having said that, we would soon discover that they seemed to have significantly reduced overhead by mostly eliminating customer service.

Tesla Crew

After ordering, reviewing designs, applying for HOA and City approval, we were finally ready to get the system installed. Two different crews arrived over a span of a week to get the system installed. The panels went on first. The panel install crew was professional and friendly. However, when they left I noticed that the handsome frames we were so excited about were only installed on the front of the house (street facing roof). It looked great but I had expected to get them on the back as well. I reported it and in just a few days a technician came out and added the frames to the back roof panels. He explained that they usually only install the frames on the front. So, please note, if you want frames on all your panels, make sure you let them know in advance.

Panel Mount – Torx T30

While installing the frame for the back, I noticed one of the panel edges was sticking up about 1/2″ higher than the rest. The technician tried to fix it but he didn’t have all the tools. He only had what was needed to install the frame. He asked me if I had a Torx T30 driver. I didn’t but he explained how I could adjust the panels myself. I picked up a T30 at our Newhall True Value store. I climbed up on the roof and found the adjustment area. I used a vice grip on the screwdriver to get enough leverage (mostly because I’m pretty weak especially when I’m up on the roof). I was able to lower the panel 1/2″ so it was flush. It looked beautiful.

A week later, another crew showed up to wire it in and commission the system. This meant a day without power, but we were prepared for that. I tried not to be a nuisance, but couldn’t help but watch and ask questions. I made sure all of the crew had plenty of bottled water, Gatorade and snacks, including ice cream candy bars since it was so hot. They installed the Powerwalls in our garage and wired in the breaker panels and Tesla Gateway by the utility meter. After everything was installed, they powered it up and began the commissioning process.

The Problem

During commissioning, the first signs of trouble started showing up. The installers downloaded the latest software updates but were unable to get the Solar Panels to work correctly. The solar assembly was only producing 160W in full sun which doesn’t even show up in the app. They tried for hours, upgrading, rebooting, calling. They eventually gave up after showing me that the Powerwall could power our house if they cut the mains (based on 22% charge from the factory). They explained that Tesla would send out a software patch to fix the Solar panels, most likely. 

I contacted our Tesla Advisor to report the problem and to see if they had an update. After several days of emailing, texting and calling, I received a note from the advisor that our inspection would be scheduled in 3-4 weeks and he would contact the electrician about the problem. I spent another week requesting updates but my Advisor had gone radio silent. It turns out that this is a common experience with Tesla. Assuming best intentions, I can only imagine that the advisors are understaffed and overwhelmed. Regardless, it all results in a very frustrating experience for the customer.

I did manage to finally get an update and a promise to further investigate the issue. While I waited, I decided to do some more research on the system to see if I could find the problem myself…

HIGH VOLTAGE WARNING: I need to stop here and remind everyone that these systems contain extremely high voltages and are dangerous. Hopefully it goes without saying, but please be careful if you poke around inside these electrical boxes. High voltage can be fatal.

The Investigation

You can connect to the Tesla Gateway by scanning the QR code inside the box. It will have your phone connect to the Gateway’s access point. You will need to stay close to the gateway if you connect this way (and browse to https://192.168.91.1). However, keep in mind that it is also connected to your home network and if you know how to find the IP address, you can point your browser to that IP and login as the Installer to see more details about the system.  Your browser will likely require that you ignore the security certificate warning (more on this in the observations section below) and you will need to toggle the power switch to one of your Powerwalls but it will let you in. That is essentially their 2nd factor system to ensure you are authorized.  Here is what my system looked like after installation on the main screen and on the “System” screen :

The System screen also shows details about the solar generation, Powerwalls and power usage:

Below the above list was a section for “Remote Meter” that would occasionally appear. This was particularly interesting:

Remote Meter (Vxxxxxxxxxxxxxxx)
CT 1 (Solar): —W

That seemed odd.  Also when I clicked on the solar panel icon on the main screen, it would say “Stale Meter Data” – that had me wondering if the solar meter was the real issue.

The Fix?

I first disabled the Solar Assembly by clicking “Disable” on the System screen.

I opened up the Inverter, the box above the first Powerall. There is a small latch on the bottom that will unlock and let the panel swing up. I found a wooden dowel to prop it open so it would bang on my head the whole time I was investigating.

I noticed that there was a box on the right that had a “n” LED flashing.  The code on the box was the same code that was listed as the “Remote meter” in the system’s display (the one showing no power).  Some more research and I discovered that this module is a Neurio W2-Tesla WiFi based current reader that sends the solar power data to the Gateway.

Neurio was recently purchased by Generac but you can still find manuals and some models for sale online. This particular model, W2, has been customized for Tesla. It is designed to connect to the access point of the Tesla Gateway and send the solar power data.

CT-1 Amp Probe Wire

The Neurio has a wire plugged in to the top in the CT-1 (current transformer) port. I traced it over to the solar inverter where a clamp is wrapped around the solar inverter output AC line to measure the amperage. I re-seated that connector.  

Antenna

I then noticed that there was an antenna jammed below it that was tucked to the left, under the massive metal inverter shield.  I turned the antenna to the right, in the open unshield space.  

As soon as I did these two things, the LED “n” on the box began to change and a tune started coming out of the box.  It sounded like “I am connected now”.  The flashing “n” became a solid blue light.

Eureka!

I went back to the System screen and re-activated the Solar Assembly by clicking “Enabled”.  This takes several minutes and you will see the system go through and activate the solar arrays, test relays and impedance before the assembly comes online.

As soon as the Solar Assembly came online, I started seeing kW of power show up on the Systems screen.  5800W of power was coming in, fully powering the house and charging the Powerwalls!

Not so fast…

Sadly, just two hours later my elation was destroyed.  The solar energy dropped back to zero. 

I checked the inverter.  Sure enough, the Neurio was flashing again.  I attempt the above process again, several times, but no joy this time. It would chime and go green, but then started flashing again.  Based on my research, the Neurio connects to the Tesla Gateway WiFi only.  The beeps and flashes indicate that it is unable to connect to the Gateway WiFi. 

The Workaround

I love a challenge. In fact, when something is not working, it is almost an addiction to me. I have to figure it out and fix it! So, I had two thoughts at this point. First, I wanted to see what the Neurio was actually doing. I thought about setting up an ESP8266 to be an WiFi access point to intercept the Neurio’s communication attempts with the Gateway. But before I do that, it occured to me, I wonder what would happen if the system didn’t have a solar meter at all. In my investigation, I discovered that the solar power meter feature is often an add-on or post-install enhancement. Maybe this was more of an add-on feature than a requirement?

At the minimum, I wanted to see if there are alternatives to the Neurio in the Tesla configuration. Unfortunately, there isn’t an easy way to edit this data. I discovered that settings could only be set during the initial setup time. That would require running the setup wizard again. I decided to be bold and fire up the installation Wizard. At the bottom of the system portal is the “Run Wizard” link. Of course, I clicked it.

WARNING: I’m fairly confident that you can completely break your Tesla Solar setup using the Wizard, maybe even disable power to your house permanently. It is intended for installers. I’m taking the risk, but you should consider this first and be cautious about proceeding. I’m also fairly confident I’m going to void something in the process, but if you put something in my house, fair game, I must hack.

The wizard is straightforward. It requires you to Stop the system, but the settings are mostly intuitive. When I arrived at the Meter screen, it had 3 different meters displayed. I apologize, I did not take screenshots but will update this blog if I capture them in the future but the screens are very basic.

Two of the sensors were for the built-in CT’s used to measure the power in the Tesla Backup Gateway (you can see them on the main bus if you open the Gateway panel – which I did).

Tesla Backup Gateway
DANGER: High voltage – Seriously!

These tested extremely fast (subsecond) were working correctly and tested “good”. The third meter, a WiFi meter, was the Solar Meter (Neurio) and it’s status was Error, unable to connect. I clicked the connect button which reported it would take 3 minutes to configure the WiFi sensor. No shock, it didn’t work. I tried it 3 more times. The “Advanced” drop down allows you to add MAC address and IP, but this didn’t help. There was a “Delete” button. I thought it might be worth a try to delete and re-add. At the bottom were options to add “WiFi” or “Wired” CTs. I tried to add the Neurio (WiFi) again, multiple times, rebooting the Neurio occasionally to see if that would help. Nothing.

Here is where something interesting happened. The Wizard would NOT let me advance because the WiFi sensor was not healthy (connected). Hum… Well, I figured I would just have to delete it to see what other screens I could find in the Wizard. I deleted the Neurio. I advanced to the next screen and was presented with a “Warning – you do not have a solar sensor selected.” Naturally, I ignored that and continued.

Commissioned! I completed the Wizard setup and the system came back online. Surprisingly, the system screen looked basically the same but the dynamic flow diagram was actually working. There were no sensor errors or warnings. Power was flowing from the Grid to the House. It was the middle of the night so I signed off and went to bed.

I know what you are thinking. This is dangerous, right? I mean, we seemed to have removed solar power observability from the platform. Will the Gateway and Inverter sill know what to do? Well, it turns out… it does!

The Power of the Sun

Next morning, I woke to discover solar generation was charging the powerwalls and our house was completely powered by the sun! I still want Telsa to fix the Neurio or, better yet, provide some hardwire CT to monitor Solar power generation. I’m assuming that the display below means that the Gateway is computing the the solar generation based on other CTs. In any case, my workaround is in place and we now have a working system again.

As I’m looking at my phone, I realize… I’m holding the power of the sun in the palm of my hand. Yes, that is a Doc Ock reference, but it is incredibly rewarding to finally see our home go green, with 100% solar energy.

Conclusion

I will post any updates as I continue to work with Telsa and discover more about my system. Along the way I uncovered some items that I believe would be important for Tesla to address. I will post my notes in the observation section.

The Tesla Solar system has been an adventure for us. I don’t regret going with Telsa even though they have improvement areas, especially related to consumer experience. We love the look of the panels and the equipment. In typical Tesla fashion, the design is stunning and feels like quality. If you do select Tesla Solar, my advice is to plan on being the project manager. Stay on top of the details to keep things moving and make sure items are not dropped. You will be glad you did.

While a similar adventure may not be for everyone, if you are in the market for a Solar system, I still recommend checking out Tesla’s options. Use this link and you can save $300 if you do order and I get a reward too: http://ts.la/jason50054

I took note of several areas of concern and improvement during my investigation and problem solving. I have recorded them here.

  • WPA TKIP Command Access Point – The Tesla Gateway uses this weaker method to host its WiFi access point. As I discovered the Neurio uses this same access point to send Solar Power data (if it works). WPA TKIP has been dropped due for security reasons and more modern access points use WPA2 and AES encryption (WPA2-AES).
  • HTTPS Security Certificate – The HTTPS certificate the Gateway uses will create a browser warning (or error) when you go to the system control portal, either via your home network or via the access point at https://192.168.91.1.
  • Second Factor – For setup, the user is required to toggle the switch on a Powerwall as a 2nd factor to prove authorization, which is a good thing. That works well for me since my Powerwalls are locked in my garage, but if your Powerwalls are outside next to the Gateway, an attacker on-location could easily join and toggle without you even knowing.
  • IoT Sensors – The main problem on my system was the Neurio W2 WiFi based sensor. This IoT device sends back power data it measure to the Gateway controller. Generally, this is an elegant way to handle transmitting sensor data between systems without having to wire things. The irony is that the Gateway and Inverter already have several wires and control signal between them. Why not add another wire and avoid any WiFi communication outages? Hopefully I will be able to replace my Neurio with a wired solution.
  • Solar System Plan – I asked the Tesla Advisor to provide me with the design plans developed for the City Permit. They do not provide this without asking. I am glad I asked. The plans have all the schematics for the wiring as well as the layout. I discovered several things that I wanted changed and was able to get them to update before they came onsite. If you wait until they come onsite, they may not have the materials to make the adjustment and, worse, could charge you for any changes.

I found the following github projects, references and diagrams during my investigation into my Tesla Solar Adventure. I’m pasting them all here to be helpful for anyone else experiencing the same problems. The information may not be directly related but could provide a clue.

Github Projects

Neurio W2 LED Indicator

Tesla Powerwall Installation – Metering Installation (link)

Life without Fences

“We were born to be free, to expand our horizons by going where we have never gone before, and not to hang out in the relative comfort and safety of the nest, the known. There is a place within us that is courageous beyond our human understanding; it yearns to explore beyond the boundaries of our daily life.” – Dennis Merritt Jones

My mom was a teacher.  She taught 3rd grade most of her career.  Growing up, my friends thought it was great that my mom was a teacher, but it wasn’t always great for me.  Most kids got sent to the principal’s office for acting up or misbehaving.  Not me!  I got sent to my mother’s classroom.  Corporal punishment was still in play and the principal proudly hung his “board of education” paddle on the wall of his office as a deterrent.  I would have preferred a meeting with that “board” any day, if I could have avoided being sent to my mom’s classroom.  I can still picture the horror on her face as the teacher explained my activity and then hearing the stern, “Just wait till we get home.”  

It seemed like most of my “acting up” happened during recess and on the playground.  I guess it was an irresistible smorgasbord of trouble waiting for me, but really, I just loved experimenting.  I tested Newtonian physics of balls bouncing off all sorts of surfaces, including other kids.  I loved exploring fencing techniques with pretend light saber sticks and branches. I even helped other kids prove or disprove their own theories.   A cute little girl with glasses and a confident attitude once declared to me an axiom, “You can’t hit a person with glasses.”  Much to her dismay, I was able to prove her wrong.  I fully expected to be thanked if not awarded some scientific prize for my discovery, but instead, I was granted another visit to my mother’s classroom, one that had particular impact on me, literally.  

Recess!  What a glorious thing.  There are a lot of life lessons that you can learn during recess.  You also get a chance to see curious characteristics of our human traits on display.  One particularly interesting observation my mom made was the power of fences.  Having had the opportunity to work in many schools with different playgrounds, she noticed that when kids went out to play, if there was no fence, the kids would huddle together, close to the building or by the door.  Nobody ventured out very far.  When there was a fence, the kids would scatter and run all the way out to the edges, running up and down the fence line.  Why is that?

For those kids, the fences created a sense of safety, confidence and certainty.  With that in place, the kids used the entire space to explore, create adventures and have fun together.  I believe there is a good leadership lesson here.  We often talk about unleashing the potential of our team by empowering individuals to creatively solve problems, take on responsibility and innovate.  A good leader can help us manage the unknowns by bringing clarity, direction, and expectations.  They also promote psychological safety by establishing a culture that avoids the blame game, encourages risk and values continuous learning. Those structures help us navigate, explore and create results without fences.  But that is only part of the story.

The leadership lesson here also applies to us individually.  So much of our life and the world we live in has no fences.  It can be intimidating.  Our human tendencies, evoking survival instincts at times, will be to huddle close to what we know, where we feel safe, secure, confident and comfortable.  But if we do that, we miss out on the greater prospects and the rewards that can be waiting for us.  We need to take bold risks.  We should gather our courage and set out on an adventure into the unknown.  Experimenting, exploring, discovering, creating, solving and ultimately enjoying greater outcomes than we would otherwise.  

Spend some time this week in recess!  Examine your fence line.  Ask yourself, what is helpful and what isn’t.  Take a brave step and go beyond, explore outside the fences and enjoy our spectacular world.  And, hopefully, you won’t end up in my mom’s classroom.