Category: Technology

Posted on

My Tesla Solar Adventure

The Order

25 June 2021

After research and talking with several solar companies, we decided on an 8.5kW Tesla Solar plus Powerwall+ system with their new high efficiency 425W panels. The main reason for our decision was the low cost, handsomely framed panels and the whole house backup capability. Other solar companies had good backups systems but we did not find any who would provide whole house backup. And, more importantly, we were delightfully surprised to see that Tesla came in with the best price. Having said that, we would soon discover that they seemed to have significantly reduced overhead by mostly eliminating customer service.

The Install

25 September 2021

Tesla Crew

After ordering, reviewing designs, applying for HOA and City approval, we were finally ready to get the system installed. Two different crews arrived over a span of a week to get the system installed. The panels went on first. The panel install crew was professional and friendly. However, when they left I noticed that the handsome skirts (frames) we were so excited about were only installed on the front of the house (street facing roof). It looked great but I had expected to get them on the back as well. I reported it and in just a few days a technician came out and added the skirts to the back roof panels. He explained that they usually only install the skirts on the front. So, please note, if you want skirts on all your panels, make sure you let them know in advance. Also, the skirts are only put on the left, right and bottom. There are no skirts on the top to allow heat to escape from the panels during the hot summer.

Panel Mount – Torx T30

While installing the skirts for the back, I noticed one of the panel edges was sticking up about 1/2″ higher than the rest. The technician tried to fix it but he didn’t have all the tools. He only had what was needed to install the skirts. He asked me if I had a Torx T30 driver. I didn’t but he explained how I could adjust the panels myself. I picked up a T30 at our Newhall True Value store. I climbed up on the roof and found the adjustment area. I used a vice grip on the screwdriver to get enough leverage (mostly because I’m pretty weak especially when I’m up on the roof). I was able to lower the panel 1/2″ so it was flush. It looked beautiful.

  • Front
  • Back

A week later, another crew showed up to wire it in and commission the system. This meant a day without power, but we were prepared for that. I tried not to be a nuisance, but couldn’t help but watch and ask questions. I made sure all of the crew had plenty of bottled water, Gatorade and snacks, including ice cream candy bars since it was so hot. They installed the Powerwalls in our garage and wired in the breaker panels and Tesla Gateway by the utility meter. After everything was installed, they powered it up and began the commissioning process.

The Problem

During commissioning, the first signs of trouble started showing up. The installers downloaded the latest software updates but were unable to get the Solar Panels to work correctly. The solar assembly was only producing 160W in full sun which doesn’t even show up in the app. They tried for hours, upgrading, rebooting, calling. They eventually gave up after showing me that the Powerwall could power our house if they cut the mains (based on 22% charge from the factory). They explained that Tesla would send out a software patch to fix the Solar panels, most likely. 

I contacted our Tesla Advisor to report the problem and to see if they had an update. After several days of emailing, texting and calling, I received a note from the advisor that our inspection would be scheduled in 3-4 weeks and he would contact the electrician about the problem. I spent another week requesting updates but my Advisor had gone radio silent. It turns out that this is a common experience with Tesla. Assuming best intentions, I can only imagine that the advisors are understaffed and overwhelmed. Regardless, it all results in a very frustrating experience for the customer.

I did manage to finally get an update and a promise to further investigate the issue. While I waited, I decided to do some more research on the system to see if I could find the problem myself…

HIGH VOLTAGE WARNING: I need to stop here and remind everyone that these systems contain extremely high voltages and are dangerous. Hopefully it goes without saying, but please be careful if you poke around inside these electrical boxes. High voltage can be fatal.

The Investigation

You can connect to the Tesla Gateway by scanning the QR code inside the box. It will have your phone connect to the Gateway’s access point. You will need to stay close to the gateway if you connect this way (and browse to https://192.168.91.1). However, keep in mind that it is also connected to your home network and if you know how to find the IP address, you can point your browser to that IP and login as the Installer to see more details about the system.  Your browser will likely require that you ignore the security certificate warning (more on this in the observations section below) and you will need to toggle the power switch to one of your Powerwalls but it will let you in. That is essentially their 2nd factor system to ensure you are authorized.  Here is what my system looked like after installation on the main screen and on the “System” screen :

The System screen also shows details about the solar generation, Powerwalls and power usage:

Below the above list was a section for “Remote Meter” that would occasionally appear. This was particularly interesting:

Remote Meter (Vxxxxxxxxxxxxxxx)
CT 1 (Solar): —W

That seemed odd.  Also when I clicked on the solar panel icon on the main screen, it would say “Stale Meter Data” – that had me wondering if the solar meter was the real issue.

The Fix?

I first disabled the Solar Assembly by clicking “Disable” on the System screen.

I opened up the Inverter, the box above the first Powerall. There is a small latch on the bottom that will unlock and let the panel swing up. I found a wooden dowel to prop it open so it would bang on my head the whole time I was investigating.

I noticed that there was a box on the right that had a “n” LED flashing.  The code on the box was the same code that was listed as the “Remote meter” in the system’s display (the one showing no power).  Some more research and I discovered that this module is a Neurio W2-Tesla WiFi based current reader that sends the solar power data to the Gateway.

Neurio was recently purchased by Generac but you can still find manuals and some models for sale online. This particular model, W2, has been customized for Tesla. It is designed to connect to the access point of the Tesla Gateway and send the solar power data.

CT-1 Amp Probe Wire

The Neurio has a wire plugged in to the top in the CT-1 (current transformer) port. I traced it over to the solar inverter where a clamp is wrapped around the solar inverter output AC line to measure the amperage. I re-seated that connector.  

Antenna

I then noticed that there was an antenna jammed below it that was tucked to the left, under the massive metal inverter shield.  I turned the antenna to the right, in the open unshield space.  

As soon as I did these two things, the LED “n” on the box began to change and a tune started coming out of the box.  It sounded like “I am connected now”.  The flashing “n” became a solid blue light.

Eureka!

I went back to the System screen and re-activated the Solar Assembly by clicking “Enabled”.  This takes several minutes and you will see the system go through and activate the solar arrays, test relays and impedance before the assembly comes online.

As soon as the Solar Assembly came online, I started seeing kW of power show up on the Systems screen.  5800W of power was coming in, fully powering the house and charging the Powerwalls!

Not so fast…

Sadly, just two hours later my elation was destroyed.  The solar energy dropped back to zero. 

I checked the inverter.  Sure enough, the Neurio was flashing again.  I attempt the above process again, several times, but no joy this time. It would chime and go green, but then started flashing again.  Based on my research, the Neurio connects to the Tesla Gateway WiFi only.  The beeps and flashes indicate that it is unable to connect to the Gateway WiFi. 

One thought I had was to reach out to Neurio (which is now owned by Generac) to see if they could provide the API, pinout or schematics for the W2 device so I could troubleshoot at the firmware and component level. When I contacted them, Generac replied that the serial number for my device contains proprietary firmware by Tesla that they cannot support. They recommend that I contact Tesla at: 888-518-3752. Oh well, it was worth a shot.

The Workaround

I love a challenge. In fact, when something isn’t working, it is almost an addiction to me. I have to figure it out and fix it! So, I had two thoughts at this point. First, I wanted to see what the Neurio was actually doing. I thought about setting up an ESP8266 to be an WiFi access point to intercept the Neurio’s communication attempts with the Gateway. But before I do that, it occured to me, I wonder what would happen if the system didn’t have a solar meter at all. In my investigation, I discovered that the solar power meter feature is often an add-on or post-install enhancement. Maybe this was more of an add-on feature than a requirement?

At the minimum, I wanted to see if there are alternatives to the Neurio in the Tesla configuration. Unfortunately, there isn’t an easy way to edit this data. I discovered that settings could only be set during the initial setup time. That would require running the setup wizard again. I decided to be bold and fire up the installation Wizard. At the bottom of the system portal is the “Run Wizard” link. Of course, I clicked it.

WARNING: I’m fairly confident that you can completely break your Tesla Solar setup using the Wizard, maybe even disable power to your house permanently. It is intended for installers. I’m taking the risk, but you should consider this first and be cautious about proceeding. I’m also fairly confident I’m going to void something in the process, but if you put something in my house, fair game, I must hack.

The wizard is straightforward. It requires you to Stop the system, but the settings are mostly intuitive. When I arrived at the Meter screen, it had 3 different meters displayed. I apologize, I did not take screenshots but will update this blog if I capture them in the future but the screens are very basic.

Two of the sensors were for the built-in CT’s used to measure the power in the Tesla Backup Gateway (you can see them on the main bus if you open the Gateway panel – which I did).

Tesla Backup Gateway
DANGER: High voltage – Seriously!

These tested extremely fast (subsecond) were working correctly and tested “good”. The third meter, a WiFi meter, was the Solar Meter (Neurio) and it’s status was Error, unable to connect. I clicked the connect button which reported it would take 3 minutes to configure the WiFi sensor. No shock, it didn’t work. I tried it 3 more times. The “Advanced” drop down allows you to add MAC address and IP, but this didn’t help. There was a “Delete” button. I thought it might be worth a try to delete and re-add. At the bottom were options to add “WiFi” or “Wired” CTs. I tried to add the Neurio (WiFi) again, multiple times, rebooting the Neurio occasionally to see if that would help. Nothing.

Here is where something interesting happened. The Wizard would NOT let me advance because the WiFi sensor was not healthy (connected). Hum… Well, I figured I would just have to delete it to see what other screens I could find in the Wizard. I deleted the Neurio. I advanced to the next screen and was presented with a “Warning – you do not have a solar sensor selected.” Naturally, I ignored that and continued.

Commissioned! I completed the Wizard setup and the system came back online. Surprisingly, the system screen looked basically the same but the dynamic flow diagram was actually working. There were no sensor errors or warnings. Power was flowing from the Grid to the House. It was the middle of the night so I signed off and went to bed.

I know what you are thinking. This is dangerous, right? I mean, we seemed to have removed solar power observability from the platform. Will the Gateway and Inverter sill know what to do? Well, it turns out… it does!

The Power of the Sun

Next morning, I woke to discover solar generation was charging the powerwalls and our house was completely powered by the sun! I still want Telsa to fix the Neurio or, better yet, provide some hardwire CT to monitor Solar power generation. I’m assuming that the display below means that the Gateway is computing the the solar generation based on other CTs. In any case, my workaround is in place and we now have a working system again.

As I’m looking at my phone, I realize… I’m holding the power of the sun in the palm of my hand. Yes, that is a geeky Doc Ock reference. We are now powering our home with an ancient but reliable and self-regulating, thermonuclear fusion reactor… our sun.

The Return of Tesla

1 November 2021

I gave up on waiting on Tesla to respond to me about the Neurio. I figured it didn’t matter since I had a working system. A month after the install and I still didn’t have an inspection date. Then it happened. I received a text message and email from Tesla that my inspection was scheduled. There was NO DATE or TIME given. Instead of asking, I figured it didn’t matter. We would see what would happen.

The day of inspection had arrived. A surprise knock on the door and there was Ishmael from Tesla. He explained he was there to meet with the City inspector for the final inspection. I showed him the gear, the Powerwalls, the gateway and the breaker panels. He looked at me and asked, “Did the install crew not put on the warning labels?” Nope.

This was something I had noticed after the installers left. In the Tesla plans are specific instructions on where to place the red warning labels on all of the gear. It includes a helpful diagram for anyone wanting to know how to kill all power in case of emergency. I had raised this issue with my project advisor a few times, but as usual, told me he would look into it and of course, nothing happened. I explained this to Ishmael who rolled his eyes and expressed apologies and said he would need to call to get the labels or it would not pass inspection. He would wait for the delivery and get them installed and ready for the City.

Shortly after meeting Ishmael, another Tesla vehicle pulled up. I figured it was the inspection stickers, but instead, it was Rocío, a Quality Assurance technician. She told me that her job was to make sure everything was installed correctly and running. I almost hugged her! I expressed my delight and appreciation that she would check on us. I explained everything that happened including how the installers said it must have been a Tesla software bug and gave up after trying for hours to get it work. I also told her about the Neurio hack I had done to get it working. She was shocked, sympathetic and determined to fix the issue.

Naturally, Rocío attempted to reset the Neurio and discovered the same thing that I did, with the exception that she was able to get the Neurio to work if she held the connector, pressing on it in a certain way. “There is clearly something wrong with the hardware and it needs to be replaced,” she concluded. I hate to be cynical, but I was definitely thinking this new chapter in my Tesla adventure would turn into an multi-week RMA, repair order and a return visit that may get scheduled sometime next year, if I’m lucky.

To my delight, Rocío looked straight at me and said, “And we’re going to get this fixed today!” She was right! She made a phone call and 30 minutes later another Tesla van showed up with the replacement Neurio!

Rocío got it working. Less than 30 minutes later she had the entire system back online and working correctly. “That’s amazing!” I told her. She clearly saw my astonishment and said, “I used to be an installer, I know what’s needed.” Well, that was completely accurate. She didn’t stop there. She examined all the gear and climbed up on the roof to ensure all the panels were in good order.

Shortly after the good news, the warning labels arrived and were attached to the new gear, ready for the official inspection. I started passing out my sincere appreciation, candy bars, water and Gatorade to these brilliant Tesla soldiers that had come to save the day. After bidding farewell to our new friends, Rocío drove off on her shiney white stallion… uh, I mean Tesla van.

About 30 minutes later, the City Inspector arrived and after a quick survey of the installed gear with Ishmael, signed his approval. Now we are on to the Permission to Operate (PTO) by Southern California Edison.

Power Gremlins

3 November 2021

I should definitely learn to be more guarded in my optimism about this Tesla adventure. After two days of having the new Neurio re-installed, I started noticing something odd. After solar production when I would expect the Powerwalls to kick in and power the house, I would see grid power start to show up and the Powerwalls drop to zero. It would only last for a few minutes then return to normal operations. Looked at the Powerwall Monitor I set up and can even see the grid power spiking during the day when solar production was more than enough to power the house.

Powerwall Monitor: Notice the purple (grid power) spikes throughout the day.
These just started showing up after the new Neurio was installed.

The grid power spikes did not exist before the new Neurio. I went out to look at the inverter. The Neurio’s purple light was mostly solid but would “flicker” blue. It was random, like a candle flame not like the error condition of the previous Neurio. It was happening constantly as I watched. When the flickering would get bad, I would see the powerwall drop to zero and grid power surge. There seemed to be a correlation. At any rate, I wasn’t going to let the flickering continue.

HIGH VOLTAGE WARNING: I need to stop here again and remind everyone that these systems contain extremely high voltages and are dangerous. Hopefully it goes without saying, but please be careful if you poke around inside these electrical boxes. High voltage can be fatal.

I powered off the Neurio by unplugging the power next to the antenna at the bottom. I noticed the antenna was once again tucked under the massive shield. I guess that was the typical install. I changed it so it was pointing away from the inverter shield and reworked the cables to plug it back in. The Neurio went through the startup (flashing, then solid green, then blue and then purple). I watched it for a while and noticed it stayed solid purple, no flickering.

I don’t know if this was a fix or a sign of things to come. Other people have reported similar problems with the Neurio, including a YouTube video on how to reset it the way I did. It is rather shocking how unreliable this little box is. I understand it is a “revenue grade” meter which is likely why Tesla is using it, allowing them to report “Solar Renewable Energy Credits” (SRECs). The Inverter itself seems to have a decent meter without the Neurio which is why my workaround hack worked while waiting for the Neurio replacement. If the reset doesn’t work, I will likely revisit my “fix.”

5 January 2022

The “fix” was temporary. It appears to be a resource leak requiring the Neurio to be restarted. The good news is that Tesla finally recognized the instability and sent out a 21.44 firmware update that fixed the Powerwall from disable solar when the Neurio goes into a bad state. Finally! I was planning on ripping the Neurio out after PTO, but now I don’t have to do that. I’m currently on firmware 21.44.1 and just heard from the community that others are seeing an upgrade to 22.1 that also upgrades the Neurio from firmware “1.6.1-Tesla” to “1.7.1-Tesla” (you have to access the vitals API and decode the protobuf binary payload to see this – see here). Hopefully that helps with stability.

Permission to Operate

2 February 2022

PTO, finally! Our utility company, Southern California Edison (SCE) granted permission to operate. It took Tesla several tries to get the PTO request submitted correctly. SCE was notifying us of all the transactions but we were not able to see the full application or help. Believe me, I tried! In any case, our installation adventure has finally come to an end. It has been seven months since we started this epic journey. It is good to finally have a fully operational system.

To be fair, we have had a working solar system with Powerwall backup since October, but without PTO. PTO means that our system is no longer in self-consumption test mode. We can now push excess solar production to the grid for a credit. For the first time ever, we see grid graph going negative!

24 hour graph of energy usage including Powerwall (green) charging and pushing to the grid (purple).

Conclusion

The Tesla Solar system has been an adventure for us. I don’t regret going with Tesla even though they have improvement areas, especially related to consumer experience. We love the look of the panels and the equipment. In typical Tesla fashion, the design is stunning and feels like quality. If you do select Tesla Solar, my advice is to plan on being the project manager. Stay on top of the details to keep things moving and make sure items are not dropped.

While a similar adventure may not be for everyone, if you are in the market for a Solar system, I still highly recommend checking out Tesla’s options. Use this link and you can save $300 if you do order and I get a reward too: http://ts.la/jason50054

Fun Toys

I have to confess. I love toys. To me, this new Tesla Powerwall+ Energy systems is a gigantic (and expensive) toy. I have thoroughly enjoyed tinkering with the system and building electronic accessories and software to manage it. As you have seen in this post, I wrote my own python API library (pyPowerwall) and created a dashboard to better see what the system is doing over time (credit to other open source projects I mention below).

One thing that the Tesla is missing is a good instrument panel display. Sure, you can run the app all the time, but I wanted something that would show the solar production and other details like a physical dashboard but without opening an app. I built one. It is hanging next to the Powerwalls in our garage. Much to my wife’s initial trepidation, I also built one and hung it in our kitchen. It turns out that I’m not the only one to appreciate it… well, after a while anyway. 🙂

Here are some of my toys that I gladly share with you. Please reach out and let me know if you find these useful:

pyPowerwall Dashboard with String Data

Powerwall Web Dashboard – The Tesla App and web based portal present great animations showing the solar generation and usage. However, the information is very limited and not design for visualizing the energy data in multiple ways. I wanted to see a year at a glance as well as the string data (how much power each group of solar panels on the roof are producing). I found this Grafana based dashboard and made some minor changes including the addition of my own python based Powerwall API Proxy. Here is a simple python API module pyPowerwall to pull data from the Powerwall Gateway using your “customer” credentials: https://github.com/jasonacox/pypowerwall. If you are wanting your own Powerwall Dashboard, it is fairly easy to set up with the instructions here using Docker Compose: https://github.com/jasonacox/pypowerwall/tree/main/dashboard

Powerwall Digital Display

Powerwall Wall Mounted Display – I really wanted to see the current solar generation and state of the Powerwall on a simple LED digital display. I 3D printed a Tesla themed case and installed the displays to show solar, house, battery, and grid power data. The display show the solar production power at the top. House, Powerwall and Grid power data rotate through the middle display and the battery level of the Powerwalls is at the bottom (89% in this picture). You can see a video of the display running below.

The display uses a WiFi enabled systems-on-chip (SoC) ESP8266 WeMos controller and three simple TM1637 7-segment LED display modules. Naturally I used my own Arduino API library (TM1637TinyDisplay) for those and the pyPowerwall proxy to display the results. It would have been nice if Tesla had built a wall monitor to show vitals like this. I’m sure it would have been a nice animated OLED display of some sorts. But this was fun. I needed to build another toy and I love my retro-LED display. If you want to build your own, I have open sourced the design and code and uploaded here for you to use: https://github.com/jasonacox/Powerwall-Display

Observations and Suggestions

I took note of several areas of concern and improvement during my investigation and problem solving. I have recorded them here.

  • WPA TKIP Command Access Point – The Tesla Gateway uses this weaker method to host its WiFi access point. As I discovered the Neurio uses this same access point to send Solar Power data (if it works). WPA TKIP has been dropped due for security reasons and more modern access points use WPA2 and AES encryption (WPA2-AES).
  • HTTPS Security Certificate – The HTTPS certificate the Gateway uses will create a browser warning (or error) when you go to the system control portal, either via your home network or via the access point at https://192.168.91.1.
  • Second Factor – For setup, the user is required to toggle the switch on a Powerwall as a 2nd factor to prove authorization, which is a good thing. That works well for me since my Powerwalls are locked in my garage, but if your Powerwalls are outside next to the Gateway, an attacker on-location could easily join and toggle without you even knowing.
  • IoT Sensors – The main problem on my system was the Neurio W2 WiFi based sensor. This IoT device sends back power data it measure to the Gateway controller. Generally, this is an elegant way to handle transmitting sensor data between systems without having to wire things. The irony is that the Gateway and Inverter already have several wires and control signal between them. Why not add another wire and avoid any WiFi communication outages? Hopefully I will be able to replace my Neurio with a wired solution.
  • Solar System Plan – I asked the Tesla Advisor to provide me with the design plans developed for the City Permit. They do not provide this without asking. I am glad I asked. The plans have all the schematics for the wiring as well as the layout. I discovered several things that I wanted changed and was able to get them to update before they came onsite. If you wait until they come onsite, they may not have the materials to make the adjustment and, worse, could charge you for any changes.

References and Notes

I found the following github projects, references and diagrams during my investigation into my Tesla Solar Adventure. I’m pasting them all here to be helpful for anyone else experiencing the same problems. The information may not be directly related but could provide a clue.

Github Projects

Neurio W2 LED Indicator

https://support-pwrview.generac.com/s/article/Troubleshooting-Networking-Issues

Tesla Powerwall Installation – Metering Installation (link)

messages_0 (1).jpeg
https://teslamotorsclub.com/tmc/threads/tesla-inverter-neurio
Posted on

Gullible?

We are in the midst of a series of critical repairs at the Cox home. The pandemic forced us to postpone many of them, but slowly they are starting back up.  This last week involved the removal and upgrade of our failing main electrical breaker box panel.  Of course, that meant an extended power outage and the hourly “When will power be back on?” questions from my girls.  I must say, we all gained a greater appreciation for our ancestors who navigated the 1800s with gas lights, candles and no air conditioning.

Speaking of appreciating our 21st century lifestyle, I love using Apple Pay!  With the team of contractors working hard in the heat to get our power back on, I decided to make a run to our local grocery store to pick up some bottled water, Gatorade and snacks for them.  As is my custom, I paid with my Apple watch.  The customer behind me was shocked and struck up a conversation.  

I grew up in the Midwest where you expected friendly conversations with random fellow human travelers all the time.  However, that’s typically not how we do things in California.  Here, everyone tends to be more focused on their own business, mostly without even making eye contact.  But I find I still revert to my Midwest roots on occasion, much to the embarrassment of my kids, especially when there happens to be a cute baby in line with us.  I just can’t help myself.  Babies are irresistible.  In any case, I happened to run into this concerned citizen in line with me at the grocery store who was seriously worried about my Apple watch.   The conversation was really quite fun.

“Hey, aren’t you worried someone is going to steal all your information with that thing?”  I responded, “Actually, it uses an encrypted token, not my info, to complete the transaction.”   

“Like whatever, encrypted nothing, they got you!  That’s dangerous!  Can’t someone just decrypt it?”  I really wanted to start drawing a diagram to explain how it worked, but I knew the rest of the customers in line were not interested in an extended lecture.  I still switched into professor mode, “Sure, but just keep in mind, this isn’t my information directly, it is just a token identifier.”  

“Man, you really are gullible.”   I wasn’t making any progress.  He shook his head but then proceeded to pull out his credit card in plain sight.  I was able to clearly read his name, card number and expiration date printed on the front.  No, I didn’t try to memorize it but was struck by the irony.  He swiped his card with his in-clear-text magnetic stripe, also showing the CVV.  Sigh.  Yes, I guess I’m gullible.

I appreciate my friend’s paranoia, despite his negligence in protecting his own identity.  No system is 100% secure.  We know that.  Several years ago, I had the privilege of teaching a cybersecurity class at USC where we explored the anatomy of an attack.  One particular study was the 2013 Target breach. We examined all the points of vulnerability that existed in the system at that time.  It began with a phishing scheme that equipped the attackers with a contractor’s credentials to log in to the energy management system for the stores.  That led the attackers to a vulnerable Windows PC that just so happened to bridge the HVAC network with the global store network. That network was home to all the point of sale systems for all their stores.  The card readers on those systems only accepted plain text magnetic stripe data. The hackers installed BlackPOS, a malware opensource package that intercepts track data.  It began reading all of that data, sending it off to a server hosted in Russia.  They managed to extract 40 million credit cards before they were discovered.  A year later, the nearly exact same attack occurred at Home Depot, but for 56 million credit cards.

Vigilance is needed.  Reliability engineering is not just about system performance or uptime, it is also about running secure systems.  As we help design, build and run systems, this is a great reminder that we can all help safeguard our customers’ and company’s data.  Have you found a vulnerability?  Are you concerned about some missing measures or designs that should be modernized or addressed?  If so, don’t wait, raise those issues.  Speak up and act.  You can make a difference.  Let’s continue to help make our systems more secure for the good of the kingdom, our guests, businesses and fellow employees.

Posted on

Automate, Accelerate, Optimize, but first, Delete

“I think it’s very important to have a feedback loop, where you’re constantly thinking about what you’ve done and how you could be doing it better. I think that’s the single best piece of advice: constantly think about how you could be doing things better and questioning yourself.” – Elon Musk

The Tesla Model 3 production line was too slow. Demand was high but delivery was low.  The entire line was being delayed by one particular step in the battery production line.  Specifically, it was a step where a fiberglass mat was added between the battery pack and the floor pan.  Elon Musk talks about the focus that was suddenly placed on this choke point.  In an interview he gave, he says he was basically living on that production line until they could get it fixed.

Automate, Accelerate, Optimize. To address the constraint in the system that was choking the throughput, Elon goes on to explain how they focused on the automation.  To make the robot better, they adjusted the programming.  They increased the speed from 20% to 100%, optimized the paths it would take, increased the torque, removed unnecessary motion and reduced the amount of product needed.  Instead of spackling glue on the entire mat, they programmed it to deliver dabs of glue that were just enough to hold it in place, sandwiching it between the battery pack and floor.  These all added up to some minor time savings.

After investing a lot of time into the efficiency improvements, it occurred to Elon that he didn’t even know the reason for these mats.  He asked the battery safety team, “Are these mats for fire protection?”  They answered, “No, they are for noise and vibration.”  He then went to the noise vibration analysis team and asked them, “Are these mats for noise reduction?”  They answered, “No, they are for fire safety.”  

“I’m trapped in something like a Kafkaesque or Dilbert cartoon!”  Elon discovered the mats had no reason to be included.  After verifying with testing, they eliminated the unneeded parts that were choking the Model 3 production line.  Production throughput increased.

How many times have you optimized a bit of code, a process or a system only to finally realize that the best optimization was to simply delete it?  Before we take on some new work, a new project or even an improvement effort, ask yourself and others, “Do we even need this?”  We all have limited time and resources.  Some upfront investment in validating the real need can pay material dividends.  Seek to eliminate waste.  Instead of focusing on improving unneeded processes, let’s focus our efforts on things that deliver real value and outcomes.  

Before automating, ask yourself if the time investment will deliver more value than we put in.  Before accelerating, ask yourself if the haste will actually eliminate waste.  And before improving something, ask yourself if we should just delete it instead.  Challenge assumptions so we can ultimately deliver bold results that matter.

Posted on

Eureka! Eureka!

“Eureka!  Eureka!” –  Archimedes

The famous Greek mathematician, physicist and astronomer, Archimedes had been given the task to verify that the king’s crown was made of pure gold.  The king suspected the goldsmith had somehow cheated him, perhaps by mixing in a cheaper metal like silver.  But he had no way of proving that so he asked Archimedes to figure it out. One day Archimedes was contemplating this problem while taking a bath. He happened to notice how the water was being displaced as he stepped into the full tub, spilling out all over the floor.  He remembered that silver weighs less than gold by volume.  It suddenly dawned on him that if he were to take the same amount of pure gold by weight as the crown, and put it into water, it should displace (spill) the same amount as the crown.  Archimedes was so thrilled with this discovery that he immediately hopped out of the bath and ran to tell the king, exclaiming “Eureka!” which means, “I found it!”.   In case you are curious, when Archimedes tested the crown, he discovered that it displaced more water than gold, indicating it was less dense (not pure gold).  So, indeed the king had been cheated by the goldsmith. You can probably guess what happened to the goldsmith!

Learning is hard.  There really isn’t a way around it.  If you want to learn something, it’s going to take effort.  I often use the excuse that the human brain is optimized to save energy.  We build models and synaptic connections to do things “without conscious thinking.”  We process huge amount of sensory data every day.  We are faced with a plethora of problems we need to solve. It would be overwhelming if it wasn’t for these optimized unconscious neural pathways that allow us to sort, react and perform our tasks without much thought.  

Learning builds more capability.  At some point in our past, we learned something new and, Eureka!  That learning was forged in our brains.  It allowed us to perform our duties while we engage our higher brain functions for more important tasks like daydreaming, pondering the next season of Loki, or wondering what’s for dinner.  Ok, so maybe those aren’t more important tasks, but you get the idea.  By leveraging our learning, we expand our capacity to respond well to incoming tasks, difficult challenges, complex changes and even enjoyable exercises. 

Learning builds on learning.  I know that sounds a bit meta, but if you examine your own experience, you know that learning builds pathways to future learning. I remember the first time I learned to program a computer.  It was hard!  I was 12 years old and wanted to make my new computer display a Christmas tree for the holidays.  Somewhere in the midst of typing in some code from a Dr. Dobbs Journal article, a Eureka moment hit and I understood the procedural flow that was happening.  I had looked at more advanced programming techniques but they were out of reach for me, at least until I hit that Eureka moment.  Suddenly that complexity was unlocked.  That eventually led me down the path to discover microprocessor design, compiler construction and operating system development.

I recently purchased some dev kits, including a LIDAR kit.  This past weekend I decided to learn how to use it to image my room as a stepping stone to my larger robotics navigation project.  The funny thing about learning is that it often takes you on roads you didn’t expect to go.  LIDARs are basically spinning measuring devices that use a laser to measure distance and send back angle and distance data.  I wanted to visualize what the LIDAR was reading but the kit didn’t provide any imaging tools.  So, I decided to learn OpenGL to render the output on my Mac. That became an exercise in itself but by the end of the weekend, I had a working project (see https://github.com/jasonacox/OpenGL-LIDAR-Display).  It was challenging and frustrating at times.  But as with any good learning effort, I had a Eureka moment that unlocked excitement and plans for future learning.  I’m looking forward to the next phase!

What are you learning?  When was the last time you had a Eureka moment?  If you haven’t already, make plans this week to tackle something new to learn.   Keep learning!

Posted on

Keep Learning

“The illiterate of the 21st century will not be those who cannot read and write but those who cannot learn, unlearn, and relearn.” – Alvin Toffler

As a kid, I was inspired by movies like TRON to explore computers.  I caught the bug as soon as I saw how easily software could be used to shape reality and improve things.  I spent 6 years skipping around between computer science, mathematics, physics, electrical and civil engineering.  I even invested a year working toward a masters in divinity until I realized Koine Greek wasn’t a new computer language.  

My dad was a civil engineer with crazy mad RPN calculator skills. He could crank out and tabulate thousands of calculations per day.  But one day I wrote a bit of software and suddenly a week’s worth of tedious error prone work was done in minutes.  He immediately hired me to “computerize” the rest of his business.  With my educational background, I thought I was set… I could change the world (or at least my corner of it), one program at a time… but I still had a lot to learn. I soon discovered I didn’t know enough about statics and dynamics for the structural engineering work I needed to do.  I also needed to dive deeper into 3D geometry and computer graphics for modeling complex systems.  I went back to school.

There is a thought that education is just a box you need to check to begin your career.  You may get certified, degreed or licensed and now you are done, right?  Nope! Those are all great, but they are just visible footprints on the stairway of continual learning.  Learning expands the mind and prepares us for more learning.  Each step along the way reveals new vistas of knowledge to explore, new puzzles to solve and new mastery to obtain.  We are forever on this learning adventure, growing and thriving on the food of knowledge, discovery and experimentation.  Every new cognitive challenged unlocked is like an awakening.  It is the physics of life.  Each discovery expands who we are and carries us to a new level.  We are alive and growing when we are learning.

Learning can be fun, but it isn’t easy.  It takes work.  It can be exhausting and even frustrating at times.  Because of that, there is a temptation to become comfortable with our current state.  We may find we can get by with what we already know, often for years without having to learn something new.  But we aren’t growing.  We become stale.  Our capacity to move ourselves, our teams and our organization forward, withers.  Like physical exercise, mental exercise is required to stay healthy, vibrant and alive.  Just as we should plan to exercise our physical bodies to stay healthy, we should plan to exercise our minds as well. 

What are you learning?  May I suggest, if you aren’t learning anything new right now, pick something new and start learning it today?  I know it isn’t easy, but embrace the constructive discomfort and expand your knowledge and skill.  Tackle it.  Imagine yourself thriving and growing.  Don’t limit yourself. Set a goal to upgrade your knowledge every week.  Study, grow and keep learning!

Posted on

A Sonic Revolution

“We knew it could become big but could have never imagined it would be a revolution.” – Lou Ottens

Magnetic tape was genius!  A ferric oxide placed on a thin plastic film allowed the world to record, store and playback audio.  It revolutionized broadcasting, radio and especially, music.  In 1960, the portability of that music was still captured in a clumsy 7-inch reel.  It wasn’t easy to play. You needed a hefty machine the size of a small suitcase.  Lou wasn’t happy with that.  

Lou loved technology.  He was ever the engineer and loved solving problems.  As a teenager during World War II, Lou made a radio for his family so they could listen to programs like Radio Oranje.  To avoid the Nazi jammers, he even constructed a primitive directional antenna.  Just as he had made those freedom broadcasts accessible to his family, he now turned his attention to democratizing music.

The 7-inch music reel was too bulky and awkward for the general population.  He wanted music to be portable and accessible.  He thought a lot about what it should be like.  Trying to envision something that didn’t yet exist, he began shaping it into a small wooden block.  It needed to be small enough to easily fit in your hand and more importantly, fit in your pocket.  His “compact cassette” tape came to life in 1963 and quickly became a must-have sensation across the world.  It unleashed a multi-billion dollar industry but also taught us how to use our own voice.  The cassette became an audio canvas for the masses to self-create their own albums or compile their own mixtapes to share with others.  For those of you who don’t have as many candles on your cake as I do, mixtapes were basically the playlists of the 1980s.

The cassette tape was a raging success, but Lou wasn’t happy with it.  He complained about the noise and distortion that would eventually plague the aging tapes. In his mission for higher fidelity, he worked with his company, Philips and Sony to co-develop the digital optical storage system, the compact disk (CD).  As he had done with the cassette tape, Lou championed a portable disk size that could be easily held and used.

Lou Ottens passed away earlier this month, at the age of 94.  He sparked a worldwide sonic revolution, but humbly dismissed his role as nothing special, instead crediting other engineers and designers for bringing his ideas to life.  Lou reminds us that making science and technology accessible is just as important as the discovery itself.  

Do we make our complex technical inventions and solutions accessible?  When technology is done well, the technology itself fades to the background and becomes “indistinguishable from magic”.  That is to say, it provides a human experience and value and doesn’t get in the way.  My challenge to us this week is to examine the solutions we deliver and ask ourselves, can we make them more accessible and magical?  

As Lou taught us, the genius of a great idea is not just in the science, it is making that technology portable and accessible.

Posted on

See Problems

“Failure isn’t fatal, but failure to change might be.” – John Wooden

On June 4, 1942, the Japanese Navy arrived at the island of Midway to battle the United States Navy.  They had twice the number of pilots, planes and firepower.  Clearly with this much difference between these two forces, the Japanese should have won.  But that’s not what happened. In a surprising turn of events that became lore and even feature length movies, the United States won the Battle of Midway and that catastrophic defeat devastated the Japanese Navy.  It led to their inability to wage war in the Pacific for the remainder of the Second World War. 

Dr. Steven Spear from MIT, tells this story and asks the question, “When do you suppose the Japanese lost the Battle of Midway?”  Books have been written on the exact details of each maneuver during the battle to try to determine the exact moment that spelled the loss for Japan.  Prepare for a surprise.  The Battle of Midway was lost in 1929, not 1942.  Over a decade before!  Here’s the deal, by 1929 the Japanese Admiralty had locked in their assumption on how wars would be fought and won on the sea.  Everything was built upon the assumption that the entire fleet of one nation would face the entire fleet of the other head on. That doctrine dictated how they designed their aircraft, their carriers, their procedures and tactics.  They scripted the entire battle plan for Midway and conducted war games to rehearse it.

During the war game they set up a huge table with the layout of the two sides.  The Japanese Admirals sat on one side of the table and brought in junior officers to play the side of the US.  Both sides used sticks to push the wooden ships around the map.  After a few back and forth moves, a referee blew a whistle and accused the junior officer of not playing according to the battle plan.  He was kicked out and another junior officer was recruited.  This officer did the same thing as the former one, he looked down the table, realizing he was significantly outmatched, he too deviated from the battle plan and began to win against the Japanese side.  Once again, he was accused of not understanding the battle plan and dismissed.  This same thing happened until they went through all the junior officers, then petty officers and even brought in noodle vendors off the street.  Each time the US side won and the Japanese Admirals were frustrated that nobody was playing by the battle plan.  Instead of seeing the problem that these exercises were showing, they fixated on pathologically rehearsing their failed plan.  That is how the battle was lost.

The lesson here is powerful.  We often believe we know the best way to solve problems.  We can go to great lengths and details in defining and prescribing the solution.  But if the solution is not tested or we are unwilling to observe and build ways to see problems and learn, we can suffer catastrophic failures similar to the Japanese Admiralty.  We should design and test our systems in such a way that we can clearly detect problems, learn from them and alter course when discovery is made.  Avoiding that is effectively setting a course for failure.

A growth mindset seizes upon unexpected events or failures as golden moments of learning.  I believe this applies to all of life, not just our engineering efforts.  We all make plans, sometimes elaborate plans, and yet how do we react when those plans are thwarted?  Do we dismiss the opponent and try to get back to plan, or do we learn and alter our course?  I know this is a growth area for me.  I often want to push ahead with full force to get something done.  This pandemic has thwarted and change a lot of our plans.  But do we surrender or do we embrace the discovery as new opportunity?  Don’t become discouraged, fatigued or apathetic.  Convert problems into energy and redirect it towards a positive direction.  The secret power of successful businesses, teams and individuals is the ability to quickly learn and adjust to discovery. 

Are you struggling with your own plan failures?  Don’t give up!  Look at those failures as opportunities for learning and adopt the change.  The battle is not lost.  Glean the learning and become better.  You can do this.  Keep learning!

Posted on

Road Construction Ahead

“We believe that people with passion can change the world for the better… and those people that are crazy enough to think that they CAN change the world, are the ones that actually do.” – Steve Jobs

Road work ahead!  For the past two weeks we have had road construction in our neighborhood.  Like a marching band, the big equipment moved in with all the familiar drumming, scraping, cutting, dumping and rolling sounds.  In the past, I doubt I would have given it much thought as I commuted to and from the train to get to work.  But this time I have the pleasure of soaking it all in.  On our morning walks I get to see it all up close, greeting the workers moving about in some sort of construction choreography, adorned with branded masks and new coronavirus safety rules.  Watching this work, I’m reminded of the incredible value of building and maintaining this vital infrastructure for our society.  Roads, bridges, utilities, vehicles and buildings all create a platform for our communities, country and world to live, connect, conduct business and shape new ideas into reality.  Infrastructure propels us forward. 

I love technology.  Right out of college, my dad convinced me to join his civil engineering firm to help him, as he called it, “computerize the business.”  I didn’t know anything about civil engineering, but leaned into my science background and learned the complexities of land surveying, hydrology, material science, logistics and structural engineering.  I was amazed at the work that was done manually calculating, planning, drawing, erasing and drafting again.  I introduced the staff to AutoCAD and the coordinate geometry tools I had built to help accelerate their work.  But I was about to learn a big lesson in change management. 

“You’re crazy!”  The engineers were adamant that they didn’t want those “darn computers” (and other colorful adjectives) anywhere near their projects.  I couldn’t understand why everyone wouldn’t want to embrace new technology.  Instead of giving up, I pivoted and took on a housing development project myself.  Working with a skeptical but supportive colleague, we loaded all the elevation data to build contour maps and went to work planning streets, utilities, houses and storm systems.  The client loved the initial plans but after a review by the city, a major rework was required to expand the housing lots and add a park.  This is where the magic happened.  In the past, that would have been a start over scenario, but by having the entire project in the computer we only needed to make a few modifications to the model and the entire set of plans were ready to be released for construction.  The skeptics were blown away with the turnaround.  All the initial resistance gave way to aggressive adoption as they all saw how the computer had chewed away the toil, tedium and time to deliver plans. 

Technology amplifies human ability.  We leverage information technology to connect, to accelerate and displace manual steps, to elevate our capabilities and extend our horizon.   As technologists, we have an incredible opportunity to build a better world.  Through our trade, we can construct next generation digital infrastructure, better connect people to people, people to ideas, and ideas to reality. 

Technology builds upon technology.  Prior generations of tools stack to scaffold us up to the next level, which becomes the platform for the next.  Despite the entire world being in time out for COVID-19, people are still working, business is still being conducted, things are still moving forward, because of technology put into place by people like you. 

As technologists, we can make a difference.  Our human family is counting on us to help keep the information roadways maintained and improved.  Our companies are counting on us to use our expertise to power our businesses, to ship value better, faster, safer and happier. 

What crazy ambitious ideas do you have to help change the world?  What can we do to better help each other through technology?  I challenge you this week to give that some thought.  Let’s be some of the crazy ones who want to change the world.  Who knows, we might actually do it!

Posted on

Digital Scale for 3D Printer Filament

This project is to add a load cell to the filament spool holder of a Creality Ender 3 Pro 3D-printer to measure and display weight of spool. Using the tare function of the scale on an empty spool, the user can determine the amount of filament (in grams) remaining on a spool.

This project uses an Arduino or ATtiny85 microcontroller with the HX711 load cell module for weight measurement and a TM1637 4-digit LED display. Project GitHub Link

Components

  • ATiny85 Microcontroller (DigiKey)
  • TM1637 4 Digit Segment LED Display (Amazon)
  • Load Cell with HX711 Load Cell Amplifier (Amazon)
  • 100uF Electrolytic Capacitor
  • 5V Power Supply – Micro USB to DIP 5-Pin Pinboard (Amazon)

Schematic

Schematic

How to Build

The filament scale mounts on top of the Ender 3 top rail where the filament spool is located. You will need to print the load cell mount and the LED display box. This is available on Thingiverse or Tinkercad. See example build pictures below.

Setup & Calibration

Download the Arduino code here: https://github.com/jasonacox/Ender3-Filament-Digital-Scale

This sketch requires that you calibrate the load cell. This involves the following steps:

  1. Run the sketch with DEBUG true (using a Arduino Uno or other microcontroller with serial)
  2. Record the “HX711 reading” values with NO load on the scale – this is your “CAL_OFFSET“
  3. Use an trusted scale and weigh an object (grams or kg) – record this value as your “KNOWN-VALUE“
  4. Place the object on the load cell and record the “HX711 reading” – this is “CAL_VALUE“
  5. Compute the CAL_RATIO = (CAL_VALUE – CAL_OFFSET) / KNOWN-VALUE
  6. Edit the #defines in the code
    CAL_OFFSET  = -148550
    KNOWN_VALUE =  382.7186 g
    CAL_VALUE   = -107150
    CAL_RATIO   = (CAL_VALUE - CAL_OFFSET) / KNOWN_VALUE
    CAL_RATIO   = ((-107150) - (-148550 )) / (382.7186)
    CAL_RATIO   = 108.17

Programming Notes

The TARE button uses PB0. If you use the Tiny AVR Programmer from Sparkfun it drives an LED on PB0 and once the sketch is uploaded, the ATTiny will read PB0 as LOW and assume you wish to TARE the scale. You will need to remove the the chip from the programmer after uploading to get it to work correctly in the circuit.

Tare Function

On start the circuit will read the last TARE value from EEPROM and display the the current weight. Press and hold the TARE button and the current weight value will be recorded in EEPROM and subtracted from the current reading to “Zero” out the scale.

Assembly

The 3D model for the case and load cell mount is on Tinkercad and available for download on Thingiverse. The case is open-back for simplicity and it has mounting holes that use the existing spool bolts and fasteners.

I used a small 20mm wide project circuit board to mount the ATtiny85 socket, resistor, electrolytic capacitor and microswitch. The microswitch is located on the bottom and will face the hole on the front of case. A circular 3D printed button will fit in the hole and press against the microswitch to activate the TARE function.

The case is designed to hold a USB plugin board, the TM1637 display, the controller board and the HX711 load cell module. The HX711 slides in with a hole on the side to feed the load cell wires.

You will need M5 bolts to mount the load cell onto the 3D printed case bracket (see pictures below) with bolts going up into the threaded load cell holes. The filament spool holder that came with the Ender 3 Pro will attach to the “load” end of the load cell (the end marked with the down arrow and max weight). A 3D printed shim adapter will go between the filament holder and the load cell. An M4 bolt will go down and tighten into the load cell. The other M4 bolt will need a M4 nut under the shim.

Attach the scale and filament holder back on to the top of the Ender 3 Pro using the two M5 bolts and M5 T-nuts that came with the printer.

Plug in the scale and put an empty filament spool on the holder. Press the TARE button to zero out the weight (it records this TARE value in EEPROM memory so it remembers it on power cycle – you shouldn’t have to use TARE again). Now replace the empty spool with one with filament. It will now show you the estimated amount of filament remaining. it has been surprising accurate for my project. Notice the example below shows a slightly used 1kg spool is reading 992 grams.

Optional Roller Addition

I immediately noticed that when printing, as the extruder stepper pulls the filament, the weight will change, reflecting the dynamic force of the pull and the sliding friction resistance of the static spool holder rod.

I decided to addd a real roller to the spool holder so I printed this model that uses two bearings to remove friction: https://www.thingiverse.com/thing:3209211

References

Posted on

ATtiny85 Weather Station

Weather Station

This project will show you how to build an ATtiny85 based mini Weather Station that displays temperature, humidity and pressure using four easy to read 7-segment LEDs.

Requirements

This sketch requires a version of the Wire library that is compatible with the ATtiny85 for the I2C communication to work with the BME-280 sensor. I used the ATTinyCore arduino core by Spence Konde which has a version of the Wire library that works with the ATtiny85. I was able to use the standard Adafruit_BME280 library to pull data from the BME-280. You can install ATTinyCore by putting the board manager URL in the Arduino IDE preferences:

http://drazzy.com/package_drazzy.com_index.json

Set the board to the ATtiny85 chip at 1Mhz (internal).

Circuit

Components:

  • ATiny85 Microcontroller (DigiKey)
  • BME-280 Sensor (Temperature, Pressure, Humidity) (Amazon)
  • 74HC595 8-bit Shift Register (Qty 4) (DigiKey)

Sensors

  • 7-Segment LED Display (Qty 4) (DigiKey)
  • 0.1uF Ceramic Capacitor (Qty 2)
  • 100uF Electrolytic Capacitor
  • 5V Power Supply (Alternatively you can use a 5V Solar cell, 3.7V lithium ion battery and a TP4056 constant-current/constant-voltage linear charger to charge the battery during the day).

Schematic

Circuit Board

ATtiny85 Microcontroller

Circuit Board

Programming Notes:

Code for this project is located on Github: https://github.com/jasonacox/ATtiny85-Weather-Station

I2C communication with BME-280 uses pins PB0/SDA and PB2/SCL. If you use the Tiny AVR Programmer from Sparkfun or something similar, keep in mind that it drives an LED on PB0 which will interfere with I2C communication. You will need to remove the chip from the programmer after uploading to get it to work in the circuit.

Memory Warning

This sketch uses nearly all of the ATtiny85 program storage space (8K) so you may get an overflow error if the libraries change or you add any code. To address this, I created a minimized BME280 library to reduce the PROGMEM space required. You will need to download and install the Tiny_BME280_Library library in:

~/Documents/Arduino/libraries/ directory and restart the Arduino IDE.

Display

  [ 70'] - Temperature in degree F 
  [ 24r] - Relative Humidity %
  [_970] - Pressure in hPa with rising/falling animation
  [ 21c] - Temperature in degree C

Construction

I used two circuit boards: Display and Logic board. The Display board holds the four 7-segment LEDs with the array of resistors. On the back are header pins that plug in to the logic board. The Logic board holds the 4 register chips and the ATtiny85. On the back are headers for the Display board to plug in.

Logic and Display boards: 

Both boards together and running: 

I printed a simple cylinder case (Thingiverse) to mount the weather station and have it on the back patio: 

Power Supply

The circuit will run on 5V or 3.3V. I used an existing solar panel, battery and TP4056 charger circuit to power this project with a 3.3V regulator.

Solar Power Option

  • 1 x 2.5W 5V/500mAh Solar Cell – Amazon
  • 1 x 5V Micro USB 1A TP4056 Lithium Battery Charging Board with Protection Charger Module Amazon
  • 1 x 3.7V 3000mAh 755068 Battery Rechargeable Lithium Polymer ion Battery Pack – Amazon
  • 1 x 3.3V Linear Regulator 250MA MCP1700-3302E/TO – DigiKey