Category: Science

Posted on

My Tesla Solar Adventure

The Order

25 June 2021

After research and talking with several solar companies, we decided on an 8.5kW Tesla Solar plus Powerwall+ system with their new high efficiency 425W panels. The main reason for our decision was the low cost, handsomely framed panels and the whole house backup capability. Other solar companies had good backups systems but we did not find any who would provide whole house backup. And, more importantly, we were delightfully surprised to see that Tesla came in with the best price. Having said that, we would soon discover that they seemed to have significantly reduced overhead by mostly eliminating customer service.

The Install

25 September 2021

Tesla Crew

After ordering, reviewing designs, applying for HOA and City approval, we were finally ready to get the system installed. Two different crews arrived over a span of a week to get the system installed. The panels went on first. The panel install crew was professional and friendly. However, when they left I noticed that the handsome skirts (frames) we were so excited about were only installed on the front of the house (street facing roof). It looked great but I had expected to get them on the back as well. I reported it and in just a few days a technician came out and added the skirts to the back roof panels. He explained that they usually only install the skirts on the front. So, please note, if you want skirts on all your panels, make sure you let them know in advance. Also, the skirts are only put on the left, right and bottom. There are no skirts on the top to allow heat to escape from the panels during the hot summer.

Panel Mount – Torx T30

While installing the skirts for the back, I noticed one of the panel edges was sticking up about 1/2″ higher than the rest. The technician tried to fix it but he didn’t have all the tools. He only had what was needed to install the skirts. He asked me if I had a Torx T30 driver. I didn’t but he explained how I could adjust the panels myself. I picked up a T30 at our Newhall True Value store. I climbed up on the roof and found the adjustment area. I used a vice grip on the screwdriver to get enough leverage (mostly because I’m pretty weak especially when I’m up on the roof). I was able to lower the panel 1/2″ so it was flush. It looked beautiful.

Front
Back

A week later, another crew showed up to wire it in and commission the system. This meant a day without power, but we were prepared for that. I tried not to be a nuisance, but couldn’t help but watch and ask questions. I made sure all of the crew had plenty of bottled water, Gatorade and snacks, including ice cream candy bars since it was so hot. They installed the Powerwalls in our garage and wired in the breaker panels and Tesla Gateway by the utility meter. After everything was installed, they powered it up and began the commissioning process.

The Problem

During commissioning, the first signs of trouble started showing up. The installers downloaded the latest software updates but were unable to get the Solar Panels to work correctly. The solar assembly was only producing 160W in full sun which doesn’t even show up in the app. They tried for hours, upgrading, rebooting, calling. They eventually gave up after showing me that the Powerwall could power our house if they cut the mains (based on 22% charge from the factory). They explained that Tesla would send out a software patch to fix the Solar panels, most likely. 

I contacted our Tesla Advisor to report the problem and to see if they had an update. After several days of emailing, texting and calling, I received a note from the advisor that our inspection would be scheduled in 3-4 weeks and he would contact the electrician about the problem. I spent another week requesting updates but my Advisor had gone radio silent. It turns out that this is a common experience with Tesla. Assuming best intentions, I can only imagine that the advisors are understaffed and overwhelmed. Regardless, it all results in a very frustrating experience for the customer.

I did manage to finally get an update and a promise to further investigate the issue. While I waited, I decided to do some more research on the system to see if I could find the problem myself…

HIGH VOLTAGE WARNING: I need to stop here and remind everyone that these systems contain extremely high voltages and are dangerous. Hopefully it goes without saying, but please be careful if you poke around inside these electrical boxes. High voltage can be fatal.

The Investigation

You can connect to the Tesla Gateway by scanning the QR code inside the box. It will have your phone connect to the Gateway’s access point. You will need to stay close to the gateway if you connect this way (and browse to https://192.168.91.1). However, keep in mind that it is also connected to your home network and if you know how to find the IP address, you can point your browser to that IP and login as the Installer to see more details about the system.  Your browser will likely require that you ignore the security certificate warning (more on this in the observations section below) and you will need to toggle the power switch to one of your Powerwalls but it will let you in. That is essentially their 2nd factor system to ensure you are authorized.  Here is what my system looked like after installation on the main screen and on the “System” screen :

The System screen also shows details about the solar generation, Powerwalls and power usage:

Below the above list was a section for “Remote Meter” that would occasionally appear. This was particularly interesting:

Remote Meter (Vxxxxxxxxxxxxxxx)
CT 1 (Solar): —W

That seemed odd.  Also when I clicked on the solar panel icon on the main screen, it would say “Stale Meter Data” – that had me wondering if the solar meter was the real issue.

The Fix?

I first disabled the Solar Assembly by clicking “Disable” on the System screen.

I opened up the Inverter, the box above the first Powerall. There is a small latch on the bottom that will unlock and let the panel swing up. I found a wooden dowel to prop it open so it would bang on my head the whole time I was investigating.

I noticed that there was a box on the right that had a “n” LED flashing.  The code on the box was the same code that was listed as the “Remote meter” in the system’s display (the one showing no power).  Some more research and I discovered that this module is a Neurio W2-Tesla WiFi based current reader that sends the solar power data to the Gateway.

Neurio was recently purchased by Generac but you can still find manuals and some models for sale online. This particular model, W2, has been customized for Tesla. It is designed to connect to the access point of the Tesla Gateway and send the solar power data.

CT-1 Amp Probe Wire

The Neurio has a wire plugged in to the top in the CT-1 (current transformer) port. I traced it over to the solar inverter where a clamp is wrapped around the solar inverter output AC line to measure the amperage. I re-seated that connector.  

Antenna

I then noticed that there was an antenna jammed below it that was tucked to the left, under the massive metal inverter shield.  I turned the antenna to the right, in the open unshield space.  

As soon as I did these two things, the LED “n” on the box began to change and a tune started coming out of the box.  It sounded like “I am connected now”.  The flashing “n” became a solid blue light.

Eureka!

I went back to the System screen and re-activated the Solar Assembly by clicking “Enabled”.  This takes several minutes and you will see the system go through and activate the solar arrays, test relays and impedance before the assembly comes online.

As soon as the Solar Assembly came online, I started seeing kW of power show up on the Systems screen.  5800W of power was coming in, fully powering the house and charging the Powerwalls!

Not so fast…

Sadly, just two hours later my elation was destroyed.  The solar energy dropped back to zero. 

I checked the inverter.  Sure enough, the Neurio was flashing again.  I attempt the above process again, several times, but no joy this time. It would chime and go green, but then started flashing again.  Based on my research, the Neurio connects to the Tesla Gateway WiFi only.  The beeps and flashes indicate that it is unable to connect to the Gateway WiFi. 

One thought I had was to reach out to Neurio (which is now owned by Generac) to see if they could provide the API, pinout or schematics for the W2 device so I could troubleshoot at the firmware and component level. When I contacted them, Generac replied that the serial number for my device contains proprietary firmware by Tesla that they cannot support. They recommend that I contact Tesla at: 888-518-3752. Oh well, it was worth a shot.

The Workaround

I love a challenge. In fact, when something isn’t working, it is almost an addiction to me. I have to figure it out and fix it! So, I had two thoughts at this point. First, I wanted to see what the Neurio was actually doing. I thought about setting up an ESP8266 to be an WiFi access point to intercept the Neurio’s communication attempts with the Gateway. But before I do that, it occured to me, I wonder what would happen if the system didn’t have a solar meter at all. In my investigation, I discovered that the solar power meter feature is often an add-on or post-install enhancement. Maybe this was more of an add-on feature than a requirement?

At the minimum, I wanted to see if there are alternatives to the Neurio in the Tesla configuration. Unfortunately, there isn’t an easy way to edit this data. I discovered that settings could only be set during the initial setup time. That would require running the setup wizard again. I decided to be bold and fire up the installation Wizard. At the bottom of the system portal is the “Run Wizard” link. Of course, I clicked it.

WARNING: I’m fairly confident that you can completely break your Tesla Solar setup using the Wizard, maybe even disable power to your house permanently. It is intended for installers. I’m taking the risk, but you should consider this first and be cautious about proceeding. I’m also fairly confident I’m going to void something in the process, but if you put something in my house, fair game, I must hack.

The wizard is straightforward. It requires you to Stop the system, but the settings are mostly intuitive. When I arrived at the Meter screen, it had 3 different meters displayed. I apologize, I did not take screenshots but will update this blog if I capture them in the future but the screens are very basic.

Two of the sensors were for the built-in CT’s used to measure the power in the Tesla Backup Gateway (you can see them on the main bus if you open the Gateway panel – which I did).

Tesla Backup Gateway
DANGER: High voltage – Seriously!

These tested extremely fast (subsecond) were working correctly and tested “good”. The third meter, a WiFi meter, was the Solar Meter (Neurio) and it’s status was Error, unable to connect. I clicked the connect button which reported it would take 3 minutes to configure the WiFi sensor. No shock, it didn’t work. I tried it 3 more times. The “Advanced” drop down allows you to add MAC address and IP, but this didn’t help. There was a “Delete” button. I thought it might be worth a try to delete and re-add. At the bottom were options to add “WiFi” or “Wired” CTs. I tried to add the Neurio (WiFi) again, multiple times, rebooting the Neurio occasionally to see if that would help. Nothing.

Here is where something interesting happened. The Wizard would NOT let me advance because the WiFi sensor was not healthy (connected). Hum… Well, I figured I would just have to delete it to see what other screens I could find in the Wizard. I deleted the Neurio. I advanced to the next screen and was presented with a “Warning – you do not have a solar sensor selected.” Naturally, I ignored that and continued.

Commissioned! I completed the Wizard setup and the system came back online. Surprisingly, the system screen looked basically the same but the dynamic flow diagram was actually working. There were no sensor errors or warnings. Power was flowing from the Grid to the House. It was the middle of the night so I signed off and went to bed.

I know what you are thinking. This is dangerous, right? I mean, we seemed to have removed solar power observability from the platform. Will the Gateway and Inverter sill know what to do? Well, it turns out… it does!

The Power of the Sun

Next morning, I woke to discover solar generation was charging the powerwalls and our house was completely powered by the sun! I still want Telsa to fix the Neurio or, better yet, provide some hardwire CT to monitor Solar power generation. I’m assuming that the display below means that the Gateway is computing the the solar generation based on other CTs. In any case, my workaround is in place and we now have a working system again.

As I’m looking at my phone, I realize… I’m holding the power of the sun in the palm of my hand. Yes, that is a geeky Doc Ock reference. We are now powering our home with an ancient but reliable and self-regulating, thermonuclear fusion reactor… our sun.

The Return of Tesla

1 November 2021

I gave up on waiting on Tesla to respond to me about the Neurio. I figured it didn’t matter since I had a working system. A month after the install and I still didn’t have an inspection date. Then it happened. I received a text message and email from Tesla that my inspection was scheduled. There was NO DATE or TIME given. Instead of asking, I figured it didn’t matter. We would see what would happen.

The day of inspection had arrived. A surprise knock on the door and there was Ishmael from Tesla. He explained he was there to meet with the City inspector for the final inspection. I showed him the gear, the Powerwalls, the gateway and the breaker panels. He looked at me and asked, “Did the install crew not put on the warning labels?” Nope.

This was something I had noticed after the installers left. In the Tesla plans are specific instructions on where to place the red warning labels on all of the gear. It includes a helpful diagram for anyone wanting to know how to kill all power in case of emergency. I had raised this issue with my project advisor a few times, but as usual, told me he would look into it and of course, nothing happened. I explained this to Ishmael who rolled his eyes and expressed apologies and said he would need to call to get the labels or it would not pass inspection. He would wait for the delivery and get them installed and ready for the City.

Shortly after meeting Ishmael, another Tesla vehicle pulled up. I figured it was the inspection stickers, but instead, it was Rocío, a Quality Assurance technician. She told me that her job was to make sure everything was installed correctly and running. I almost hugged her! I expressed my delight and appreciation that she would check on us. I explained everything that happened including how the installers said it must have been a Tesla software bug and gave up after trying for hours to get it work. I also told her about the Neurio hack I had done to get it working. She was shocked, sympathetic and determined to fix the issue.

Naturally, Rocío attempted to reset the Neurio and discovered the same thing that I did, with the exception that she was able to get the Neurio to work if she held the connector, pressing on it in a certain way. “There is clearly something wrong with the hardware and it needs to be replaced,” she concluded. I hate to be cynical, but I was definitely thinking this new chapter in my Tesla adventure would turn into an multi-week RMA, repair order and a return visit that may get scheduled sometime next year, if I’m lucky.

To my delight, Rocío looked straight at me and said, “And we’re going to get this fixed today!” She was right! She made a phone call and 30 minutes later another Tesla van showed up with the replacement Neurio!

Rocío got it working. Less than 30 minutes later she had the entire system back online and working correctly. “That’s amazing!” I told her. She clearly saw my astonishment and said, “I used to be an installer, I know what’s needed.” Well, that was completely accurate. She didn’t stop there. She examined all the gear and climbed up on the roof to ensure all the panels were in good order.

Shortly after the good news, the warning labels arrived and were attached to the new gear, ready for the official inspection. I started passing out my sincere appreciation, candy bars, water and Gatorade to these brilliant Tesla soldiers that had come to save the day. After bidding farewell to our new friends, Rocío drove off on her shiney white stallion… uh, I mean Tesla van.

About 30 minutes later, the City Inspector arrived and after a quick survey of the installed gear with Ishmael, signed his approval. Now we are on to the Permission to Operate (PTO) by Southern California Edison.

Power Gremlins

3 November 2021

I should definitely learn to be more guarded in my optimism about this Tesla adventure. After two days of having the new Neurio re-installed, I started noticing something odd. After solar production when I would expect the Powerwalls to kick in and power the house, I would see grid power start to show up and the Powerwalls drop to zero. It would only last for a few minutes then return to normal operations. Looked at the Powerwall Dashboard I set up and can even see the grid power spiking during the day when solar production was more than enough to power the house.

Powerwall Dashboard: Notice the purple (grid power) spikes throughout the day.
These just started showing up after the new Neurio was installed.

The grid power spikes did not exist before the new Neurio. I went out to look at the inverter. The Neurio’s purple light was mostly solid but would “flicker” blue. It was random, like a candle flame not like the error condition of the previous Neurio. It was happening constantly as I watched. When the flickering would get bad, I would see the powerwall drop to zero and grid power surge. There seemed to be a correlation. At any rate, I wasn’t going to let the flickering continue.

HIGH VOLTAGE WARNING: I need to stop here again and remind everyone that these systems contain extremely high voltages and are dangerous. Hopefully it goes without saying, but please be careful if you poke around inside these electrical boxes. High voltage can be fatal.

I powered off the Neurio by unplugging the power next to the antenna at the bottom. I noticed the antenna was once again tucked under the massive shield. I guess that was the typical install. I changed it so it was pointing away from the inverter shield and reworked the cables to plug it back in. The Neurio went through the startup (flashing, then solid green, then blue and then purple). I watched it for a while and noticed it stayed solid purple, no flickering.

I don’t know if this was a fix or a sign of things to come. Other people have reported similar problems with the Neurio, including a YouTube video on how to reset it the way I did. It is rather shocking how unreliable this little box is. I understand it is a “revenue grade” meter which is likely why Tesla is using it, allowing them to report “Solar Renewable Energy Credits” (SRECs). The Inverter itself seems to have a decent meter without the Neurio which is why my workaround hack worked while waiting for the Neurio replacement. If the reset doesn’t work, I will likely revisit my “fix.”

5 January 2022

The “fix” was temporary. It appears to be a resource leak that requires the Neurio to be restarted. The good news is that Tesla finally recognized the instability and sent out a 21.44 firmware update that fixed the Powerwall from disabling solar when the Neurio goes into a bad state. Finally! I was planning on ripping the Neurio out after PTO, but now I don’t have to do that. I’m currently on firmware 21.44.1 and just heard from the community that others are seeing an upgrade to 22.1 that also upgrades the Neurio from firmware “1.6.1-Tesla” to “1.7.1-Tesla” (you have to access the vitals API and decode the protobuf binary payload to see this – see here). Hopefully that helps with stability.

Permission to Operate

2 February 2022

PTO, finally! Our utility company, Southern California Edison (SCE) granted permission to operate. It took Tesla several tries to get the PTO request submitted correctly. SCE was notifying us of all the transactions but we were not able to see the full application or help. Believe me, I tried! In any case, our installation adventure has finally come to an end. It has been seven months since we started this epic journey. It is good to finally have a fully operational system.

To be fair, we have had a working solar system with Powerwall backup since October, but without PTO. PTO means that our system is no longer in self-consumption test mode. We can now push excess solar production to the grid for a credit. For the first time ever, we see grid graph going negative!

24 hour graph of energy usage including Powerwall (green) charging and pushing to the grid (purple).

Conclusion

The Tesla Solar system has been an adventure for us. I don’t regret going with Tesla even though they have improvement areas, especially related to consumer experience. We love the look of the panels and the equipment. In typical Tesla fashion, the design is stunning and feels like quality. If you do select Tesla Solar, my advice is to plan on being the project manager. Stay on top of the details to keep things moving and make sure items are not dropped.

Regardless of who you go with, I recommend you set some non-negotiables to help you filter. Here were my non-negotiables:

  • Aesthetics – We wanted something that looked high-tech, neat, clean and symmetrical on our roof. I wanted the dark panels (no white lines) with a clean looking frame.
  • Whole House Backup – We wanted batteries capable of running our house over 24 hours in the event of a power outage and a system that would charge the batteries during the day even if power was out for extended days.
  • Home Automation and Monitoring APIs – I wanted a system I could hack, use tools to monitor, dashboard, trend and even make decision on home automation components to optimize our energy usage.
  • Off the Grid – We wanted the system sized to allow us to be self sustaining with no need to use the grid even at night, fully expecting that at some point Net Energy Metering disappears or becomes less attractive, the system will still provide us with a zero grid usage option.

This helped considerably. It eliminated the list down to a handful and the lowest cost on our list was Tesla, who also had the best aesthetics IMHO. Now to be clear, as I mentioned above, Tesla is seeing explosive demand for their option and their customer service struggles a lot. I also had the opportunity to work with several incredible Tesla technicians who helped us.

I am extremely happy with our Tesla system and would recommend it to anyone, despite the bumps along the way. Tesla hit on all my non-negotiables and is elegant, fun and powerful. It has become a delightful hobby as well as a powerful utility for our green energy mission.

While a similar adventure may not be for everyone, if you are in the market for a Solar system, I still highly recommend checking out Tesla’s options. Use this link and you can save $300 if you do order and I get a reward too: http://ts.la/jason50054

Fun Toys

I have to confess. I love toys. To me, this new Tesla Powerwall+ Energy systems is a gigantic (and expensive) toy. I have thoroughly enjoyed tinkering with the system and building electronic accessories and software to manage it. As you have seen in this post, I wrote my own python API library (pyPowerwall) and created a Powerwall Dashboard to better see what the system is doing over time (credit to other open source projects I mention below).

One thing that the Tesla is missing is a good instrument panel display. Sure, you can run the app all the time, but I wanted something that would show the solar production and other details like a physical dashboard but without opening an app. I built one. It is hanging next to the Powerwalls in our garage. Much to my wife’s initial trepidation, I also built one and hung it in our kitchen. It turns out that I’m not the only one to appreciate it… well, after a while anyway. 🙂

Here are some of my toys that I gladly share with you. Please reach out and let me know if you find these useful:

pyPowerwall Dashboard with String Data
pyPowerwall Dashboard with String Data

Powerwall Web Dashboard – The Tesla App and web based portal present great animations showing the solar generation and usage. However, the information is very limited and not design for visualizing the energy data in multiple ways. I wanted to see a year at a glance as well as the string data (how much power each group of solar panels on the roof are producing). I found this Grafana based dashboard and made some minor changes including the addition of my own python based Powerwall API Proxy. Here is a simple python API module pyPowerwall to pull data from the Powerwall Gateway using your “customer” credentials: https://github.com/jasonacox/pypowerwall. If you are wanting your own Powerwall Dashboard, it is fairly easy to set up with the instructions here using Docker Compose: https://github.com/jasonacox/Powerwall-Dashboard#powerwall-dashboard

Powerwall Digital Display

Powerwall Wall Mounted Display – I really wanted to see the current solar generation and state of the Powerwall on a simple LED digital display. I 3D printed a Tesla themed case and installed the displays to show solar, house, battery, and grid power data. The display show the solar production power at the top. House, Powerwall and Grid power data rotate through the middle display and the battery level of the Powerwalls is at the bottom (89% in this picture). You can see a video of the display running below.

The display uses a WiFi enabled systems-on-chip (SoC) ESP8266 WeMos controller and three simple TM1637 7-segment LED display modules. Naturally I used my own Arduino API library (TM1637TinyDisplay) for those and the pyPowerwall proxy to display the results. It would have been nice if Tesla had built a wall monitor to show vitals like this. I’m sure it would have been a nice animated OLED display of some sorts. But this was fun. I needed to build another toy and I love my retro-LED display. If you want to build your own, I have open sourced the design and code and uploaded here for you to use: https://github.com/jasonacox/Powerwall-Display

Observations and Suggestions

I took note of several areas of concern and improvement during my investigation and problem solving. I have recorded them here.

  • WPA TKIP Command Access Point – The Tesla Gateway uses this weaker method to host its WiFi access point. As I discovered the Neurio uses this same access point to send Solar Power data (if it works). WPA TKIP has been dropped due for security reasons and more modern access points use WPA2 and AES encryption (WPA2-AES).
  • HTTPS Security Certificate – The HTTPS certificate the Gateway uses will create a browser warning (or error) when you go to the system control portal, either via your home network or via the access point at https://192.168.91.1.
  • Second Factor – For setup, the user is required to toggle the switch on a Powerwall as a 2nd factor to prove authorization, which is a good thing. That works well for me since my Powerwalls are locked in my garage, but if your Powerwalls are outside next to the Gateway, an attacker on-location could easily join and toggle without you even knowing.
  • IoT Sensors – The main problem on my system was the Neurio W2 WiFi based sensor. This IoT device sends back power data it measure to the Gateway controller. Generally, this is an elegant way to handle transmitting sensor data between systems without having to wire things. The irony is that the Gateway and Inverter already have several wires and control signal between them. Why not add another wire and avoid any WiFi communication outages? Hopefully I will be able to replace my Neurio with a wired solution.
  • Solar System Plan – I asked the Tesla Advisor to provide me with the design plans developed for the City Permit. They do not provide this without asking. I am glad I asked. The plans have all the schematics for the wiring as well as the layout. I discovered several things that I wanted changed and was able to get them to update before they came onsite. If you wait until they come onsite, they may not have the materials to make the adjustment and, worse, could charge you for any changes.

References and Notes

I found the following github projects, references and diagrams during my investigation into my Tesla Solar Adventure. I’m pasting them all here to be helpful for anyone else experiencing the same problems. The information may not be directly related but could provide a clue.

Github Projects

Neurio W2 LED Indicator

https://support-pwrview.generac.com/s/article/Troubleshooting-Networking-Issues

Tesla Powerwall Installation – Metering Installation (link)

messages_0 (1).jpeg
https://teslamotorsclub.com/tmc/threads/tesla-inverter-neurio
Posted on

Eureka! Eureka!

“Eureka!  Eureka!” –  Archimedes

The famous Greek mathematician, physicist and astronomer, Archimedes had been given the task to verify that the king’s crown was made of pure gold.  The king suspected the goldsmith had somehow cheated him, perhaps by mixing in a cheaper metal like silver.  But he had no way of proving that so he asked Archimedes to figure it out. One day Archimedes was contemplating this problem while taking a bath. He happened to notice how the water was being displaced as he stepped into the full tub, spilling out all over the floor.  He remembered that silver weighs less than gold by volume.  It suddenly dawned on him that if he were to take the same amount of pure gold by weight as the crown, and put it into water, it should displace (spill) the same amount as the crown.  Archimedes was so thrilled with this discovery that he immediately hopped out of the bath and ran to tell the king, exclaiming “Eureka!” which means, “I found it!”.   In case you are curious, when Archimedes tested the crown, he discovered that it displaced more water than gold, indicating it was less dense (not pure gold).  So, indeed the king had been cheated by the goldsmith. You can probably guess what happened to the goldsmith!

Learning is hard.  There really isn’t a way around it.  If you want to learn something, it’s going to take effort.  I often use the excuse that the human brain is optimized to save energy.  We build models and synaptic connections to do things “without conscious thinking.”  We process huge amount of sensory data every day.  We are faced with a plethora of problems we need to solve. It would be overwhelming if it wasn’t for these optimized unconscious neural pathways that allow us to sort, react and perform our tasks without much thought.  

Learning builds more capability.  At some point in our past, we learned something new and, Eureka!  That learning was forged in our brains.  It allowed us to perform our duties while we engage our higher brain functions for more important tasks like daydreaming, pondering the next season of Loki, or wondering what’s for dinner.  Ok, so maybe those aren’t more important tasks, but you get the idea.  By leveraging our learning, we expand our capacity to respond well to incoming tasks, difficult challenges, complex changes and even enjoyable exercises. 

Learning builds on learning.  I know that sounds a bit meta, but if you examine your own experience, you know that learning builds pathways to future learning. I remember the first time I learned to program a computer.  It was hard!  I was 12 years old and wanted to make my new computer display a Christmas tree for the holidays.  Somewhere in the midst of typing in some code from a Dr. Dobbs Journal article, a Eureka moment hit and I understood the procedural flow that was happening.  I had looked at more advanced programming techniques but they were out of reach for me, at least until I hit that Eureka moment.  Suddenly that complexity was unlocked.  That eventually led me down the path to discover microprocessor design, compiler construction and operating system development.

I recently purchased some dev kits, including a LIDAR kit.  This past weekend I decided to learn how to use it to image my room as a stepping stone to my larger robotics navigation project.  The funny thing about learning is that it often takes you on roads you didn’t expect to go.  LIDARs are basically spinning measuring devices that use a laser to measure distance and send back angle and distance data.  I wanted to visualize what the LIDAR was reading but the kit didn’t provide any imaging tools.  So, I decided to learn OpenGL to render the output on my Mac. That became an exercise in itself but by the end of the weekend, I had a working project (see https://github.com/jasonacox/OpenGL-LIDAR-Display).  It was challenging and frustrating at times.  But as with any good learning effort, I had a Eureka moment that unlocked excitement and plans for future learning.  I’m looking forward to the next phase!

What are you learning?  When was the last time you had a Eureka moment?  If you haven’t already, make plans this week to tackle something new to learn.   Keep learning!

Posted on

Keep Learning

“The illiterate of the 21st century will not be those who cannot read and write but those who cannot learn, unlearn, and relearn.” – Alvin Toffler

As a kid, I was inspired by movies like TRON to explore computers.  I caught the bug as soon as I saw how easily software could be used to shape reality and improve things.  I spent 6 years skipping around between computer science, mathematics, physics, electrical and civil engineering.  I even invested a year working toward a masters in divinity until I realized Koine Greek wasn’t a new computer language.  

My dad was a civil engineer with crazy mad RPN calculator skills. He could crank out and tabulate thousands of calculations per day.  But one day I wrote a bit of software and suddenly a week’s worth of tedious error prone work was done in minutes.  He immediately hired me to “computerize” the rest of his business.  With my educational background, I thought I was set… I could change the world (or at least my corner of it), one program at a time… but I still had a lot to learn. I soon discovered I didn’t know enough about statics and dynamics for the structural engineering work I needed to do.  I also needed to dive deeper into 3D geometry and computer graphics for modeling complex systems.  I went back to school.

There is a thought that education is just a box you need to check to begin your career.  You may get certified, degreed or licensed and now you are done, right?  Nope! Those are all great, but they are just visible footprints on the stairway of continual learning.  Learning expands the mind and prepares us for more learning.  Each step along the way reveals new vistas of knowledge to explore, new puzzles to solve and new mastery to obtain.  We are forever on this learning adventure, growing and thriving on the food of knowledge, discovery and experimentation.  Every new cognitive challenged unlocked is like an awakening.  It is the physics of life.  Each discovery expands who we are and carries us to a new level.  We are alive and growing when we are learning.

Learning can be fun, but it isn’t easy.  It takes work.  It can be exhausting and even frustrating at times.  Because of that, there is a temptation to become comfortable with our current state.  We may find we can get by with what we already know, often for years without having to learn something new.  But we aren’t growing.  We become stale.  Our capacity to move ourselves, our teams and our organization forward, withers.  Like physical exercise, mental exercise is required to stay healthy, vibrant and alive.  Just as we should plan to exercise our physical bodies to stay healthy, we should plan to exercise our minds as well. 

What are you learning?  May I suggest, if you aren’t learning anything new right now, pick something new and start learning it today?  I know it isn’t easy, but embrace the constructive discomfort and expand your knowledge and skill.  Tackle it.  Imagine yourself thriving and growing.  Don’t limit yourself. Set a goal to upgrade your knowledge every week.  Study, grow and keep learning!

Posted on

A Sonic Revolution

“We knew it could become big but could have never imagined it would be a revolution.” – Lou Ottens

Magnetic tape was genius!  A ferric oxide placed on a thin plastic film allowed the world to record, store and playback audio.  It revolutionized broadcasting, radio and especially, music.  In 1960, the portability of that music was still captured in a clumsy 7-inch reel.  It wasn’t easy to play. You needed a hefty machine the size of a small suitcase.  Lou wasn’t happy with that.  

Lou loved technology.  He was ever the engineer and loved solving problems.  As a teenager during World War II, Lou made a radio for his family so they could listen to programs like Radio Oranje.  To avoid the Nazi jammers, he even constructed a primitive directional antenna.  Just as he had made those freedom broadcasts accessible to his family, he now turned his attention to democratizing music.

The 7-inch music reel was too bulky and awkward for the general population.  He wanted music to be portable and accessible.  He thought a lot about what it should be like.  Trying to envision something that didn’t yet exist, he began shaping it into a small wooden block.  It needed to be small enough to easily fit in your hand and more importantly, fit in your pocket.  His “compact cassette” tape came to life in 1963 and quickly became a must-have sensation across the world.  It unleashed a multi-billion dollar industry but also taught us how to use our own voice.  The cassette became an audio canvas for the masses to self-create their own albums or compile their own mixtapes to share with others.  For those of you who don’t have as many candles on your cake as I do, mixtapes were basically the playlists of the 1980s.

The cassette tape was a raging success, but Lou wasn’t happy with it.  He complained about the noise and distortion that would eventually plague the aging tapes. In his mission for higher fidelity, he worked with his company, Philips and Sony to co-develop the digital optical storage system, the compact disk (CD).  As he had done with the cassette tape, Lou championed a portable disk size that could be easily held and used.

Lou Ottens passed away earlier this month, at the age of 94.  He sparked a worldwide sonic revolution, but humbly dismissed his role as nothing special, instead crediting other engineers and designers for bringing his ideas to life.  Lou reminds us that making science and technology accessible is just as important as the discovery itself.  

Do we make our complex technical inventions and solutions accessible?  When technology is done well, the technology itself fades to the background and becomes “indistinguishable from magic”.  That is to say, it provides a human experience and value and doesn’t get in the way.  My challenge to us this week is to examine the solutions we deliver and ask ourselves, can we make them more accessible and magical?  

As Lou taught us, the genius of a great idea is not just in the science, it is making that technology portable and accessible.

Posted on

Restlessly Pursue Learning

“We now accept the fact that learning is a lifelong process… And the most pressing task is to teach people how to learn.” – Peter Drucker

It was hard for little Bill to sit still. He was ready to run into the world, pushing through barriers and making a difference.  He was a natural leader and soon became the first black student body president of Foshay Middle School in Los Angeles.  He continued to break norms by being the first black student body president at Polytechnic High School before graduating with honors.  He went on to get a degree at UC Berkeley before being drafted into the U.S. Army.  He served 2 years during the Korean War, attaining the rank of Captain.

After the Army, Bill enrolled in the pre-med program at UCLA.  He was accepted as the first African American medical student.  After graduating he interned at Harbor General Hospital.  He specialized in obstetrics and gynecology and opened two women’s clinics in Los Angeles.  He later became the first African-American resident at Queen of Angels Hospital in Los Angeles.  Bill loved people and had a special affection for children.  He served his longest time at Cedars-Sinai Medical Center, where he once held the record for the most infants delivered.  

Forever learning, restless and driven, Bill at the age of 52 and after 14 years of medical practice, returned to school and received his J.D. from Southwestern University School of Law.  After passing the bar, he worked as a forensic attorney helping victims in malpractice suits.  He served on the Board of Governors of the UCLA Foundation and even after retiring, went back to practice medicine and law until his passing.  Through his life, he touched so many other lives.  His impact was far reaching and he was even recognized by the U.S. Congress for his life’s work and achievements. 

Bill’s life reminds us that we can all restlessly pursue learning and improvement.  We can challenge the limits others place on us or we place on ourselves.  I hope Bill’s story inspired you as it did me.  Keep learning.  Keep striving.  Keep helping.  We can make this world a better place, if we try.  And, like Bill, we can also leave behind a legacy that can inspire the next generation. 

Oh, and one more thing…  Dr. Lawrence William “Bill” Scott not only left behind a legacy of “firsts”, he also passed on his passion of science and the restless pursuit of learning through his children.  You might know at least one of them.  His son is a leader at Disney and in the industry, a champion of technology and the restless pursuit of learning, Brian L. Scott.

Keep learning!

Posted on

A Pocket of Curiosity

“I’m just very curious—got to find out what makes things tick… all our people have this curiosity; it keeps us moving forward, exploring, experimenting, opening new doors.” – Walt Disney

Percy Spencer only had a fifth-grade education. His father passed away when he was a toddler and he left school to get a job to support his family when he was only 12. His formal education may have been cut short but that didn’t stop his learning.  He began to experiment with electricity and learning at night, after work.  He became intrigued with wireless radio when he read how it was used to direct the ship Carpathia to rescue the Titanic passengers. He joined the Navy and managed to get ahold of textbooks to teach himself mathematics and science. After his service, he was hired at Raytheon, a newly formed company designing and manufacturing vacuum tubes.  Percy was particularly interested in producing radiation, specifically the use of magnetrons to generate signals used in radar.  That was something the US Government was keen to get for the war efforts.

One day in 1945, Percy showed up at work with a chocolate candy bar hidden in his pocket. While standing in front of the magnetron he was working on, he noticed the candy bar was melting.  He was fascinated by this behavior so he sent out for some unpopped popcorn and put it in front of the magnetron.  When it popped, he knew this small wave radar radiation could be used for cooking. He put the magnetron in a metal box and thus was born the first microwave oven.

Curiosity leads to discovery.  A disadvantage can often lead to a profound benefit.  What makes the difference?  In the case of Percy Spencer, his self-guided education taught him to ask why, to experiment and learn.  An unexpected occurrence, which by all rights could be viewed as an embarrassing disaster by many of us (melted chocolate pocket anyone?) turned into a critical discovery that has brought about an amazing benefit to humanity.  His creative idea was born out of curiosity, observation and action.

This year has been challenging for all of us. The new ways of working and the difficulties before us can be perplexing and discouraging at times.  But don’t give up.  Turn that melted chocolate bar into a discovery.  Ask, what can we learn from this crisis?  What experiment can we conduct to lead us on to discovery?   Are you limiting yourself or your thinking by the echo chamber we can easily find ourselves in?   Don’t.  Try something new this week.  Observe, ask why and then seek to answer it.  I suspect we are all sitting on a goldmine of new discoveries that we have yet to entertain.  Tap your opportunities and explore the unknown to see where it leads.

The next time you heat something up in the microwave remember how a melted candy bar and an inquisitive person handed us that useful invention.

Posted on

Make a Difference

Aurora see in Wisconsin

“If you could only sense how important you are to the lives of those you meet; how important you can be to the people you may never even dream of. There is something of yourself that you leave at every meeting with another person.” ― Fred Rogers

Joan loved science.  When she was 8 years old, she declared to her family that she wanted to be a scientist.  Her mother scolded her, “Women’s brains can’t do science.”  She was crushed and went sobbing into a pillow, wondering if she had to let go of her dream.  On her 14th birthday her brother, Richard, gave her a college textbook titled “Astronomy” which included an impressive chart of scientific data produced by a female astrophysicist.  That was what she needed to encourage her to pursue her career. 

Joan earned her doctorate in physics in 1958 and went on to work at NASA and JPL where she made critical discoveries about the nature and cause of auroras, specifically the interaction of the Earth’s magnetosphere and the magnetic field of the solar wind.  She was recognized and awarded many honors for her contributions to astrophysics, sunspot cycles, environmental hazards to spaceships and climate change.  Before passing away this past July, Joan Feynman had pushed through the barriers of bad advice she had received as a child and went on to make a dent in the universe of human understanding, space travel and our world.  

We are often told what we can and cannot do.  Our families, others and our jobs can intentionally or unintentionally cast us into roles that limit our potential.  I think many of us can relate to bad advice that we have received from others or have given to ourselves.  There is a tendency for us to undervalue our significance or limit our own potential. We are surviving but are we thriving?  We turn the cogs of the machine, but are we living our potential?

You are important.  You make a difference.  The truth is that you individually bring a distinctive value to our human family.  Your individual contribution, diverse traits, history, strengths, challenges, specific talents and nuanced skills fit together into the unique puzzle that is us.  You belong.  Our teams, our organizations and our world would not be the same without you.  That is the incredible truth.  The collection of our uniqueness builds the fabric of who we are as individuals and as a group.  When someone leaves, we become less. 

What are you doing to challenge the barriers you or others have placed upon you?  What would you change?  Are you hiding any of your talents or distinctives that could make us better as a group?  Please don’t!  Bring you.  Make us all greater by being greater yourself.  Embrace the strengths and unique talents of yourself and others as part of our collective power.  Our gaps and our strengths combine to make a diverse spectrum of formidable capability that will help us, our companies and our human family become even greater.  

Each one of us has a unique opportunity to make a dent in the universe.  Encourage yourself.  Encourage others and leave a bit of yourself behind at every encounter.  Together, we become even greater. 

Posted on

An Ocean of Science

“You must never confuse faith that you will prevail in the end—which you can never afford to lose—with the discipline to confront the most brutal facts of your current reality, whatever they might be.” – Admiral Jim Stockdale

It was hot in our valley this past week!  I built a simple outdoor weather station that displays current temperature, pressure and humidity.  This is the first time I saw it go above 114°F (45°C).  I was never so glad that I had planned time off and we had arranged for home healthcare to stay with my mother-in-law so that my wife, daughters and I could get away for a day.  Our big vacation was a trip down the 126 freeway to Ventura. 

Sometimes the simple things are best.  We parked at the beach, rolled down the windows and enjoyed the cool breeze.  We ate lunch in our van and watched the surf perform its dance across the shore.  It was like each wave was an ocean exhale reminding us that time keeps moving forward.  Its cool breath swept up the beach and gently across our faces.  It was serene and relaxing. 

As you can imagine, we were not the only ones to have this brilliant idea.  The streets and beaches were full of cars and people.  Sadly, most were not wearing masks or even attempting to social distance as they wandered about between the parking lots and beaches.  It struck me how difficult it has been for us to maintain vigilance in this area as we enter our 6th month of this pandemic.  I understand the frustration and know the desire to get back to normal, without face masks, distancing or shields.  There is a temptation to dismiss the science, minimize the seriousness or even justify rebellion against these safety measures.  Some of us figure that if we ignore it, it will just go away.  Unfortunately, that can only prolong and increase the impact.

We must never lose hope.  As the ocean reminds us that time marches on, so must we.  But that faithful determination must be coupled with discipline to confront reality.  As engineers, science is the illumination and tool of our profession.  We practice the scientific method to systematically experiment, learn and devise solutions.  Uncertainty, mystery and fear are chasms that we can bridge with methodical, step by step discovery and progress.  We can tunnel through difficult realities with cunning application of knowledge and persistence. The same can apply to this coronavirus pandemic and to the challenges in our businesses.  We can use our skills and expertise to help chart a solution forward.

Are we or others assuming or inventing a reality inconsistent with our scientific training?  Are there problems in front of us that could use a methodical approach to fully uncover and fix?  Do we set the example for others of being helpful, but logical, optimistic but scientific in our approach?  While 2020 has been an extremely challenging year, it is also a reminder that we have come a long way as a human family.  Behind us is an ocean of knowledge, discovery and tools that can amplify our ability to help those before us.  This week, I challenge you to tap that reservoir and heroically apply your talents to the problems at hand.  Strengthen your mind with hope and logic and let the winds of knowledge propel us forward.  And please, like other super heroes, wear a mask.  Stay scientific (and safe) out there!

Posted on

Luminous Beings

Glowing Print

“Luminous beings are we.” – Yoda

I love building things.  During the past several weeks my girls and I have been 3D printing all sorts of characters, figures and models. It’s amazing what you can find online or build yourself with free or online tools like Tinkercad or Meshmixer.  Recently we started printing with glow-in-the-dark filament.  In a funny way, it unlocked a new nighttime routine for us.  Before going to bed, my girls will charge up their figures by holding them next to the light to have their accompanying glow.  We observed how different lights influence the glow, with the sun and UV light being the most powerful for long term glow.  

Of course, this led to the question, so how does glow-in-the-dark work?  I love those questions!  The phosphorescence material we printed is absorbing the radiation and causing a quantum magic show where the electrons absorb the energy from the light source photons. They are jumping to a higher energy state which slowly degrades over time, emitting that glow.  The unique nature of glow-in-the-dark materials like zinc sulfide and strontium aluminate is that the energy is not released immediately.  The higher energy state causes the electrons to get “trapped” in a higher state and released over the course of several minutes and even hours. Quantum mechanics loves to do this forbidden magic.  Ok, to be fair, I lost my girls on that explanation about the same way I lost some of you… so moving on.

How are you glowing?  It is amazing to me how many metaphors surround us.  This glow-in-the-dark adventure reminded me how we as humans, often radiate what we are exposed to.  I often find that in my life that I begin to emit what I allow myself to be exposed to.  If I become fixated on negative news, I become negative.  If I spend all my time hanging around critical people, I become critical.  On the flip side, if I seek and surround myself with positive people and mentors, I become more optimistic. If I change my diet to include good news as well as bad, I find that I am more encouraged and encouraging to others.  What are you feasting on?  What light sources are you orbiting?  Who and what are you bringing into your life to help you absorb good energy so that you too can glow?

In our fast action, twitter abbreviated, news cycle world I find that I often become carried away by the currents.  This little glow-in-the-dark lesson reminded me that we have a choice on where we are going and how we shape ourselves to be the people we want to be.  This pandemic can be discouraging and rob us of energy and joy.  There are a lot of negative and depressing conversations going on.  I understand that.  But we shouldn’t limit our charging to only those sources.  Find some new light sources this week.  Look for opportunities to jump to a higher energy state this week… and glow.  Here’s to a brighter future!

Posted on

ATtiny85 Weather Station

Weather Station

This project will show you how to build an ATtiny85 based mini Weather Station that displays temperature, humidity and pressure using four easy to read 7-segment LEDs.

Requirements

This sketch requires a version of the Wire library that is compatible with the ATtiny85 for the I2C communication to work with the BME-280 sensor. I used the ATTinyCore arduino core by Spence Konde which has a version of the Wire library that works with the ATtiny85. I was able to use the standard Adafruit_BME280 library to pull data from the BME-280. You can install ATTinyCore by putting the board manager URL in the Arduino IDE preferences:

http://drazzy.com/package_drazzy.com_index.json

Set the board to the ATtiny85 chip at 1Mhz (internal).

Circuit

Components:

  • ATiny85 Microcontroller (DigiKey)
  • BME-280 Sensor (Temperature, Pressure, Humidity) (Amazon)
  • 74HC595 8-bit Shift Register (Qty 4) (DigiKey)

Sensors

  • 7-Segment LED Display (Qty 4) (DigiKey)
  • 0.1uF Ceramic Capacitor (Qty 2)
  • 100uF Electrolytic Capacitor
  • 5V Power Supply (Alternatively you can use a 5V Solar cell, 3.7V lithium ion battery and a TP4056 constant-current/constant-voltage linear charger to charge the battery during the day).

Schematic

Circuit Board

ATtiny85 Microcontroller

Circuit Board

Programming Notes:

Code for this project is located on Github: https://github.com/jasonacox/ATtiny85-Weather-Station

I2C communication with BME-280 uses pins PB0/SDA and PB2/SCL. If you use the Tiny AVR Programmer from Sparkfun or something similar, keep in mind that it drives an LED on PB0 which will interfere with I2C communication. You will need to remove the chip from the programmer after uploading to get it to work in the circuit.

Memory Warning

This sketch uses nearly all of the ATtiny85 program storage space (8K) so you may get an overflow error if the libraries change or you add any code. To address this, I created a minimized BME280 library to reduce the PROGMEM space required. You will need to download and install the Tiny_BME280_Library library in:

~/Documents/Arduino/libraries/ directory and restart the Arduino IDE.

Display

  [ 70'] - Temperature in degree F 
  [ 24r] - Relative Humidity %
  [_970] - Pressure in hPa with rising/falling animation
  [ 21c] - Temperature in degree C

Construction

I used two circuit boards: Display and Logic board. The Display board holds the four 7-segment LEDs with the array of resistors. On the back are header pins that plug in to the logic board. The Logic board holds the 4 register chips and the ATtiny85. On the back are headers for the Display board to plug in.

Logic and Display boards: 

Both boards together and running: 

I printed a simple cylinder case (Thingiverse) to mount the weather station and have it on the back patio: 

Power Supply

The circuit will run on 5V or 3.3V. I used an existing solar panel, battery and TP4056 charger circuit to power this project with a 3.3V regulator.

Solar Power Option

  • 1 x 2.5W 5V/500mAh Solar Cell – Amazon
  • 1 x 5V Micro USB 1A TP4056 Lithium Battery Charging Board with Protection Charger Module Amazon
  • 1 x 3.7V 3000mAh 755068 Battery Rechargeable Lithium Polymer ion Battery Pack – Amazon
  • 1 x 3.3V Linear Regulator 250MA MCP1700-3302E/TO – DigiKey