Category: Technology

Posted on

Wiring for Outcomes

“Come back in half an hour,” the host at Brix Italian Restaurant in Belleville, New Jersey, said with a sympathetic smile. The catering order for the wedding rehearsal dinner for about 100 people was not quite ready. When I had been sent to pick it up, what I didn’t know was that the restaurant was waiting to make a call back or see a “real person” show up for the order before starting on the final preparation. My daughter and I fancied a trip to Starbucks while we waited. I know, those of you who know me, are shocked to hear that. When we returned, tins full of food and utensils were waiting for us. We packed up the food and delivered it to the dining hall just in time to discover that there was an error with the order. Several, actually. My brother-in-law started on a list of “Oh no! Where is this?” questions. Critical items were missing, demanding a journey back to Brix. Arriving back at the restaurant, I learned that there had been several miscommunications between the person ordering and the host. Both sides misunderstood things. A few calls and transactions later, and another trip to Starbucks (of course), and we were ready for final delivery back to the venue. Urgent text and calls were coming in. We were late. We carried in the last dishes as the first guests started to arrive. It worked out, eventually.

Two weeks ago, my family and I made a trip to New Jersey to attend my niece’s wedding. My wife had helped her sister plan the event. As you can see from the story above, I had the audacious task of being the gopher, picking up supplies and orders. I wasn’t involved in the planning or decisions; I was just following orders. I didn’t mind. I got to spend time with my kids, even if it was just doing errands around the city. And yes, on many occasions it involved a coffee stop. But during this whole experience, I couldn’t help but see the inefficiencies and problems with this system. I was getting instructions to do things without any context as to why. Blindly following orders often means information gaps, inefficiencies, and lower quality, suboptimal outcomes. I saw that play out a dozen times. 

Information is gold, but if you don’t have access to it, it is no different than any other rock in the mine. Empowering the person doing the work with the information they need to do the work is critical. For example, if I had understood the dietary plans for the rehearsal dinner or had been given insight into the rest of the menu, I would have been able to make decisions and double check the order before even leaving the restaurant. We were not operating as a team, but as siloed functions. The same thing happens in organizations. We often create towers of expertise and create transactional methods between those groups to get the work done. But sadly, there is often catastrophic context loss between those silos that results in a lack of clarity, misunderstandings, and errors. Tickets bounce back and forth between groups like ping pong balls. Multiple meetings are scheduled to close the gaps. Deliveries are delayed. Estimates are breached. Service is reduced. Teams are frustrated and outcomes are barely adequate. Sound familiar?

I’m a big proponent of aligning full-stack teams around outcomes. Enable low latency collaboration through proximity. Embed expertise close to the problem and enrich those team members with the greater context. In my example, if I had been embedded in the planning team, I would have understood the nuances needed to ensure alignment with the goals. When supply issues at the restaurant resulted in the need to pivot away from the written requirements, I could have easily and quickly made the changes that would have aligned with the menu goals because I was part of those plans. The same applies to our engineering teams. Don’t just understand the tasks in the user story you pull off the backlog, understand the why. When the technical or demand landscape changes, the engineer is empowered to apply problem solving skills that are relevant and contribute to the final outcome. By being embedded in the product team, each team member, regardless of their functional expertise, understands the goals, the common purposes, and each is able to quickly adapt and solve for unexpected complexities and changes. Gene Kim and Dr. Steven Spear call this “wiring for a winning organization.”

“Part of wiring an organization to win is to ensure that leaders at all levels are able to create conditions in which people can give the fullest expression to their problem-solving potential, both individually and through collective action toward a common purpose.” 
- Gene Kim and Steven J. Spear, Wiring the Winning Organization

I’m a big fan of embedding engineers into business and product teams. It promotes proximity powered empathy engineering, unlocking information flow and enabling all the engineers and the rest of the product team to move fast. With context powered agility, team members can react to complex and problematic occurrences with elegance and innovation. I’m also fully aware that we all still have work to do. There are gaps we can close and other things we can do to make things better. If you find yourself driving around New Jersey blindly delivering wrong things at the wrong time, I can relate. Let’s collaborate! Let’s rewire and make it better. Oh, and of course, let’s stop by Starbucks on the way.

Posted on

AI Assistants

“That’s not AI, that’s three IF statements in a trench coat”

“This can’t be happening!” John was stressed out. He stared intently at the screen with bloodshot eyes betraying his failing attempt to hide his all-nighter. He never intended to stay up all night on this coding binge, but he was eager to impress his new team. 

Fresh out of college, this was John’s first real project. It had been going exceptionally well and earlier in the night, he was euphoric with the progress. But now he was stuck. The complex logic that had previously worked was no longer delivering the right results with the new test data. What changed? Quickly he began adding debug prints and assertions to narrow in on the defect. 

This was going to take several more hours, he thought to himself. Anxiety set in. Just four hours before the demo was scheduled. “Why in the world did I schedule that demo?”

Then it hit him. Didn’t Julie tell him that they had just rolled out a new AI tool for coders? He flipped over to his email inbox and found the announcement. “Step 1: Download this plugin to your IDE.” He followed the steps and soon the plugin came to life. A dropdown menu appeared highlighting quick action features like “Explain this”, “Document this”, “Test this”, and then he saw the new AI gourmet hamburger menu serve up a glorious “Fix this” tile.

“Yes!” Click! He literally held his breath. The AI went to work. A spinning wheel soon started churning out text. It first described the section of code he was debugging, correctly outlining how it was building the result, even complimenting him on the code. Ugh, that’s not helping, he thought. But then the AI assistant added at the end, “However, this one line seems to have an incorrect indentation that could be preventing expected results. Would you like me to fix it (Y/n)?”

John laughed and almost cried as he clicked yes. “Of course! I can’t believe I missed that!” Suddenly, his code was working as expected. He was ready for the demo, even if he was more ready for a good night’s sleep.

—-

Sasha was the departmental wizard. She was the most senior engineer and had more history in the company than anyone else. Need to know how something worked or the history on why it worked the way it did? Just ask Sasha. She probably built it! As she fired up her IDE to start the new project, she smiled. “I’m going to AI the heck out of this” she said to herself. The keyboard exploded to life as her fingers flooded the screen with instructive text. She described the data structures, global settings, APIs and logic required to complete the project. Like magic, classes and functions began to appear in translucent text below her cursor. 

“Tab. Tab. Enter.” she verbalized her actions, smiling with each keystroke as code materialized on the screen. The AI assistant was filling in all the code. It was powerful! Quickly scanning the logic, she hummed her approval. 

“Nice!” she exclaimed and scrolled down and entered more instructive comments, again followed by the AI assistant quickly filling out the details. She made some minor changes to variables to match the company style. The AI adapted and started using the same style in the next coding blocks. 

Sasha shook her head, “This is just brilliant,” she laughed. Further down she began writing the complex logic to complete the project. The AI didn’t get all of it right. But it was easy to tweak the changes she needed. She occasionally ignored some of the suggestions from the AI but was quick to accept the suggestions that would hydrate data structures when she needed them, removing that tedium and making it easier for her to tackle the more difficult sections.

“Done!” Sasha folded her arms and looked at the team around her with a great deal of satisfaction. “It’s working!” This 6-hour job only took 3 hours to complete, thanks to this AI assistant.

—-

Coming soon, to an IDE near you… These new AI assistants are starting to show up everywhere. They are ready to help. They can code, test, debug, and fix. They are always ready to serve. But the question is, are you ready for them?

Well, I don’t know about you, but I’m ready! I first started using GitHub CoPilot for my personal side projects, allowing it to help write code, translate code, review, and even fix my code. Like those fanciful stories above, I’ve been nothing but amazed at this incredible tool and its ability to amplify my efforts. It feels so good, so empowering and expressive.

I confess, I love coding. I believe every technologist, including leaders, should stay “in the code” to some degree. It’s both grounding and inspiring at the same time. Coding is art. It’s so satisfying to sculpt a digital canvass and watch a program emerge. But I admit, these AI coding assistants took it to the next level for me. I feel like the creative director for my projects, not just the keyboard hacker. I nudge my idea out there and the AI reads my mind, filling in the tedium and doing the toil for me. It’s simply brilliant!

Some adult supervision required. Every suggestion the AI makes is an opportunity for human judgement. I confess that I have learned a lot from the AI suggesting an approach I wouldn’t have done myself, but I have also seen it make a miss or two. All good. I don’t mind helping my digital Padawan navigate the complexities of programming. As the coding Jedi Masters, that is my role after all. Review the work. Validate the logic. Yes, and even learn a thing or two myself.

Someone once said, “You’re not going to lose your job to AI, you’re going to lose your job to someone who knows how to use AI.” Get busy learning how to use these new tools. I think you will love them. Prove me wrong! Are you using tools like GitHub CoPilot yet? What are your experiences? I would love to hear from you.

These tools are the worst they will ever be, they are just going to get better. But I believe the same thing about all of you. We have an incredible ability to adapt, create and become more than we were before. Go at it, learn something new, and grow.

Posted on

1202

“That’s one small step for man, one giant leap for mankind.” – Neil Armstrong

July 20, 1969. Neil Armstrong and Edwin “Buzz” Aldrin became the first humans to ever set foot on the moon. But it almost didn’t happen and it almost ended in tragedy. As the Apollo 11 Lunar Excursion Module (LEM) was preparing to land on the moon, the onboard navigational computer started flashing a “1202” alarm. The crew had been meticulously following their checklist. Each step, nominal. But now, something was wrong. Abort? As the crew radioed in the situation to mission control, they could feel the adrenaline surge and anxiety rise.

For months, the crew, the nation and the world were anticipating this historic moment. It was one of the most heavily covered and widely watched events in history. An estimated 600 million people were watching worldwide. The mission had captured the imagination of people. Now, all of it was in jeopardy. “1202” alarm! The alarms kept going off. Each time the LEM guidance computer flashed that alarm, it would reboot and restart. Not good! I can almost feel that tension myself. This was a critical stage that would demand precision to guarantee the safe landing of the module on the treacherous moon’s surface below. Sounds like bad news, right? Would this require the mission to abort?

With millions of people, sitting on the edge of their seats, Mission Control finally responded. The mission would proceed. Relief! It turns out that this was a “known error” that NASA had seen many times before during simulation testing. The computer had a capacity of 2KB erasable memory and 16KB of fixed memory. The computer would run several concurrent programs related to navigation, all competing for the limited memory. If a program couldn’t allocate memory, the “1202” alarm would be raised and the system would reboot. At restart, the most important programs would start up again where they left off. Thankfully, the mission would proceed. Neil Armstrong would soon step off of the LEM and millions of people would hear him say those “one small step” historic words.

But the mission wasn’t over. The mission was to get them safely home as well. Unfortunately, while the astronauts were suiting up for their moon walk, they accidentally bumped into the button of a circuit breaker. It broke off. This switch controlled the power running the ascent engine, the one responsible for getting them off of the moon. Unless it could be fixed, they would be stranded on the moon. NASA and US President Nixon were preparing for the worse, drafting speeches to be given when their oxygen supply ran out. Thankfully, it wouldn’t be needed. Mission control didn’t have a solution, but Buzz Aldrin did. His background in mechanical engineering paid off! He looked at the small opening where the circuit breaker had been and realized he could manage to depress the breaker with a small felt-tip marker. He did and it worked! Mission control reported the circuit was closed. In my mind’s eye, I can’t help but play out that scenario. I imagine Buzz pushing in that pen and saying with confidence, “To Infinity and Beyond!”

Problems always happen. It isn’t a matter of “if” but “when”. What do we do to prepare for them? What do we do when they happen? The story above reminds me of the importance of preparation. The “1202” alarm could have killed the mission, but it didn’t because NASA had invested in time to play through the simulation many times. Seeing this exact alarm gave them confidence in the LEM computer’s ability to recover from this condition. Testing is important, not just to prove that something is ready for launch, but to build knowledge. The testing didn’t remove the alert, but gave the mission team a foundation of experience to make difficult decisions in the heat of the moment.

Not every possible condition can be tested or will be discovered during simulation. As the circuit breaker example highlights, creative problem solving is still needed. The Apollo mission is full of stories like this, but it isn’t alone. We need engineers. We need smart creatives who are capable of plotting solutions across seemingly impossible odds.

Hopefully you won’t find yourself stranded on the moon anytime soon, but I bet you could be running simulations for learning or plotting solutions to problems. You are engineers. You are creatives. You are critical to the mission! Thanks for all you do in helping making the impossible, possible, every day.

To infinity and beyond!

References

Images

  • NASA – Aldrin on the LM footpad
    https://history.nasa.gov/ap11ann/kippsphotos/5869.jpg
  • NASA – Aldrin beside solar wind experiment https://history.nasa.gov/ap11ann/kippsphotos/5873.jpg
Posted on

The Next Word

“I’m just very curious—got to find out what makes things tick… all our people have this curiosity; it keeps us moving forward, exploring, experimenting, opening new doors.” – Walt Disney

One word at a time. It is like a stream of consciousness. Actions, objects, colors, feelings and sounds paint across the page like a slow moving brush. Each word adds to the crescendo of thought. Each phrase, a lattice of cognition. It assembles structure. It conveys scenes. It expresses logic, reason and reality in strokes of font and punctuation. It is the miracle of writing. Words strung together, one by one, single file, transcending and preserving time and thought.

I love writing. But it isn’t the letters on the page that excite me. It is the progression of thought. Think about this for a moment. How do you think? I suspect you use words. In fact, I bet you have been talking to yourself today. I promise, I won’t tell! Sure, you may imagine pictures or solve puzzles through spatial inference, but if you are like me, you think in words too. Those “words” are likely more than English. You probably use tokens, symbols and math expressions to think as well. If you know more than one language, you have probably discovered that there are some ways you can’t think in English and must use the other forms. You likely form ideas, solve problems and express yourself through a progression of those words and tokens.

Over the past few weekends I have been experimenting with large language models (LLMs) that I can configure, fine tune and run on consumer grade hardware. By that, I mean something that will run on an old Intel i5 system with a Nvidia GTX 1060 GPU. Yes, it is a dinosaur by today’s standards, but it is what I had handy. And, believe it or not, I got it to work! 

Before I explain what I discovered, I want to talk about these LLMs. I suspect you have all personally seen and experimented with ChatGPT, Bard, Claude or the many other LLM chatbots out there. They are amazing. You can have a conversation with them. They provide well-structured thought, information and advice. They can reason and solve simple puzzles. Researchers agree that they would probably even pass the Turing test. How are these things doing that?

LLMs are made up of neural nets. Once trained, they receive an input and provide an output. But they have only one job. They provide one word (or token) at a time. Not just any word, the “next word.” They are predictive language completers. When you provide a prompt as the input, the LLM’s neural network will determine the most probable next word it should produce. Isn’t that funny? They just guess the next word! Wow, how is that intelligent? Oh wait… guess what? That’s sort of what we do too! 

So how does this “next word guessing” produce anything intelligent? Well, it turns out, it’s all because of context. The LLM networks were trained using self-attention to focus on the most relevant context. The mechanics of how it works are too much for a Monday email, but if you want to read more see the paper, Attention Is All You Need which is key in how we got to the current surge in generative pre-trained transformer (GPT) technology. That approach was used to train these models on massive amounts of written text and code. Something interesting began to emerge. Hyper-dimensional attributes formed. LLMs began to understand logic, syntax and semantics. They began to be able to provide logical answers to prompts given to them, recursively completing them one word at a time to form an intelligent thought.

Back to my experiment… Once a language model is trained, the read-only model can be used to answer prompts, including questions or conversations. There are many open source versions out there on platforms like Huggingface. Companies like Microsoft, OpenAI, Meta and Google have built their own and sell or provide for free. I downloaded the free Llama 2 Chat model. It comes in 7, 13 and 70 billion parameter models. Parameters are essentially the variables that the model uses to make predictions to generate text. Generally, the higher the parameters, the more intelligent the model. Of course, the higher it is, the larger the memory and hardware footprint needed to run the model. For my case, I used the 7B model with the neural net weights quantized to 5-bits to further reduce the memory needs. I was trying to fit the entire model within the GPU’s VRAM. Sadly, it needed slightly over the 6GB I had. But I was able to split the neural network, loading 32 of the key neural network layers into the GPU and keeping the rest on the CPU. With that, I was able to achieve 14 tokens per second (a way to measure how fast the model generates words). Not bad!

I began to test the model. I love to test LLMs with a simple riddle*. You would probably not be surprised to know that many models tell me I haven’t given them enough information to answer the question. To be fair, some humans do to. But for my experiment, the model answered correctly: 

> Ram's mom has three children, Reshma, Raja and a third one. What is the name of the third child?

The third child's name is Ram.

I went on to have the model help me write some code to build a python flask based chatbot app. It makes mistakes, especially in code, but was extremely helpful in accelerating my project. It has become a valuable assistant for my weekend coding distractions. My next project is to provide a vector database to allow it to reference additional information and pull current data from external sources.

I said this before, but I do believe we are on the cusp of a technological transformation. These are incredible tools. As with many other technologies that have been introduced, it has the amazing potential to amplify our human ability. Not replacing humans, but expanding and strengthening us. I don’t know about you, but I’m excited to see where this goes!

Stay curious! Keep experimenting and learning new things. And by all means, keep writing. Keep thinking. It is what we do… on to the next word… one after the other… until we reach… the end.

Posted on

JasonGPT-1 : Adventures in AI

Distorted sci-fi black and blue world.

“Imperfect things with a positive ingredient can become a positive difference.” – JasonGPT

I don’t know how you are wired, but for me, I become intoxicated with new technology. I have a compulsive need to learn all about it. I’m also a kinesthetic learner which means I need to be hands on. So into the code I go. My latest fixation is large language models (LLMs) and the underlying generative neural network (NN) transformers (GPTs) that power them. I confess, the last time I built a NN, we were trying to read George H.W. Bush’s lips. And no, that experiment didn’t work out too well for us… or for him! 

Do you want to know what I have discovered so far? Too bad. I thought I would take you along for the ride anyway. Seriously, if you are fed up with all the artificial intelligence news and additives, you can stop now and go about your week. I won’t mind. Otherwise, hang on, I’m going to take you on an Indiana Jones style adventure through GPT! Just don’t look into the eyes of the idol… that could be dangerous, very dangerous!

Where do we start? YouTube of course! I have a new nerd crush. His name is Andrej Karpathy. He is a Slovak-Canadian computer scientist who served as the director of artificial intelligence and Autopilot Vision at Tesla and currently works for OpenAI. He lectured at Standford University and has several good instructional lectures on YouTube. I first saw him at the Microsoft Build conference where he gave a keynote on ChatGPT but what blew me away was his talk, “Let’s build GPT: from scratch, in code, spelled out.” (YouTube link). It’s no joke. He builds a GPT model on the works of Shakespeare (1MB), from scratch. After spending nearly 2 hours with him, Google Colab and PyTorch, I was left with a headache and some cuts and bruises. But I also had an insatiable desire to learn more. I have a long way to go. 

The way I learn is to fork away from just repeating what an instructor says and start adding my own challenges. I had an idea. I have done a lot of writing (many of you are victims to that) and much of that is on my blog site. What if I built a GPT based solely on the corpus of all my writing? Does that sound narcissistic a bit to you too? Oh well, for the good of science, we go in! Cue the Indy music. I extracted the text (468k). It’s not much, but why not? 

By the way, if you are still with me, I’ll try to go faster. You won’t want to hear about how I wasted so much time trying to use AMD GPUs (their ROCm software sucks, traveler beware), switched to CPUs, Nvidia CUDA and eventually Apple Silicon MPS (Metal Performance Shaders built in to the M1). All the while, I was using my fork of the code I built with Andrej Karpathy (ok, not him directly, but while watching his video). I started off with the simple Bigram NN Language model. And it is “Bi-Gram” not “Big RAM” but I found that to be ironically comical in a dad joke sort of way. 

My JasonGPT bigram.py started learning. It ran for 50,000 iterations and took about 8 hours. It even produced an output of random musings. While there was quite a bit of nonsensical output, I was amazed at how well this small run did at learning words, basic sentence structure and even picked up on my style. Here are some samples from the output I found interesting, comical and sometimes, well, spot on:

  • It’s a lot of time… But I think we also need science.
  • What are your big ideas?
  • Set our management to the adjacent ground (GND) pin.
  • I have a task to Disneyland out that this day.
  • I love the fun and fanciful moments as kids get to dream into their favorite characters, embrace the identity of their heroes, wrap themselves up starfish back.
  • Bring on the “power” of his accidental detail.
  • Your character provided faith, all kindness and don’t care.
  • Grab a difference too.
  • After several days of emailing, texting and calling, I received a text message.
  • Curl has the ability to provide timing data for DNS lookup, it will easily show or avoided.
  • Imperfect things with a positive ingredient can become a positive difference, just get that time.
  • I also believe we should exploit the fusion power that shows up each day in our company’s data.
  • Have you found a vulnerability? Are you concerned about some missing measures or designs that should be modernized or addressed? If so, don’t wait, raise those issues. Speak up and act. You can make a difference.
  • “I know what you are thinking.” the irony
  • We are the ones who make a brighter day.
  • The journey ahead is ahead.
  • What are you penning today? What adventures are you crafting by your doing? Get up, get moving… keep writing.

Look, it’s no ChatGPT, but it blew my mind! I’m only using a 4 layer NN with 7 million parameters. In comparison, ChatGPT uses 96 layers and 175 billion parameters! Before the weekend ended, I set up nanoGPT to build a more elaborate model on my data set. It’s still running, but already I can see it has learned a lot more of my style but seems to lack some focus on topics. It’s easily distracted and interrupts its own train of thoughts with new ideas. Squirrel! Nothing like me.

So my JasonGPT won’t be writing my Monday updates anytime soon, but who knows, maybe it will help me come up with some new ideas. I just hope it stays benevolent and kind. I would hate for it to suddenly become self-aware and start…

Connection to imac.local closed.
Posted on

Generative AI

Lightning across a digital eye of a typhoon

Typhoon warning! My nephew is a Lt. Commander in the US Navy currently stationed in Guam. He teaches and manages trauma and emergency care at the hospital. Last night, he was preparing his family for the typhoon that would be sweeping across the small Pacific island in just a few hours. They closed the storm shutters, stored their Jeep in the basement and ensure their backup power and pumps were working. My nephew drew the short straw at the hospital and will be managing the ER while the storm rolls through. I worried about the hospital being built for these type of events and he assured me that it was, but of course, he was quick to add that the generators were built by the lowest bidder.

There is another typhoon coming. Gazing out over the technology horizon we can see a storm forming. But this one seems to be more than heavy winds and rain. I’m talking about the recent astonishing developments in generative artificial intelligence (GAI). I’m increasingly convinced that we are sitting on the edge of another major tectonic shift that will radically reshape the landscape of our world. Anyone who has spent time exploring OpenAI’s ChatGPT or Dall-E, Google’s Bard, Microsoft’s Bing or Co-Pilot, Midjourney, or any of the hundreds of other generative AI tools out there, will immediately recognize the disruptive power that is beginning to emerge. It’s mind blowing. GAI’s capacity to review and create code, write narratives, empathetically listen and respond, generate poetry, transform art, teach and even persuade, seems to double every 48 hours. It even seems that our creation has modeled the creator so well that it even has the uncanny ability to hallucinate and confidently tell us lies. How very human.

I have never seen a technology grow this fast. I recall the internet in the late 1980’s and thinking it had the amazing potential as a communication platform. Little did I realize that it would also disrupt commerce, entertainment, finance, healthcare, manufacturing, education and logistics. It would create platforms for new businesses like the gig economy and provide whole new levels of automation and telemetry through IoT. But all of that took decades. Generative technology is announcing breakthrough improvements every week, sometimes every 48 hours. To be fair these large language models (LLMs) are all using decades old research in neural network (NN) technology. However, when you combine those NN with enhancements (e.g. newer transformers, diffusion algorithms), hardware (e.g. GPUs) and rich data sets (e.g. the internet) they unleash new capabilities we don’t even fully understand. The latest generations of the LLMs even appear to be doing some basic level reasoning, similar to how our own organic NNs help us solve problems.

Businesses are already starting to explore the use of this technology to increase productivity, improve quality and efficiency. Wendy’s recently announced that they are partnering with Google to use GAI to start taking food orders at their drive-throughs.1 Gannett, publisher of USA Today and other local papers, is using GAI to simplify routine tasks like cropping images and personalizing content.2 Pharmaceutical companies like Amgen are using GAI to design proteins for medicines.3 Autodesk is using GAI to design physical objects, optimizing design for reduced waste and material efficiency.4 Gartner identifies it as one of the most disruptive and rapidly evolving technologies they have ever seen.5 Goldman Sacks is predicting that GAI will drive a 7% increase in global GDP, translating to about $7 trillion!6

It’s time to prepare for the typhoon. I’m excited about the future! As a technologist, I know disruptions will come, challenging our thinking and changing how we work, live and play. I know it can also be terrifying. It can prompt fear, uncertainty and doubt. But now is the time to prepare! Don’t wait to be changed, be the change. Start exploring and learning. I have a feeling that this new technology will be a 10x amplifier for us. Let’s learn how we can use it, work with it and shape it to be the next technological propellent to fuel our journey to a greater tomorrow!

This blog text was 100% human generated but the image was created with OpenAI Dall-E2.

  1. Wendy’s testing AI chatbot that takes drive-thru orders. (2023, May 10). CBS News. https://www.cbsnews.com/news/wendys-testing-ai-chatbot-drive-thru-orders/
  2. Publishers Tout Generative AI Opportunities to Save and Make Money Amid Rough Media Market. (2023, March 26). Digiday. https://digiday.com/media/publishers-tout-generative-ai-opportunities-to-save-and-make-money-amid-rough-media-market/
  3. Mock, M. (2022, June 7). Generative biology: Designing biologic medicines with greater speed and success. Amgen. https://www.amgen.com/stories/2022/06/generative-biology–designing-biologics-with-greater-speed-and-success
  4. Autodesk. (2022, May 17). What is generative design? Autodesk Redshift. https://redshift.autodesk.com/articles/what-is-generative-design
  5. Gartner, Inc. (2022, December 8). 5 impactful technologies from the Gartner emerging technologies and trends impact radar for 2022. https://www.gartner.com/en/articles/5-impactful-technologies-from-the-gartner-emerging-technologies-and-trends-impact-radar-for-2022
  6. Goldman Sachs (2023, May 12). Generative AI could raise global GDP by 7%. https://www.goldmansachs.com/intelligence/pages/generative-ai-could-raise-global-gdp-by-7-percent.html
Posted on

The Best Pottery

It was the first day of the pottery class. The instructor welcomed the students and began to orient them on the material. He announced that the final grade would be determined by one of two measures. For half the class, he said that their final grade would be determined by the “quality” of their pottery. Their goal was to work on a single high quality product. For the other half of the class, he said that their final grade would be determined by “quantity”. Their goal was the sheer amount of pottery produced. Fifty pounds of pots would be rated an “A”, forty pounds a “B”, and so on. The class began and the students began their work.

The last day of class finally came and a curious fact emerged. The works of highest quality were not produced by the group focused on quality. Instead, the highest quality works were all produced by the group graded for quantity! It seemed that the “quantity” group got busy producing piles of work and learning from their mistakes as they went along. In contrast, the “quality” group sat around theorizing about perfection, and in the end had little to show for their work than some theory of perfection and a lump of dead clay.[1]

The key to becoming a great artist, writer, musician, etc., is to keep creating! Keep drawing, keep writing, keep playing! Quality emerges from the quantity. It strikes me that the same thing applies to software and systems we run. When we focus purely on the quality, we actually miss the mark. The way to improve quality is to keep creating, testing and learning. In the software sense, we want to keep releasing our code as often and as fast as possible. By doing that, we build operational expertise, knowledge and automation. We develop fast feedback loops that nudge the digital clay into a better shape. We tune processes to provide faster feedback loops, remove toil through automation, and minimize human error and mistakes. We optimize for a high throughput of working products and reap the prize of high quality outcomes.

But does this hold true? In my career, I have seen this to be true time and time again. Areas where we remove friction and optimize for faster release cycles (even multiple times a day), with automated integration, testing and delivery, ultimately result in higher quality products. I see the same thing looking out to the industry. The highest performing teams optimize for highest flow. The prize of perfection comes by delivering and learning. In the book, “Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations,” Dr. Nicole Forsgren, Jez Humble, and Gene Kim ran a multi-year research project looking at practices and capabilities of high-performing technology organizations. Their conclusion was that the highest performing organizations embraced the notion of continuous delivery, the ability to deliver changes frequently, reliably and with minimal manual effort.[2]

We ship! As technologist, software engineers and SREs, our teams help design, build and run the digital trains that deliver amazing products and experiences to our customers and fellow employees every single day. Our goal is to make these experiences shine! And, as the pottery class learned, it is quantity of our practice and continuous learning that makes them more perfect.

Keep shipping. Keep improving. Keep delivering!

References

  1. The pottery parable is a true story as captured by David Bayles and Ted Orland in their book, Art & Fear. There is a similar story about photography in James Clear’s book Atomic Habits.
  2. Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations by Dr. Nicole Forsgren, Jez Humble, and Gene Kim also identifies other key traits of high performing organizations, including having loosely coupled architecture, embracing a learning culture of experimentation, adopting lean principles to optimize flow, and creating a high-trust and empowering environment.
  • Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations. IT Revolution Press.
  • Bayles, D., & Orland, T. (1993). Art & Fear. The Image Continuum.
  • Clear, J. (2018). Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones. Avery.
Posted on

Moore’s Optimism

“In order to survive and win in the ever-changing world, keep updating yourself.” – Gordon Moore 

Gordon was born during the Great Depression. His dad was the local sheriff. They lived in the small farming and ranching town of Pescadero, California. He was a quiet kid, but he was optimistic and hopeful. He loved the great outdoors and would often go fishing or play at the Pescadero Creekside Barn. He also love science. His parents bought him a chemistry set on Christmas one year which eventually inspired him to pursue a degree in Chemistry. He earned a Bachelor of Science at UC Berkeley and went on to receive his PhD at Caltech.

After college, Gordon joined fellow Caltech alumni and co-inventor of the transistor, William Shockley, at Shockley Semiconductor Laboratory. Unfortunately, things didn’t go well there. Shockley was controlling and erratic as a manager. Gordon and most of the other top scientists left after a year and joined Sherman Fairchild to start a new company. At Fairchild Semiconductor, Gordon and his friend, Robert Noyce, help devise a commercially viable process to miniaturize and combine transistors to form whole circuits on a sliver of silicon. This led to the creation of the first monolithic integrated circuit, the IC.

Gordon and Robert eventually left Fairchild and decided to form their own company. They would focus on integrated circuit development so they named their company, Integrated Electronics. They started making memory chips and focused the company on high speed innovation. The company did extremely well at first but also faced some difficult times that required significant changes. All the while, Gordon focused on pushing things forward and taking risks. They had to constantly reinvent themselves to survive. The company was later renamed to something that you might be familiar with, Intel.

Gordon believed that the key to their success was staying on the cutting edge. That led to the creation of the Intel 4004, the first general purpose programmable processor on the market. Gordon had observed that the number of transistors embedded on the chip seemed to double every year. He projected that trend line out into the future and made a prediction that the number of transistors would double at regular intervals for the foreseeable future. This exponential explosion that Gordon predicted would power the impact, scale and possibilities of computing for the world for years to come. Of course, you know that famous prediction. It was later named after him, “Moore’s Law”.

In 1971, the first Intel 4004 processor held 2,300 transistors. As of this year, the Intel Sapphire Rapids Xeon processor contains over 44 billion. The explosion of capability powered by science continues to accelerate the technology that enhances and amplifies our daily lives. This past Friday, Gordon Moore passed away at his home in Hawaii, but the inspiration, prediction and boundless technical optimism that he started continues to live on.

I know there is a lot going on right now. We are facing uncertainty and considerable change. It can create fear and apprehension. Technology is constantly being disrupted as well as its role, and our roles, in applying it to our businesses. While not comfortable, we need to embrace the change. Lean in and learn. We need to constantly find new ways to reinvent ourselves and what we do. Embrace the exponential possibility of the future! We can do this!

Moore’s Law – By Max Roser, Hannah Ritchie – https://ourworldindata.org/uploads/2020/11/Transistor-Count-over-time.png, CC BY 4.0, https://commons.wikimedia.org/w/index.php?curid=98219918
Posted on

The Art of Removal

“The sculpture is already complete within the marble block before I start my work. It is already there, I just have to chisel away the superfluous material.” – Michelangelo

A tanker truck hauling 8,600 gallons of gasoline approached the MacArthur Maze, a large freeway interchange near the east end of the San Francisco, Oakland Bay Bridge in California. The driver, traveling faster than he should, lost control, hit the guardrail and overturned the load of highly flammable fuel. It spilled out on the interchange and exploded into a violent inferno, sending flames hundreds of feet into the air. The heat weakened the steel structure of the three-lane section of Interstate 580, causing the road to collapse onto Interstate 880 below. Thankfully, the driver survived and no other vehicles were involved in the accident. 

California Department of Transportation, Caltrans, rushed in to quickly assessed the damage of this crucial interchange which handles some 160,000 vehicles per day. It would take weeks to clear the debris and several months to repair. Initial cost projections reached $10 million with an impact cost of $90 million. Bidding for the job started immediately. Due to the urgency of restoring this vital link, the state offered an incentive of $200,000 per day bonus if the work was completed before the deadline.

Bidding started. C. C. Myers had been planning for this his whole life. While other contractors in the room were offering on-time proposals well over the $10 million estimate, C. C. Myers shocked the room. He would do the work for $878,075, promising to complete the work well ahead of schedule. This was not the first time C. C. Myers had taken on heroic work. His company had a proven track record of rebuilding damage freeways well ahead of schedule, including the Santa Monica Freeway after the 1994 Northridge earthquake. Needless to say, he won the bid.

C. C. Myers went to work. He had assembled a logistic transport team and forged agreements in Texas and other areas to expedite steel delivery to the interchange. He streamlined processes and cut away any distractions and superfluous procedures that didn’t directly contribute to safely delivering the roadway ahead of schedule. As an example, the typical inspection process requires steel workers to complete all their welds before scheduling government X-ray inspection. C. C. Myers convinced the government to embed X-ray technicians in his team and perform the test immediately after the weld was complete. This allowed the crew to get real-time feedback on any area that didn’t pass and fix it immediately before moving on. 

C. C. Myers’s efforts were successful. The monumental work was completed over a month ahead of schedule, right before a busy Memorial Day weekend. C. C. Myers earned a $5 million bonus for completing the work early. He quickly gave credit to his workers and their ability to deliver, but moving the mountain had required his artistry as well.

Like Michelangelo, C. C. Myers’s genius was his ability to stare into the mountain of “marble” and see what could be removed to reveal the ultimate outcome. Procedures and processes that didn’t directly deliver value were debris that had to be swept away. Every ounce of energy, every minute, and every movement was precious and deliberate. Everything that wasn’t part of the goal was chiseled away. 

What is the work and marble before you right now? What is the goal? What sculpture are you trying to reveal? What can you remove? As all you wonderful artists head into your work channel your inner Michelangelo. Chisel away the useless motion, process and procedures to reveal the incredible work of art buried in the marble.

Credit: A friend of mine, Paul Gaffney, spoke on this at the 2023 DevOps Enterprise Forum. His story was far more eloquent than my version. It motivated me to do more research on the incident. The result is this post. I’m indebted to Paul for his inspiration.

Posted on

Grid Bugs

Oh, no! We were several hours into a major system outage and there was still no clue as to what was broken. The webservers were running at full load and the applications were pumping a constant stream of error logs to disk. Systems and application engineers were frantically looking through the dizzying logs for clues as to the cause. Of course, looking at the logs, you would assume everything was broken, and it was. But even when the application worked, the logs were full of indecipherable errors. Everyone knew that most of the “errors” in the logs weren’t really errors, but untidy notices that developers had created long ago as part of a debugging exercise. As one engineer observed in some degree of frustration, “It’s like the log file that cried wolf!” After a while, nobody notices the errors.

The teams restarted services, rebooted systems, stopped and restarted load balancers. Nothing helped. Network engineers dug into the configuration of the routers and switches to make sure nothing was amiss. Except for the occasional keyboard typing sounds, dogs barking or children crying in the background, the intense investigation had produced an uncanny silence on the call. Operation center specialists were quickly crafting their communication updates and were discussing with the incident commander on how to update their many clients that were impacted by this outage. Company leaders and members of the board of directors were calling in to get updates. Stress was high. Would we ever find the cause or should we just shut down the company now and start over? Fatigue was setting in. Tempers were starting to show. Discussion ensued on the conference call to explore all mitigation options and next steps.

“I found it!” The discussion on the call stopped. Everyone perked up, anxious to hear the discovery. “What did you find?” the commander asked in a hopeful way. The giddy engineer took center stage on the call, eager to tell the news. “It’s the inventory service! The server at the fulfillment center seems to be intermittently timing out. Transactions are getting stuck in the queue.” The engineer paused, clearly typing away at some commands on his computer. “I think we have a routing problem. I try to trace it but it seems to bounce around and disappear. Sometimes it works, but to complete the transaction, multiple calls are required and too many of them are failing. I’m chatting with the fulfillment center and they report the inventory system is running.”

The engineer sent the traceroute to the network engineer who started investigating and then asked, “Can you send me the list of all the addresses used by the inventory system?”  After some back and forth, the conclusion came, “I found the problem! There are two paths to the fulfillment center, one of which goes through another datacenter. That datacenter link looks up but it is clearly not passing traffic.” After more typing, the conclusion, “Ah, it seems the telco made a routing change. I’m getting them to reverse it now.” Soon the change was reversed and transactions were flowing again. The dashboards cleared and “green” lights came back on. Everyone on the bridge quietly, and sometimes not so silently, celebrated and felt an incredible emotional relief. Sure, there would be more questions, incident review and learning, but solving the problem was exhilarating.

How many of you can relate to a story like that? How many of you have been on that call?

A friend of mine, Dr. Steven Spear at MIT, often reminds us that the key to solving a problem is seeing the problem. You can’t solve what you cannot see. A big part of reliability engineering and systems dynamics is understanding how we gain visibility into problems and surface them so they can be addressed. Ideally, we find those weaknesses before they cause real business impact. That is often the attraction of chaos engineering, poking at fault domains to expose fractures that could become outages. But sometimes the issue is so complex that we just need a clear line of sight into the problem. In the story above, connectivity and those dependent links were not clearly visible. If there was some way to measure the foundational connectivity between the dependent locations, our operational heroes could have quickly seen it, fixed it, and gone back to sleep. Getting that visibility in advanced is the right thing to do for our business, our customers and our teams.

This past weekend, I found myself itching to code and tinker around with some new tech. The story above is one I have seen repeated multiple times. We often have limited visibility into point-to-point connectivity across our networks and vendors. Yet we have this grid of dependency that is needed to deliver our business powering technology. I know what you are thinking. There are millions of tools that do that. I found some and they were very elaborate and complicated, way more than what I wanted to experiment with. I finally had my excuse to code. I wanted to build a system to synthetically monitor all these links. Think of an instance in one datacenter or cloud polling an instance in another datacenter or region. I had a few hours this weekend so I blasted out some code. I created a tiny multithreaded python webservice that polls a list of other nodes and builds a graph database it displays using the JavaScript visualization library, cytoscape ,which was fun learning by itself. Of course, I packed this all into a container and gave it the catchy name, “GridBug”. Yes, I know, I’m a nerd.

You can throw a GridBug onto any instance, into any datacenter, and it will go to work monitoring connectivity. I didn’t have time to test any serverless options but it should work as well. I set up 5 nodes in 3 locations for a test, with some forced failures to see how it would detect conditions on the grid. The graph data converges overtime so that every node can render the same graph. If you want to see it, here is my test and project code: https://github.com/jasonacox/gridbug

I have no expectations on this project. It is clearly just a work of fun I wanted to share with all of you, but it occurs to me that there is still a lesson here. Pain or necessity is a mighty force in terms of inspiration. What bugs you? Like this outage example, is there some pain point that you would love to see addressed? What’s keeping you from trying to fix it? Come up with a project and go to work on it. You are going to learn something! Look, let’s be real, my project here is elementary and buggy at best (sorry, couldn’t resist the pun), but I got a chance to learn something new and see a fun result. That’s what makes projects like this so rewarding. The journey is the point, and frankly, you might even end up with something that brings some value to the rest of our human family. Go create something new this week!

Have a great week!