DevOps Enterprise Summit – London 2016

I once again had the privilege of attending the DevOps Enterprise Summit.  This time it was in the U.K. at the Hilton Metropole.  I was impressed with the representation and talks from companies and organization across the UK and the rest of Europe:  SAP, ITV, Hiscox, ING, Barclays, HMRC, Zurich, and many more.

Themes that I picked up from these DevOps leaders:

  • People – Its all about People – empathy, org change, transformation
  • Speed – Continuous Integration and Delivery
  • Quality – Investment in DevOps practices often results in higher quality output
  • Agility – Microservices and Flexible Infrastructure
  • Security – Everyone’s responsibility
  • Business – Focus on Product vs. Project with integration with business in transformation (BizDevOps?)

I was honored to speak again and talk about our DevOps journey at Disney.

Jason Cox DOES16 London

Even though I wasn’t able to record my presentation, ComputerWorld UK provided a great write-up of my talk, and even gave me a new title! 🙂

There was considerable interest in our journey to DevOps, especially our transition from Operation Specialist to embedded Systems Engineers.

Other Quotes

“If technology is done well it looks like magic”

https://twitter.com/tumble_b/status/748460191935500288

References

Systems strategy chief Jason Cox details Disney’s devops journey – ComputerWorld UK

Tips for DevOps Success from DOES 2016 – ComputerWorld UK

DevOps Across the Pond – London Reprise – ITproPortal

Overcoming the scale-up challenge of enterprise DevOps adoption – ComputerWeekly.com

 

The Art of Business Value

art-of-biz-valueA review of The Art of Business Value by Mark Schwartz

Business value, the north star of Agile, Lean and DevOps champions, is often more difficult to determine than one would imagine. This book takes the reader on a journey to discover that value while along the way, creating helpful mental models, challenging preconceived notions and proposing some creative ways to transform an organization.

The author does a great treatment of ROI (Return on Investment – does anyone ever really measure that?), NPV (Net Present Value), MVA (Market Value Added) and SVA (Shareholder Value Added). I especially enjoyed the humorous practical example that revealed a surprising truth that acquiring an MBA can easily have a negative NPV (in other words, a bad investment!) So save your money and pay attention to the author’s “four-paragraph MBA” (p. 20) that unpacks the main two principles learned in an MBA: 1) There is time value of money and 2) A business venture needs a sustainable competitive advantage.

Legacy IT architecture and bureaucracy are often considered negatives and obstacles to progressive competitive relevance but the author brings a refreshing perspective on the “value” of that complex hairball of legacy and rules and the right way to polish and transform them.

On Agile…

“The purpose of an Agile team is to self-organize and meet the underlying business need in the best way possible, often by cutting through the bureaucracy.” (p 55)

On DevOps…

“The DevOps model…looks to break down the silos that have resulted from technical specialization over the last few decades. But the DevOps spirit goes further looking to eliminate the conflicting incentives of organizational silos and the inhumane behaviors that can result from those conflicting incentives.” (p 48)

On Bureaucracy…

“Bureaucracy delivers business value. Just sometimes not enough.” (p 59)
…”developers are bureaucrats by nature. We have a tendency to solve problems by creating standard processes rather than by relying on human judgment.” (p 54)
“The pipeline is an automated bureaucracy: it applies its rules in a rigorous, unemotional way, sine ira et studio. That does not mean that the software development process is unemotional; it means that the tools are unemotional and the passion is brought to the process by the people.” (p 105)

And on Business Value…

“Business value is a hypothesis held by the organization’s leadership as to what will best accomplish the organization’s ultimate goals or desired outcomes.” (p 90)

I highly recommend this book to IT leaders, digital executives, strategic managers, and anyone seeking to make their organization more agile, effective, relevant, competitive and humane.

Docker + Mesos + Marathon in AWS EC2

I wanted to see if I could get a Docker + Mesos + Marathon platform up and running quickly in AWS EC2 using t2.micro instances.  I found this great article by Gar (@gargar454) where he has put all the components in docker containers and provides a simple  tutorial which I paste below with some minor edits:

Create a Docker Host on EC2

Create your EC2 instances (Amazon AMI t2.micro will work), set it up in your private VPC and auto-assign a public IP so you can test.  You will need to open TCP ports 5050 and 8080 in  your security group from your workstation if you want to see the Mesos and Marathon UIs.   Run these commands to set up a docker host:

sudo yum update -y
sudo yum install -y docker
sudo service docker start
# grant ec2-user ability to run docker commands without sudo
sudo usermod -a -G docker ec2-user
# you must exist to refresh user rights
exit
  1. Export out the local host’s IP
    • HOST_IP=`wget -qO- http://instance-data/latest/meta-data/local-ipv4`
  2. Start Zookeeper
    docker run -d \
    -p 2181:2181 \
    -p 2888:2888 \
    -p 3888:3888 \
    garland/zookeeper
  3. Start Mesos Master
    docker run --net="host" \
    -p 5050:5050 \
    -e "MESOS_HOSTNAME=${HOST_IP}" \
    -e "MESOS_IP=${HOST_IP}" \
    -e "MESOS_ZK=zk://${HOST_IP}:2181/mesos" \
    -e "MESOS_PORT=5050" \
    -e "MESOS_LOG_DIR=/var/log/mesos" \
    -e "MESOS_QUORUM=1" \
    -e "MESOS_REGISTRY=in_memory" \
    -e "MESOS_WORK_DIR=/var/lib/mesos" \
    -d \
    garland/mesosphere-docker-mesos-master
  4. Start Marathon
    docker run \
    -d \
    -p 8080:8080 \
    garland/mesosphere-docker-marathon --master zk://${HOST_IP}:2181/mesos --zk zk://${HOST_IP}:2181/marathon
  5. Start Mesos Slave in a container
    docker run -d \
    --name mesos_slave_1 \
    --entrypoint="mesos-slave" \
    -e "MESOS_MASTER=zk://${HOST_IP}:2181/mesos" \
    -e "MESOS_LOG_DIR=/var/log/mesos" \
    -e "MESOS_LOGGING_LEVEL=INFO" \
    garland/mesosphere-docker-mesos-master:latest
  6. Goto the Mesos & Marathon Web pages
    # You can find your EC2 instance public IP address with:
    wget -qO- http://instance-data/latest/meta-data/public-ipv4

    Mesos Web Page

  7. http://<public-ip>:5050

Marathon Web Page

http://<public-ip>:8080

Create an App (+ New App button) on the Marathon page to see the task get assigned to a Mesos slave and executed.

Marathon New App - test1

Log in to the slave container and watch the file grow:

docker exec -it mesos_slave_1 /bin/bash

tail -f /tmp/running.out

DevOps Enterprise Summit 2015

It was great to attend the DevOps Enterprise Summit again this year. The 2015 edition saw more than double the number of attendees of the 2014 conference with presentations from companies all over the world. There is definitely a feeling that DevOps is awakening across the enterprise.

I had the privilege of presenting again on Disney’s Journey,
“Disney DevOps – The Enterprise Awakens.”

Fellow DevOps Avengers from all over the world converged to swap stories, share new insight, technology and encourage each other to keep moving forward as the Force of positive change in our various industries.

DevOps Enterprise | The Agile, Continuous Delivery and DevOps Transformation Summit http://devopsenterprise.io

Some great reviews and observations:

Impressions from DevOps Enterprise Summit 2015

Gene Kim and Others Share What DevOps is Really “All About”

Impressions from DevOps Enterprise Summit 2015 – Micro Hering – Accenture

Infoworld – Gene Kim explains the joy of DevOps

Selection of favorite quotes:

  • “If you name your servers and treat them like pets they all develop individual personalities.” – @jasonacox
  • “Times and conditions change so rapidly that we must keep our aim constantly focused on the future.” – Walt Disney
  • “If you don’t know where you’re going, any road will take you there.” –@jasonacox
  • “Believe in what you do, and make a difference” – @jasonacox

DevOps Enterprise Summit 2014

I had the privilege of speaking at the first annual DevOps Enterprise Summit.

Jason Cox - DOES 2014DevOps Enterprise | The Agile, Continuous Delivery and DevOps Transformation Summit http://devopsenterprise.io

Wired Magazine – Web Insights
DevOps Innovation Takes Place in the Enterprise by Steve Brodie

Redhat Developer Blog
DevOps Enterprise Conference — Day One

Electric-Cloud
DevOps Enterprise Summit Puts Agile Transformation Front and Center
What we learned at DevOps Enterprise Summit 2014

Software Development Times
DevOps Enterprise Summit bashes silos

PuppetLabs Blog
Disney’s DevOps Journey: A DevOps Enterprise Summit Reprise

Tweets
http://seen.co/event/devops-enterprise-the-agile-continuous-delivery-and-devops-transformation-summit-san-francisco-airport-marriott-waterfront-2014-949/highlight/38769

Selection of quotes:

  • “You’ve got to pick technology that transforms. You need to find tools that change the way you think.” – Jason Cox from Disney
  • “It’s all about the people! We have to empower the edge.” – Jason Cox from Disney at #DOES14
  • #DOES14 Jason Cox of Disney: Remove waste. (Motion != Work)
  • Jason Cox #DevOps guy at @disney promotes positive rebellion. Red tape removers! #DOES14
  • “DevOps professionals must be courageous. They must have the candor to say that there are issues and be renegades enough to constantly challenge the status quo.” – Jason Cox

 

Creativity, Inc.

creativity-incCreativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration Hardcover

by Ed Catmull and Amy Wallace

 

The Phoenix Project

phoenixprojectThe Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win

by Gene Kim, Kevin Behr, & George Spafford

Digital transformation has lit a fire that is burning through traditional IT shops across the world telling us to adapt or be reduced to the ash heap of obsolescence. The Phoenix Project grapples with this disruptive change by telling an engaging story about an IT manager who is thrust into this fiery turmoil.  The main character is ushered unwillingly and unprepared into a new leadership role where he uncovers the complex and unrelenting problems any IT shop knows all too well.  Fair warning: If you have any history working in, leading or managing IT teams, you will likely have a visceral reaction to the narrative and even suspect the authors were spying on your organization.  But there is hope!  The main character in the story finds enlightenment and begins to implement changes based on the “Three Ways” discussed in this book that ultimately transforms the teams, balances life for the employees and successfully propels the business forward.

IT leaders will find the narrative and advice relevant, potent and inspiring.  By applying the principles learned in this novel, your technology organization, like the phoenix of old, can rise from the ashes to take on the challenges of our new Digital age.


There is hope!  Technology workers and leaders will find the narrative and advice woven throughout The Phoenix Project, relevant, visceral, potent and inspiring.  By applying the principles learned in this novel, business and technology leaders will see how they can transform their own organizations, identify and break down unnecessary silos, improve life for technology workers, and successfully propel their businesses forward in this new Digital age.


 

Raspberry Pi – AirPiano

Wouldn’t it be cool if you could control your MIDI equipped digital piano from your iPhone?  Imagine a player piano capable of jukebox queuing up MIDI or Audio files and playing them in order.  That was my goal!  RPI to the rescue.

RaspiModelB

Required Ingredients:

  • Raspberry Pi – Model B, 512MB RAM, 16GB SD Card, Raspbian Linux, Apple 12W USB Power Adapter, USB WiFi Dongle
  • USB MIDI Digital Device (e.g. Yamaha Clavinova CLP-440 with USB MIDI port and RCA L+R Audio Inputs)
  • Software:  Apache HTTP Server, PHP, MySQL, (Nice to have:  Netatalk file server, Shairport AirPlay server)

Instructions:

My original thought was to turn the Yamaha speakers into a AirPlay device for our many iOS devices.  It’s nice to pipe in background tunes without having to hook up a device.  AirPlay is great way to do that but I didn’t want to spend $100+ to be able to do that.  The Raspberry Pi is more than capable of doing this. I found an extremely helpful blog post here:  http://www.raywenderlich.com/44918/raspberry-pi-airplay-tutorial

NOTE: Power is a big issue for the Raspberry Pi.  If you are using a WiFi dongle or any other USB device, I recommend making sure your power supply has enough kick to keep the RPI going.   I started out with a microUSB adapter that advertised 700mA but performance became very unstable, especially under load.  I switched to a 12W Apple adapter that I had handy and the problems disappeared.   The RPI FAQ recommends 1.2A (1200mA).

Setup for AirPiano – MIDI Control

The Project Code: https://github.com/jasonacox/raspberrypi

  • MySQL: Database ‘piano’ – see piano.sql file
  • Dequeue Service: Run the cron.sh script to have the RPI scan for new midi or wave files to play. Run it with:
bash -x cron.sh 0<&- 1>/dev/null 2>/dev/null &
  • Apache: Install apache http with mod_php and mysql support
  • Website Code: Install index.php, setup.php and the folder.png files into the document root of your webserver. Upload the MIDI (*.mid) and WAVE (*.wav) files to this location. Be sure to update $globalBase in setup.php to the folder where these files are located.

 

AirPlay via Raspberry Pi

Adam Burkepile has the best tutorial I’ve found on how to set up a RPI for AirPlay:

http://www.raywenderlich.com/44918/raspberry-pi-airplay-tutorial

Apple Share (AFP) Server via Raspberry Pi

The RPI is great for a tiny file server!  At the least, having it run an AFP server will allow quick drag and drop transfer from your Macs.  A Samba service for Windows shares is easy to set up as well.

sudo apt-get install netatalk

Yes, it is that easy.  Finder will now show your Raspberry Pi under “shared” along with any other local network shares:

Screen Shot 2013-12-27 at 11.22.08 PM

 

 

 

 

iCurlHTTP – cURL for iPhone and iPad

iCurlHTTP

HTTP Server Response Diagnostic Tool

I love cURL.  Systems engineers, developers, administrators, QA engineers and web operations specialist everywhere have used the curl tool to help troubleshoot web server responses.  It gives a quick view into exactly what is being sent from a web server to a browser.

On a commute home from work I started to think how handy it would be if the small computer I was holding, my iPhone, had a simple to use app like curl to do the same troubleshooting.  I always wanted to write my own iPhone app so I started researching how I could do that.

My Xcode Adventure

I fired up Xcode on my Snow Leopard equipped MacBook Pro for the first time.  I found a few great iOS development books on SafariOnline and started learning Objective-C.  After a few chapters I started going off the page and adding views and controllers.  I wanted to deploy my test apps to my iPhone so that required that I sign up as an Apple Developer (and $100!).  I soon discovered that my Xcode version was too out of date to work with my iOS 6.1 devices so I attempted to upgrade, only to discover that I needed to upgrade my OS.   Now on to Xcode 4.6 and suddenly my projects were running on my iPhone and iPad.

My libcurl Adventure

I discovered the NSURL class and started trying to get raw responses from web servers to display to the textview.  It was too digested and would require a lot of effort to simulate a raw curl session.  I started looking at the NSStream class when a friend suggested that I investigate a precompiled version of libcurl (the cURL library from Daniel Stenberg) for the ARM processor.  Instead, I found a great post by Nick Zitzmann where he shows how he compiled libcurl and even provides a binary: http://seiryu.home.comcast.net/~seiryu/libcurl-ios.html

I managed to get the latest version of libcurl to compile and also included the Apple SecureTransport library (–with-darwinsll in libcurl config):

#!/bin/bash
cd curl-7.29.0
export IPHONEOS_DEPLOYMENT_TARGET="4.3"
export CC="/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/llvm-gcc-4.2"
export CFLAGS="-arch armv7 -arch armv7s -pipe -Os -gdwarf-2 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS6.1.sdk"
export LDFLAGS="-arch armv7 -arch armv7s -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS6.1.sdk"
./configure --disable-shared --enable-static --with-darwinssl --disable-dependency-tracking --host="armv7-apple-darwin" --prefix=~/Code/curl/device
make -j `sysctl -n hw.logicalcpu_max`
make install

I added the libcurl.a library to my Xcode project and started using the easy interface API via CURLcode curl_easy_perform(CURL * handle ). The realtime feel of iCurlHTTP comes from the use of the CURLOPT_WRITEFUNCTION option to push realtime code into a view controller method that renders to the textview object. It required  periodically having the thread give up time to the mainRunLoop (via message to NSRunLoop) in order to allow the display to render and respond to user events.

iCurlHTTP v1.0 Born

iCurlHTTP was born.  It was a simple, easy to use iOS app that allowed you to run simple cURL like tests, simulating different web browsers to retrive the raw HTTP headers and HTML response from the web servers.

icurlhttp-v1.0

OpenSSL Anyone?  iCurlHTTP v1.1

The SecureTransport library works great but provided less details on SSL connections than what I was accustomed to seeing with curl.  I wanted more x509 decoded information, specifcially SSL certificate details about expiration date.  The OpenSSL library would allow for this so I began another search to get  OpenSSL library for ARM7.  To get what I wanted, I cross-compiled both OpenSSL and libcurl together. With that I was able to get the great openssl detail in the output, including cert info:

icurlhttp-cert

I managed to cross compile libcurl and openssl separately for armv7, armv7s and i386 architectures but later found a great GitHub project script created by Miyabi Kazamatsuri that does both.  I forked his code and updated it to work with Xcode 5 (clang for iPhoneOS and iPhoneSimulator) and added the 64-bit arm64 and x86_64 architectures.  The script is available on my GitHub:  https://github.com/jasonacox/cURL-SSL-for-iOS

UPDATE: I later put together a much better build script for openssl+libcurl that creates libraries for Mac, iOS and tvOS (Apple TV) targets: https://github.com/jasonacox/Build-OpenSSL-cURL

HTTP Timing and iOS 7 support added to iCurlHTTP v1.2

Curl has the ability to provide timing data for DNS lookup, TCP Connect, SSL Handshake, First Byte and Total time.  For example…

curl -k -s -o /dev/null -w '%{time_namelookup} %{time_connect} %{time_appconnect} %{time_pretransfer} %{time_redirect} %{time_starttransfer} %{time_total}n' https://jasonacox.com

I wanted to add this to iCurlHTTP to give a mobile perspective on these numbers.  In v1.2, I added this in the detail response view and as timing data points listed along the bottom of the UI.

icurlhttp-timing3

The v1.2 version also includes the iOS 7 SDK updates and the new minimalist UI look and feel.

iCurlHTTP 1.3 Wish List 

Thanks to great feedback, I captured several feature requests from the iCurlHTTP user base that became the goal for version 1.3:

  1. Custom Headers – Allow more and customized browser emulations via a customized option.
  2. Print, Copy or Email Output – Add a one-button feature to send the output to the clipboard, printer, email, post, etc.
  3. Customized Post – Allow customized post data to simulate a POST form submit.

This meant I needed to be able to have a User Settings pop-up screen to allow users to enter custom fields for headers and posts.  I also wanted to allow the ability to have a custom user-agent, http authentication (user/password), SSL options and possibly others in the future.

I knew I wanted a table based form to allow for formatting, scrolling and auto-positioning.  I considered using the Settings bundle but I needed multi-line UITextView input (especially for custom headers).  There are several great open source solutions  but my favorite version was FXForms, written by Nick Lockwood  (https://github.com/jasonacox/FXForms).  I used a simple plist for settings persistence.

icurlhttp-settings

The following features are added in version 1.3:

  • Share Feature – Send output to Clipboard, Printer and Email
  • User Settings – Set Custom User-Agent, Custom Headers (replace defaults or add new ones), POST Data, Authentication and SSL Mode selection
  • Updates – New libcurl and openssl libraries.

iCurlHTTP 1.4 Enhanced History

The goal of the 1.4 update was to enhance the URL dropdown history by adding HEADER and POST data for easy repeated testing.  I used simple (h) and (p) symbols to denote header and post respectively:

Screen Shot 2016-03-05 at 8.53.39 AM

iCurlHTTP 1.5 Redirect

I had several requests to add a “follow redirect” features (similar to curl -L command) to make it easier for users to test.  I didn’t want this to be purely  automatic because it would defeat the purpose of the granular detail that cURL provides so I present the user with a popup dialog when a 301 or 302 redirect response is received and list the redirection URL they can follow.  In this version, I also replace the ugly text “User” (settings) and “Share” buttons with the well known icons to help make it more intuitive.

Screen Shot 2016-03-05 at 9.30.53 AM

I’m amazed at how this small and simple project continues to grow (both in users as well as feature requests).  It has been a fun adventure! I frequently recommend other try building their own apps.  iCurlHTTP is written in Objective-C but I would definitely recommend trying out Swift.  I’m in the process of porting iCurlHTTP to the Apple TV (seems ridiculous, I know but I have to do it just for fun) and hope to try my hand at Swift.

iCurlHTTP 1.6 & 1.7 – HTTP/2.0 and IPv6 Update

For those wanting the libcurl, openssl and nghttp2 (HTTP2) libraries for iOS or tvOS (AppleTV), I have posted a build script on github: https://github.com/jasonacox/Build-OpenSSL-cURL

Thanks to request from users, I added the following features:

  • Certificate Chain Details for HTTPS Sessions (Detail Mode)
  • Support for Authentication Credentials in URL (e.g. https://user:pass@jasonacox.com/gettest.php)
  • DNS Resolve Spoof Option for Manual Address Resolution (eg. HOST:PORT:ADDRESS)

iCurlHTTP 1.8 & 1.9 – Performance Improvements

  • Performance Improvements and Reported Bug Fixes
  • Added user defined DNS lookup & connection timeout setting (default 5s)

THANK YOU!

Thanks for your great feedback and support!  I really appreciate the kind words and critical advice.  Please send more!

iCurlHTTP Website:  www.jasonacox.com/icurlhttp

Download on the App Store

 

Reference and Thanks!