The Art of Scalability

 

The second edition of the Art of Scalability is my book this week. It is coauthored by Martin Abbott and Michael Fisher. As its subtitle suggests, the book is about building scalable web architecture, processes and organisations for the modern enterprise.

In this book, the authors argue that the three key components of that are people, process and technology. In the introduction video, the author talked about how they thought initially that technology was the key, only to realise that people and process are no less important based on their consulting experience. These three components are covered in the first three parts of the book. More details on that to follow.

Before getting into the details, I share with you what I like and do not like about this book. Its content is vast, fascinatingly relevant, and not dry at all. It is engaging enough that I have had no trouble enjoying many chapters from around 3am to dawn nearly all days this week. Just to abandon this book and pick up another one is a very trivial action on my part. But I did not. The discussions, technical or not, are very plainly written. It opens up my view on how to scale. The quantitative approaches towards project management and scalability topics are straightforward. The main negative attribute of this book is repetition. It could be shortened significantly. That said, repeating concepts covered in previous chapters certainly helps to refresh the reader’s memory and improve the understanding of the topic currently under discussion. It could, therefore, be the intention of the authors.

Do I recommend it? Yes. If you do not have a large chunk of time to pursue such a big book, browsing the conclusion and key points sections of each chapter on safari books online can give you a quick overview of each chapter. The figures, tables, equations etc. are all beautifully presented online too.

 

Staffing a scalable organisation

In this part, the book discusses the necessary roles and their corresponding responsibilities in a scalable technology organisation. The lack of clearly defined roles or those with overlapping responsibilities can cause confusion and conflict.

The book then progresses to talk about the two key attributes of organisations: size and structure. Both can affect the communication, efficiency, quality and scalability of the organisation. The two traditional structures are functional and matrix. The third one, agile, is gaining traction for its increased innovation, measured by shorter time to market, quality of features and availability of services. There are pros and cons for both large or small teams. It is important to be aware of the specific pitfalls of each, know where your team is, take necessary steps to mitigate the negative effects of the team size.

Further, the book presents us Leadership 101 and Management 101. I like the guidance on goal setting for leaders. The goals should be SMART: Specific, Measurable, Attainable (but aggressive), Realistic and Timely (or containing a component of time). One piece of advice stands out for me in the Management 101: “spend only 5% of your project management time creating detailed project plans and 95% of your time developing the contingencies to those plans. Focus on achieving results in an appropriate time frame, rather than laboring to fit activities to the initial plan.” When it comes people management, the analogy of gardening is interesting: seeding (as of hiring), feeding (as of developing people), weeding (as of elimination of underperforming people within an organisation).

   

Building Processes for Scale

The second part of the book covers processes. The general idea is to create the right set of processes to standardize the steps taken to perform certain tasks, eliminate confusion and unnecessary decision making, and hence free up the employees to focus on important work. The authors use the following figure to illustrate the different levels of process complexity.

In this part, the authors discusses processes answering these questions:

  • How to properly control and identify change in a production environment?
  • What to do when things go wrong or when a crisis occurs?
  • How to design scalability into your products from the beginning?
  • How to understand and manage risk?
  • When to build and when to buy?
  • How to determine the amount of scale in your systems?
  • When to go forward with a release and when to wait?
  • When to roll back and how to prepare for that eventuality?

One chapter talks about headroom calculation. The authors advise to use 50% as the amount of maximum capacity whose use should be planned. Naturally we all know a discount factor should be used in estimating headroom, but the value to use for discounting is mostly informed from experience. This shows one great benefit of reading this book: informing me of what the authors summarised from their combined decades of experience of helping to scale businesses.

 

Architecting Scalable Solutions

The third part of this book discusses about the differences of implementation and architecture, how to create fault-isolative architectures, the AKF scale cube, caching and asynchronous design for scale. The AKF scale cube method suggests scaling along three dimensions: cloning the entities or data and distributing unbiased work across workers, separation of work biased by activity or data, separation of work biased by the requestor for whom the work is being performed. For illustration purpose, I cite the AKF scale figure from the book below.

The first two dimensions of the AKF scale cube approach are very similar to our scalability studies for Exascale Computing: providing more compute nodes and duplicating data and code on each of them to perform a chunk of work (that can equally be performed on other node), partitioning and assigning a specific piece of work to its most suitable compute node in a heterogeneous environment or partitioning the data among a set of nodes and sending the corresponding compute to each node. The third dimension is to direct the service requests to different subset of nodes, based on the info available about the requests or requesters. The authors point out often these are nested together.

The last part of the book covers the issue of having too much data, grid and cloud computing, monitoring applications and planning data centers. Not to miss the appendices, the examples given there are very illustrative on availability, capacity planning, load and performance calculation.

There is a set of slides from Lorenzo Alberton available on slideshare, talking about the key concepts from this book.

Overall, I enjoyed learning about scalability and how to build scalable architecture through reading this book. It is thanks to reading books like this that the darkness of winter days is slightly more bearable than it would be.

The Art of Possibility

A great lecturer and mentor of mine, John Steinhart, recently recommended the book The Art of Possibility. John specifically mentioned its audio recording. Although I have a reasonably lengthy list of books to read already, a recommendation from John no doubt sets me into motion to check both the audiobook and paperback out. The Art of Possibility, written by Rosamund Stone Zander and Benjamin Zander, is my book of this week. The audiobook is also read by both authors. It is fascinating that the audiobook includes many pieces of music that were intimately relevant to the context. I am very fond of classical music, even more so when the music is intertwined with the stories and discussions in this book. I am grateful to John for suggesting this book.

It takes certain mindset to settle into this book. The shift from last week’s Information Retrieval to this was not a comfortable one. My very analytical mind initially responded quite badly to the vagueness of its writing and its light-weight philosophical discussions. I was constantly battling with my inner voice: Why is this the case? How did we derive this point? Is this a single instance? Do we have a sufficiently large data set to draw this kind of insights/conclusions? How do we know that we have attributed to the right causes for the effect observed? Then suddenly one sentence opened the door for me to enter this book: “do not take yourself so goddamn seriously.” Did not Oscar Wilde have a similar line: “Life is too important to be taken seriously”? It reminded me also of a piece of advice that my friend Jay Owen gifted me last year: “do not take yourself too seriously”. That sentence was very powerful. In this particular circumstance, I told my inner self off immediately, “Just shut up! Be open-minded and see what the authors have to say!” I subsequently experienced the wonder of this practice. I was curious enough and wanting to learn all the rest of the practices covered in the book such that I started again from the beginning.

This book is about possibility. The message resonates with what I learned some time ago that I am only limited by my own thinking. As the authors put it: much, much more is possible than people ordinarily think. The authors wrote this book with the objective to provide us the means to lift off from the world of struggle and sail into a vast universe of possibility.

“Our premise is that many of the circumstances that seem to block us in our daily lives may only appear to do so based on a framework of assumptions we carry with us. Draw a different frame around the same set of circumstances and new pathways come into view. Find the right framework and extraordinary accomplishment becomes an everyday experience. Each chapter of this book presents a different facet of this approach and describes a new practice for bringing possibility to life.”

Here is a short summary of a selectively few out of the 12 practices included in the book. I include the steps to get there from the book too. Some are direct quotes. Some are paraphrased by me. Purely for readability purpose, I do not use italic font to mark the quoted phrases or passages, but I happily acknowledge that all messages below are read or learned from the book.

  • It’s All Invented. Ask these questions: what assumptions am I making, that I am not aware I’m making, that gives me what I see? And ask: what might I now invent, that I haven’t yet invented, that would give me other choices?
  • Stepping into a Universe of Possibility. How are my thoughts and actions, in this moment, reflections of the measurement world? You look for thoughts and actions that reflect survival and scarcity, comparison and competition, attachment and anxiety. Recognising that your measurement mind is at work, you ask again: How are my thoughts and actions, in this new moment, a reflection of the measurement world? And how now?
  • Being a Contribution. Life is a place to contribute and we as contributors. Unlike success and failure, contribution has no other side. It is not arrived at by comparison. How will I contribute today? Declare yourself to be a contribution. Throw yourself into life as someone who makes a difference, accepting that you may not understand how or why.
  • Lighting a Spark. Ben told a story that his father said “Certain things in life are better done in person”, when Ben asked him why not making a phone call instead of making a train journey. That answer bewildered Ben in a wonderful way. Many years later, Ben made a day trip by air to persuade the world’s greatest cellist Mstislav Rostropovich to play in a concert. Rostropovich agreed to play. To light a spark, the authors suggest to practice enrollment: imagine that people are an invitation for enrollment, stand ready to participate, willing to be moved and inspired, offer that which lights you up, have no doubt that others are eager to catch the spark. It is similar to the “yes, and” practice in improv.

I would like to highlight a few passages that relate the practices in this book to a much broad world.

When one person peels away layers of opinion, entitlement, pride, and inflated self-description, others instantly feel the connection. As one person has the grace to practice the secret of Rule Number 6 (do not take yourself seriously), others often follow.

I am the framework for everything that happens in my life….If I cannot be present without resistance to the way things are and act effectively, if I feel myself to be wronged, a loser, or a victim, I will tell myself that some assumption I have made is the source of my difficulty.

The foremost challenge for leaders today, we suggest, is to maintain the clarity to stand confidently in the abundant universe of possibility, no matter how fierce the competition, no matter how stark the necessity to go for the short-term goal, no matter how fearful people are, and no matter how urgently the wolf may appear to howl at the door. It is to have the courage and persistence to distinguish the downward spiral from the radiant realm of possibility in the face of any challenge.

The term mission statement is often used interchangeably with the word vision in business and political arenas but, by and large, mission statements are expressions of competition and scarcity…A vision releases us from the weight and confusion of local problems and concerns, and allows us to see the long clear line. A vision becomes a framework for possibility when it meets certain criteria that distinguish it from the objectives of the downward spiral.

 

The book has a list of criteria as what is a vision in the universe of possibility, which I do not list here for the sake of brevity. That said, I do think they are very relevant to any organisation.

After reading this book, I understand why John recommended this book. The views and methods advocated here can be very powerful in searching for good solutions to resolve conflicts and even better in transforming the conflicts to profoundly rewarding experiences.