SCORP – the well-architected tool for Architecture Reviews

As AWS evolves, it’s increasingly necessary to adhere to best practices to help you create high-performance, secure, and cost-effective workloads. However, it can be difficult for users to achieve all this and evolve with every new AWS service or update. We recently published an article – Part 1 of a two-part series – which introduces SCORP, the well-architected tool you need to implement the framework.

This article will dive deep into our SCORP well-architected tool linked to AWS’s best practices with step-by-step guidance in your architecture reviews.

How the process works

Now that we have established the need for a well-architected organisation and a lightweight or well-architected tool to drive improvements, let’s dive deep into how the SCORP process works.

In Figure 1, we model the process cycle, which we have based on two main cadences.

• Target a well-architected review (WAR) Benchmark each quarter
• Target a group SCORP Team Dashboard review each sprint.

As you may realise, the quarterly well-architected review (WAR) process is a standard review of the team’s workload with a Solutions Architect. It is aimed to be a thorough deep dive into each of the five pillars.

SCORP Review Process

We aim to facilitate more frequent reviews of the team’s operational metrics through our SCORP review process. We look for frequent cross-team collaboration to share experiences, practices, approaches and learnings concerning developer operations and production workload performance. “Learning from the mistakes of others.” But also learning from their successes, we can tell you there are many.

The rationale for the frequent 2-week cycle is:

• Why solve the same problems in every team? A problem shared is a problem halved. But creating situational awareness of Engineering Excellence across all teams is a potent force for good.
• Connecting developers and teams. We should work together to improve and gain more economies of scale. Teams should be helping each other improve.
• Maintain emphasis on the good fight; issues with the operation and performance need to surface regularly, and sometimes that means pushing back or creating space in the development backlog for improvement.
• Build the passion for excellence/fuel curiosity, get ideas out into the open, celebrate the small wins, and generate pride in a good job.
• You can use the process as an Alignment and Well Architected tool for architecture to remain engaged at the team level.

The SCORP Review Process Flow

With the solid rationale above for the 2-week collaboration cycle, we must make the process work. Here is a high-level description of the flow and how it works for us today:

• All ten teams are represented by the team’s Principal Engineer and Delivery Lead/Scrum Master.
• The Architect facilitates the session.
• If not already fully automated, then before the session, the Principal Engineer will prepare their team’s SCORP report/dashboard for review.
• The review lasts around 90 to 120 minutes. Each team takes 10 mins to discuss key trends, typically focusing on previous reports’ deltas.
• The Architect will go through any high-level ‘Notice To All Developers” (NOTADs). These are things like enterprise impacts, security mandates, pipeline changes, etc.
• We will review current DevOps actions for each team and get a quick summary of progress. If there are items that come up during the review that the team wants to research, they will follow up.
• For deeper dive topics that come up frequently (i.e., testing methodology, review of analytics tool setup and cfg), These get added to the portfolio summary. The Architect or Principal engineer will typically set up a future tech share on the topic.

The role of the well-architected facilitator

The facilitator has an important role. On a purely mechanical level, the facilitator is responsible for keeping the cycle on track, which is no mean feat when you have ten squads. It requires discipline, and you must use the process wisely, i.e., knowing what to pack and what to get into. The facilitator ideally is an architect with plenty of experience in DevOps and the well-architected tool and framework. Still, they also need to be in a position of authority and have some influence on the team’s ability to prioritise their work.

The rationale for the facilitator role

The rationale for this is the following:

• Does the facilitator have to ask questions? They should be comfortable directing the team to look at an area that is maybe not trending well, i.e., cost, performance response times, etc., but realistically with continuous improvement in mind.
• The facilitator should always be attempting to connect the engineers and teams. If one team has a fantastic BDD technique and another could benefit from it, then the facilitator should suggest pairing up.
• They should constantly be evolving the SCORP process. It has to be a value add for all the engineers and teams. If something is not working or adding value, the facilitator must address it.
• The facilitator should always attempt to generate interest in what the teams are trying to achieve, whether in the review or post-review with product owners or management.
• They will celebrate the successes and wins of each team with the group, no matter how small. Progress is progress! Teams and engineers should leave feeling like a million bucks when improving.
• They will facilitate a positive sharing environment where all voices get heard. Failures are never negative. They are an opportunity to learn and become better as we do!

It is a challenging role to fulfil, but it is a super important one to ensure the success of the process cycle.

SCORP Process Cycle Day Zero

Having tried to facilitate similar processes earlier in my career, my most significant learning was that “You need to meet the teams where they are at.” Some teams will have been together for an extended period and have great DevOps with high levels of automation and insight. Other teams will not; they may be new or, for one reason or another, might not have had the time, expertise or capacity to achieve the operational maturity of some of the more experienced teams around them.

You also need to ensure that management and the business know the well-architected tool initiative’s aims and goals. It is because it does require investment from the teams in terms of participation, but sometimes doing the right thing involves slowing down from time to time, which is sometimes not a straightforward conversation when it comes to meeting delivery timelines.

SCORP Docs

Before entering the cycle, we have several meet-ups with the Principal Engineers and Delivery Leads for our squads. It is crucial as it’s aimed at getting the teams and lead engineers to own the SCORP Review Cycle. They have to buy in and feel like they have a stake in the process. During this time, I worked with them to agree on the following things:

Ways of Working Agreement — Frequency of the cycle, Safe Space Commitment, attendance, length of meeting, etc. These are all the typical things you would tend to find in a working agreement.

The SCORP Report Template — The SCORP report template structure was the task that we, the group, spent the most time on. The SCORP Report contains all the critical operational metrics relating to team and workload performance. The insights to be collected and reviewed via this template must be influenceable, impactful to all teams, and structured around the five pillars of the Well-Architected Framework. Initially, we agreed that each team would collect these metrics using a Confluence wiki page. It might be pretty controversial, but we told the teams upfront that manually updating their wiki page with the key metrics every two weeks was part of a rather devious plan that annoys them enough to invest in automating these insights. Luckily, this has been the case.

A sample SCORP report/dashboard template for the well-architected tool

Wrapping Up

Becoming a high-performing team takes time and effort. It requires investment in your craft, learning from your successes and failures but also the successes and failures of others. The process will evolve, but that is what we expect as the software industry constantly evolves.  To borrow a term from Jeff Bezos, we are well on creating our own operational FlyWheel.

SCORP Operational Improvements:

Now that we’re regularly observing our trends, we are noticing some positive changes. We will endeavour to share moe quantifiable data in future write-ups, but in Qualitative terms, all teams involved in our SCORP Review Cycle have made significant operational improvements. We have summarised some of my favourites and most impactful ones below:

• The majority teams have produced automated dashboards (DataDog, Splunk, etc.) We are now tracking significantly more data points than three months ago.
• We have seen a significant focus on performance improvements. All squads have reduced response times through activities such as performance tuning.
• The collaboration around test automation has been phenomenal. Once we saw that integration testing was an issue across all teams, it became a safe topic of discussion, and the innovation began. BDD is now starting to take hold across the teams.
• Security is front and centre. We have seen much more investment in activities such as threat modelling and facilitating mitigations of identified threats.
• From an operational excellence perspective, we have had a sub-community form around observability and workload release confidence.
• The progress is clear to see in subsequent WAR reviews.

As an architect, I couldn’t be happier or prouder of the team’s contributions. We are well on our way to meeting the goals of my original hypothesis. I have never been in any doubt about what we could begin to achieve through a bit of investment in continuous improvement, but seeing it, in reality, is quite rewarding. I recommend that any technical leader implement an SCORP review Process Cycle based on a Well-Architected tool to drive engineering excellence within their organisation.