Network and DevOps - Our Beginnings

In a previous post I mentioned how Network needed to adopt DevOps principles in order to achieve the speed and agility required to meet the business needs (As a reminder, that post is here).  What does that really mean and how does a company start that journey?  I will detail how we are beginning this shift.

The Beginning

We initially started by providing an opportunity for an existing network person to become a network/DevOps person.  While this started off great I feel it began to become more and more challenging as the distance between network knowledge, coding knowledge and the target environment knowledge (target network platform) was too great. There was strong network knowledge, some good coding knowledge but knowledge of the target environment was very minimal.  This set up a frustrating scenario for all parties involved.  What it did reveal is that the role needs to be:

• Strong in network skills
• Understanding of software development (coding)
• Understanding of the target environment (switch, router, firewall, etc.) API
• Be able to create a process flow diagram for what is being automated

From my view, it seems that if an individual lacks an understanding of the target environment it will be problematic for the individual and lead to frustrations as the distance is too great.  Knowing what API call to hit and what variables and values to use without knowing what the expected outcome should be can become challenging.  For me personally, once I knew what I expected to see as output the automation part came quickly.  In order for me to understand what I wanted to see I had to 1) Know what it was I wanted to automate and then 2) Create a flow diagram of what I was trying to automate.  Once I had the flow diagram things became much more clear.  I also found gaps or errors in what I was trying to do as there were missing parts. I cannot emphasize enough how important I feel flow diagrams are!!  A simple example is shown below:

The topic of this post is not flow diagrams but I feel it's an important piece to note and never miss an opportunity to stress this point.  Diagram what you are trying to automate before you automate as it will help you find gaps!  

Regrouping

OK, so we tried to make a network person a Network DevOps person and that did not appear to work out as I had hoped so what next?  We had an individual that came from the DevOps world and was now in the Network team. Their strength was more in DevOps and other areas than true 'network' but they did have network knowledge.  As our automation efforts started moving as this person became more and more engaged and started interleaving DevOps mentality and principles into our automation discussions.  

I  started to see a small mental shift happening within the small team just by this new persons involvement.   It was no longer just me talking about and attempting to apply DevOps principles and practices but now there were two us.  The more information we shared on the DevOps way of doing things the more I could see a small mental shift happening within this team and management.  

The Beginnings

We  thought we would be smart and add automation to our infrastructure partners contract.  Doing this we assumed would help facilitate our transformation to automation in a much faster manner.  What we discovered was that our providers (partners) were not as advanced with network automation as we anticipated.  In fact, the market itself was not as advanced in this area as we had thought.  In parallel, what value does a service provider get by providing automation for their customer as there is a potential of decreased revenue due to reduced workload?  All interesting challenges for sure but we needed to press forward, we needed to push people so we started inside (within the team).

Something is still missing

While from the outside it appeared we were making progress with Network Automation and adopting some DevOps principles, from inside we were not moving as quickly as I had envisioned.  Sure, we had written some playbooks and we established a central repository and a centralized API gateway but something still doesn't seem right.  Even though I have been sharing and spreading the message, the biggest barrier I expected in the beginning is/was still a barrier.  The mindset had not been successfully changed yet.  People (workers, managers and others) had not completely bought in to the concept of automation for network.  In addition we were not setup to manage the requests for automation that were now coming in due to the broadcasting of our vision.

Governance and Backlog Management

We created a Governance team to help manage the incoming automation requests and to manage our backlog.  The idea behind this was to create a technical assement of the request and then determine how we could quickly provide value.  The first output of this governance team was to create a flow diagram for requests.  The idea was that we had a picture of the process we would follow to assess requests before they were added to the backlog. This flow looks like this:

Following the process flow shown above ensures that :

• Requests are clear to Network team
• Requests are valid for Network team
• Opportunities to deliver value quickly are identified (IMPORTANT!)
• Requests are properly entered into Backlog for future sprint planning

We also began using visual management to track the requests from beginning to end.  This I feel was very important as we could document the initial ask and then visually track it across the board.  An example of our visual management is shown below (Column headings only):

Since starting the backlog management and technical assessment process we have closed over 250 automation requests.  These requests have varied in complexity from simple to complex. Some requests we have turned away as they just seemed too time consuming for the value they would provide.  All in all,  we are making steady progress to having a more automated environment.

What Next?

Now that we have a Governance team to manage the requests and we have a process flow to follow what we need is a Product Owner for Network Automation.  We need someone to own the service and determine and manage the priorities of the requests.  This Product Owner will also help in the creation of Sprints.  That will come in the future, in the mean time we start positioning and solutioning requests that come through the technical assessment process.  We do this so we can understand the workload required and then see if we have the available resources to work on them, if not then the ask gets put into the backlog and reviewed each week.

We also need to standardize on our tools to implement automation.  There are many tools out there that can be solutions for the automation requests that we have in our backlog.  The question is which one(s) do we standardize on and make a core competency for our NetDevOps team members?  We must avoid having many applications/tools to perform the job as maintence of the application/tool will become problematic.

Conclusion

We are making progress even if it is, to me, slow progress.  Trying to change a culture and habits over night, a few weeks or even within a few months is unrealistic.  It takes time but when you look behind you and see where you started from you can begin to appreciate the movement you are making.

We have begun to make behavior changes to think more like DevOps instead of by device and we've begun to put some critical pieces in place that help nurture that environment but we are not done yet.  We still have a long way to go but it's nice to begin to see some of the positives actions and behavoirs from this evolution.