DevOps for Cognitive Services

It is common to talk about “last mile” adoption problems for both Agile (usually traditional QA teams, in other cases, traditional business teams) and now DevOps (again QA and Data teams as the typical bottlenecks).

I have recently been working with the Microsoft Cognitive Toolkit (CTK), and asked myself the same question on how to create a continuous delivery pipeline for AI applications. If AI implementations grow as much as Gartner is predicting, the deploying AI apps have a good change of become the next “last mile” problem.

Researching on this, came across an article on DevOps for Data Science and AI: Team Data Science Process for Developer Operations, with the description “This article explores the Developer Operations (DevOps) functions that are specific to an Advanced Analytics and Cognitive Services solution implementation.”

However, instead of the (more recently) clear-cut how-to documentation that Microsoft has been providing as part of its VSTS products, this article seemed more like a list of building blocks that need to be distilled into at least a nice simple sample implementation.

Investigating the Azure Data Science Virtual Machine seems the next step to better understand how everything is connected. Among other things it has the CTK installed, which is what is attracting my attention in this area. As a production solution though, it is not the final answer, so to define a process to deliver from this machine as a development environment to a production business server, a lot has still to be done.

I will continue experimenting and will let you guys know next. My ultimate goal would be to use some version of Prolog.

Successful delivery of the first Microsoft DevOps FastTrack in Mexico

I just returned from Mexico where I worked for two weeks with one of the country’s biggest companies, and delivered  the first Microsoft DevOps FastTrack there (all in Spanish ;-)). It was a fantastic experience to work with such a tight knit team (about 16 people).

For obvious reasons I can’t share too many details about the engagement. I can however share one key fact that pretty much won over their hearts and minds on adopting VSTS not only for this project, but also for the rest of the company as well. Here is the list of tools we used:

Programming languages:

  • Jersey with Java 8 (JDK 1.8)
  • JavaScript with Angular 1.X
  • HTML 5, CSS3 with a Bootstrap template

Programming tools:

  • NetBeans 8.2
  • Visual Studio Code 1.19

Dependency Managers:

  • Maven 3.x
  • npm
  • bower

Testing frameworks:

  • JUnit and RestAssured for Java
  • Selenium for frontend

HTTP Servers:

  • Tomcat 8.54
  • Apache httpd 2


  • PostgreSQL 9.6, client PgAdmin 9.3
  • Flyway 5.0.5

Version Control:

  • Git 2.15.1

As you can see, it was all done using either OSS or non-Microsoft tools. The only Microsoft tool (other than VSCode) was VSTS for project Management and DevOps, and also for overall orchestration of the whole process.

So we started with a skeptical team thinking Microsoft (me on their behalf) is going to try and impose a Microsoft tool stack, and ended up with an elated team using their current existing toolset being coordinated with VSTS. For them this was a very powerful statement from Microsoft about their support for OSS and tool diversity: we can work with all kinds of toolsets.

Having worked with Microsoft during a time where it was heavily focused on just pushing its own products, the current general attitude of respect, openness and inclusiveness I am getting from all parts of Microsoft is a breath of fresh air, and customers are noticing (for a non-code example, see for instance this article).

Presenting at VSLive! 2018 in Austin

I have been selected to present at Visual Studio Live! this year. These are the two topics I will presenting on:

Many "scaling agile" efforts have failed for lack of focus on delivering working software on a regular basis. They have instead just been a refactoring of processes to gather requirements. It is easy to ignore tooling, development, and down–to–the metal DevOps practices when talking about scaling agile. However, these can be a boon or a bane to achieve working software – meaning tested software – and as with any process implementation, agile at scale is no different.

You will learn:

  • How most agile scaling efforts focus just on restructuring backlogs and leave developers behind
  • Learn how to identify DevOps and tooling bottlenecks, and align the implementation of agile at scale with the Agile Manifesto – working software over comprehensive documentation
  • Rules of thumb derived from real world experience on doing DevOps when scaling agile
    and the second talk will be

Knowledge management is often touted as one of the major applications of SharePoint and similar tools that made them widely successful. Oddly, the shoemaker's children go barefoot. Companies lose millions every year because of their inability to actively manage the intellectual capital invested in the applications they develop. VSTS lets you take control of this situation by using it as your team's communication hub and expanding it with practices and strategies to truly manage the knowledge supporting your applications.

You will learn:

  • The hidden risk of not managing the intellectual capital behind your code
  • How to use VSTS as a communications hub for your team and a knowledge repository for your application, plus usage strategies to maximize long term value
  • Fundamental strategies, best practices, and rules of thumb for preserving and managing a development team's knowledge of an application, minimizing attrition risk

The conference is from April 30th to May 4th at the Hyatt Regency. I don’t yet have the schedule of when I will be presenting, so I will update this post when I know it. You can register at

See you there!

A new DevOps user group in Austin dedicated to Microsoft tooling

There a plenty of DevOps meetups in Austin (amongst them the Agile Austin DevOps SIG), but ever since I worked with other fellow TFS users in town in leading the local TFS Users Group, later renamed to Austin ALM User group, there was always the question on whether it should actually be a DevOps group.

The initial resistance came from the misconception that DevOps was just the end part where software meets production. All the way we talked about merging our efforts with the .NET user group.

So, after a hiatus Austin finally has had since September this year the Microsoft DevOps User Group (MDOUG), spearheaded by Jeff Palermo from Clear Measure, with my-self and others as co-organizers.

Here is the group mission:

This group is for software professionals working on the Microsoft platform who wish to explore, implement, and learn DevOps. This group embraces cloud and as such will bring in the best presenters for each topic while live streaming all content. Additionally, this meetup serves as the hub location, providing an example of other MDOUG live-stream locations in major cities around the U.S. This group recognizes the technical practices included in the DevOps Handbook by Gene Kim, Jez Humble, Patrick Bebois, and John Willis. We encourage members to read this book. From a technology perspective, we seek to implement these practices on Visual Studio-based software, leveraging VSTS, and running on Azure or on Windows servers anywhere.

This is a worthy successor to the original Austin ALM/TFS User group, and I recommend all to become a member. Amongst other presenters we have had Brian Harry himself speak to the group, follow by Damian Brady, both from Microsoft. And now I am recommending that we have Arun Chandrasekhar on DevOps with Azure and OSS.

See you there!

Azure DevOps with OSS

As MVPs, we participate of meetings with Microsoft to provide feedback or to learn about some upcoming new product delivery. This is one of the finest opportunities of being an MVP: to be at the cutting edge of Microsoft technologies.

In October there was an internal MVP meeting with Microsoft on DevOps with Azure. Without going into the specifics, which are under NDA, I want to bring that this one caught me by surprise. When I got the invite, I only looked at the meeting title “Azure DevOps OSS + Q&A with Arun Chandrasekhar”, and took for granted it was about OSS VSTS integrations with Azure: nothing new for me as I have been doing this for while.

However I was blown away by Arun’s presentation: it was about how to do DevOps with OSS tooling, especially Jenkins and Terraform. At my previous job I had worked extensively with Jenkins, including being able to be one of the architects of a build infrastructure that ended up being used by a couple thousand developers.

What I was not aware though, and Arun’s presentation helped me correct that gap, was that there is an extensive set of existing integrations with Jenkins for Azure, including Azure VM Agents, and an Azure-ready VM for Jenkins among others:

To see a couple of those as demos, I recommend watching the two videos from Arun with Donovan Brown:

And the video from Jenkins World 2017 presentation:

There is an entire world of possibilities to be explored, such as Azure DevOps Solution Architectures that are based on Jenkins and OSS:

Given my previous interest and work with Jenkins I will be learning a lot after delving into those references. In conclusion: it’s always good when something forces you to step out of your comfort zone, and meeting Arun has definitely been a great opportunity.

Follow up on videos from the DevOps Fastrack training

From the last few posts you might have noticed that I have been following up on the DevOps FastTrack offering, and the training that was provided by Microsoft on September, and how valuable that has been for me in understanding how a cutting-edge DevOps practice can be. The VSTS team in that regard is as cutting edge as Facebook’s, Google’s or Netflix. They are not bound to adopting practices such as SAFe (that was a question asked in one of the talks); rather they are defining their own because they are at a place where they are the pioneers (not the individual practices, but their use at scale). They are operating at the Ri level, they are transcending what is currently known.

It was a stated purpose that we as MVPs and Microsoft partners should share these presentations with our customers. I started doing that with every engagement since then, always starting the summary one made by Lori Lamkin.

Now the great surprise: Sam Guckenheimer has worked with the team that presented in extracting any NDA references, and has now shared the whole set of talks at DevOps at Microsoft. This is a treasure trove on how to to do DevOps with VSTS and beyond. I am organizing sessions at our company (Nebbia) so we can watch it together and use it as inspiration for next steps on our own DevOps journey.

Slides from presentation on Microsoft’s Agile Transformation

image Thank you very much for attending! It was one of the biggest crowds ever, considering our small venue  (we had 30+ people in the room, and 14 in the waiting list). Next time we might need to have a bigger room, and Leland and I are considering other options depending on the interest on future presentations.

Microsoft Agile Transformation and Planning.pdf (1.1MB)


I also just learned from Aaron Bjork that he has created a clean presentation of this content (there were a lot of down-the-rabbit-hole discussions at the Fast Track training) and posted it in Youtube.

My presentation of course was based on my own experience at Microsoft VSTS in 2007, whereas Aaron joined the leadership team later, so he actually saw the pain at a critical point, whereas I had just witnessed when it was starting to build. Of course you can’t compare listening directly from the source. However as some of you pointed out, the historical perspective was just as valuable, and thank you for your kind remarks :-). That said, I strongly recommend watching Aaron’s presentation as well.

Upcoming Meetup: Agile Transformation at the Microsoft TFS/VSTS team

You might remember that I attended the DevOps Fast Track training last month. So I jumped at the opportunity of bringing some of this content to the Agile-at-Scale User Group of which I am a co-lead with Leland Newsom.

This month’s topic will be on the Microsoft Agile transformation his team went from circa 2009 and still goes on until today. At the DevOps Fast Track summit it was my favorite talk, and I will revisit the topic to also bring my experience as Microsoft TFS product team member, and how it contrasts with today’s new dynamics.

Here is the talk abstract:

Meetup Topic: Agile Transformation at Microsoft TFS/VSTS team
Microsoft went from having a cycle of 18 months between releases, to almost daily releases of VSTS. Let’s examine the inside story of this extraordinary Agile transformation, and what lessons we can take from it.

You can register for the meeting at

See you there!

DevOps Fast Track training for Partners: learning how Microsoft tools scale to allow lightning fast DevOps

A couple of weeks ago I finally attended the long waited DevOps FastTrack Airlift 2017.

clip_image002These four days were packed on advanced information of how Microsoft’s VSTS team does DevOps. The team is comprised of about 800 people, and they work on both TFS and VSTS (I learned that only about 8% of the code is different between them, pretty much the same product to me).

A brief look at the agenda of the first three days (the fourth was NDA) show how the VSTS team uses its own product to its maximum capabilities:

DAY 1.1:  [KEYNOTE] Microsoft DevOps Transformation

DAY 1.2:  Agile Transformation and Planning

DAY 1.3:  A Day in the Life of an Engineer

DAY 1.4:  Running the Business on Metrics

DAY 2.1:  Architecture and Microservices

DAY 2.2:  Security

DAY 2.3:  Quality and Testing

DAY 2.4:  Deployment Practices

DAY 3.1:  Production Telemetry and Live Site Culture

DAY 3.2:  How We Do DevOps at Microsoft: Wrap-Up Session

DAY 3.3:  Visual Studio Partner Lightning Talks

a.   Modern Requirements

b.   Redgate

c.   WhiteSource

d.   Aqua


What is great about this internal usage (known at Microsoft as “eating your own dog food”) when added to the ten thousand people that use it for Windows development, and other several thousands for Exchange, Office, etc, is that is allows for testing it at a scale that cannot be matched by any other development tool provider.

For instance, even if Atlassian has (last I heard) about 600 developers, which is in the same order of magnitude of the VSTS team, they can’t beat having 40 thousand developers using the “beta” versions and having to do their daily work on the most recent bits.

Microsoft VSTS is primed for scalability from the very core of its engineering process, and it shows. If you are looking at similar tools for your enterprise, I recommend looking closely not only at their scalability numbers, but also their track record for support escalation.

Some tool vendors will have issues, clearly visible to the public, that had been opened more than 6 years ago and have made no progress to resolution. This is not necessarily wrong but it does indicate whether the tool provider actually has enterprise customers as their priority. With Microsoft VSTS you are also engaging with a clear incident escalation path that leads no doubts to resolution time: it is days, not months, and definitely not years, and if something can’t be resolved it will be marked as such, not be left open.

Several customers have been migrating to TFS from competing tools and are amazed at how their support load just decreases, from having to be working on every weekend to put out fires caused by load on build or issue tracking systems, to just having no issue at all and getting their weekend back. With VSTS it goes even further, as now the administrators don’t have to worry about patching, upgrading etc.

These four days have been eye opening. With the follow up email I got the following invitation:

“We encourage you to leverage the content below as you present to customers, conferences, user groups and/or colleagues.  We look forward to sharing the Microsoft DevOps Transformation story across as many audiences as possible.” , which I am happy to oblige:

I am happy to oblige, given the high quality of the content.  I will be using the materials at customers starting immediately as a way to show a possible ideal implementation of a cutting edge DevOps practice. Not that we need to replicate exactly what Microsoft is doing – not many companies have the same resources, in the topic DevOps, Microsoft is at the level of Amazon, Netflix, Facebook or Google – but we can always look at their experience to extract ideas of where to direct our efforts, since Microsoft is using the same tools as we are.

Enterprise DevOps Fast Track training

I just discovered that as part of Nebbia, a Microsoft DevOps Partner, that I will be attending an advanced training for those delivering the engagements related to the now called Enterprise DevOps Accelerator offering. As you might recall from a previous post, I had been yearning to learn more about the cutting edge of what is going on in this space.

I have been working with topics that are under the DevOps umbrella since 2005, even if the moniker, and a general agreement on what it stands for, has been defined only more recently.

The buzz around DevOps is indisputable, so much that has led some like Matthew Kern to declare that Agile is dead, and DevOps is its replacement.

I don’t know exactly what the training will be, but I look forward to it as I can also personally see the commoditization of Agile concepts, and the rise of Scaling Agile and DevOps as its natural replacements. I will come back to that on some other post.


<<  May 2018  >>

View posts in large calendar

Month List