As you most likely know by now, Solar Winds is at the heart of a massive breach by what appears to be a foreign government. As each day passes by, more information comes out that poor or non-existent security practices were commonplace, budget cutting was frequent and someone, somewhere set an update server with a very simple password. Before all of this went down, I had an opportunity to work with Solar Winds as one of their clients starting in May of 2016 and then later removed the software from all of the networks we managed at Intrinium in May of 2019.
When I took over as Director in May of 2016, our biggest expense aside from labor was the software from Solar Winds. It is very specialized software, complex and complicated, and it took a skilled technologist many months to learn and years to master. This software would tell you everything you could ever want to know about all of the devices on a network and allowed you to remotely monitor and manage the network. For example, it allowed remote access to a device so you didn’t have to roll trucks when someone clicked on Outlook 1,736 times and their computer froze up.
One of my first steps was to set out and do a features/risk/benefit analysis with my team. Did this software make our lives easier? Did this software provide superior features/benefits relative to the cost, and the competition’s features? Did this software provide actionable data? Overall, was this the right software? What I did know is that the software was expensive, and the team noted that it did not feel like Solar Winds provided the level of support we expected.
I won’t get into the gory details about that process, other than to say that after two years and many attempts to get high performance out of our Solar Winds software, we made the call to move to a new solution. It was not an easy undertaking, nor one that anyone looked at with any level of excitement. The change brought up a lot of anxiety about networks falling down after migration and the nightmare of many sleepless nights we had to look forward to if the migration was not exactly perfect. And then to do this 75+ different times; one for each environment we managed or monitored.
I’ve included a snippet below from Reuters North America around security, profit, and motivation at Solar Winds. The snippet is outside of my area of expertise and beyond any knowledge I had about Solar Winds working with them. But what I can say is that the entire operation lacked coherency. It always felt disjointed. From sales to engineering to their service desk to accounting, it felt like unnecessary friction.
The question that keeps every MSP (Managed Service Provider) up late at night… What if our software was compromised and we didn’t know? How much trust would we lose if we had to acknowledge to our clients that our software created a vulnerability in their network? Take a moment to think about having to make that phone call to a client. Ugh.
The final decision to move away from Solar Winds in 2019 rested on making sure the customer outcomes were achieved. With less friction (for us and our clients), more accuracy, and at a lower cost, it was a win-win for our clients and Intrinium. And in hindsight, definitely the correct business decision.
The takeaway for me is that Solar Winds transitioned from a Managed Service Provider’s Provider to a Profit Extraction Organization. It has always been, and ideally, always will, be my outlook to partner with organizations that view their customers as partners on a mutual journey. If there is alignment, profit will happen (this sentence alone is debatable and could be the fruit for several blog posts). You might be thinking, aren’t all non-governmental and not-for-profits profit extraction organizations? Yes. But, is the WHY of the business, “profit”, or to serve customers or some other reason not cause/effect of/for profit? If the ‘why’ is profit extraction maximization, is that a partner you can trust?
20 years from now, Solar Winds will be in the same category as Enron. Sad, tragic, and totally preventable. At this point, no one is quite sure what to do with the networks that may or may not have been accessed by the alleged foreign government. Remove all of the equipment and start over again, or wipe everything and hope that anything malignant was removed? No matter the final decision, this will take several years and billions of dollars to resolve.
In full disclosure, we were not running the Orion software package, but a different software offering from Solar Winds. At this time, it does not appear to have been affected.