We have started the second product increment (PI = 10 weeks) in our Cloud Platform project and are consolidating our experiences with a list of requirements and use cases for configuration management in our cloud services.
Configuration management is key for any environment with a complexity as SDDC especially in a service provider context. We can distinguish basically two phases, the product* development and the product rollout and lifecycle:
For the product development we’re looking forward to using the iterative deployment and integrated GIT functionality for versioning, currently in discussion for Houdini and vRA. Most of the following considerations, however, are driven from service development as well.
For the product rollout we want to maintain a catalog per customer containing all the products he has ordered. As the products are reused for multiple tenants they can coexist in different version at the same time. A different version in our case can be a tenanted customization of the product or generally an older/newer version (A tenant receives a product with a certain service pack included, another tenant without).
(* the term “service” is avoided as “services” are used within vRA for product or service categories.)
The following diagram all relevant artefacts that have to be versioned and managed for development, testing and deployment in our cloud are identified:
The green box depicts one product that consists of a blueprint with day 2 actions and all the dependencies that have to be provided in order to have this product fully functional.
The blue box contains the provider workflows – a set of framework workflows that are enforced in order to replicate the assets to our ITSM, monitoring and metering, write billing records. We also want to enforce as provider some parts of the blueprint, i.e. the OS version used together with minimal VM specs if we want to guarantee SLAs.
The red box represents the product catalog as a subset of all the services we provide per tenant. As a service provider we want to be prepared to allow limited customizations (custom properties) and will allow the customer to upload their own vCenter templates.
Design goals: we want to rely on GIT as a central versioning system and like to track everything as code to provide the base for the CICD process. We’d like to describe the entire system (actually the complete SDDC!) as set of flat files serving as the reasonable default that can easily be customized by overriding or extending only where needed (convention over configuration). The default must contain a description of a standard tenant setup with reservations, network profiles, catalog items and entitlements; we want to have the tenant onboarding as a process in which we have to slightly adapt the template tenant file and have it processed by the onboarding automation workflows.
It would be very useful if vRA provides the functionality of copying a whole tenant definition.
With the big picture in mind, we can now dive into a concrete example.
MS SQL Use-Case
The following diagram depicts all artefacts needed for a working MS SQL server with additional day 2 actions:
- Windows OS vSphere template (-> here we’d like to patch the template with the windows updates periodically)
- MS SQL server binaries (versions/editions/patches) and the necessary vRA software components for the blueprint
- We are providing day 2 actions (add database, add user, backup, ….)
- We are using a lot of custom properties within tenanted (or on business group level) property groups. So we need to have an overall configuration management for handling this complexity and it would be nice to have a check for requirements i.e. placing strategies for a given blueprint scenario: when we want to provide an AlwaysOn MS SQL cluster we will have constraints for the placing (licensing on dedicated cluster nodes) and affinity rules in a scalable multi-machine blueprint.
For the rollout and lifecycle part we have the following requirements:
- A provider can update a blueprint so that existing deployments are no longer consistent with the blueprint definition. In this case, no changes should automatically be made by the system upon saving the updated blueprint — there should be a deliberate “update deployment to blueprint” action.
- In the case of a service provider, the power to do this is normally in the hands of the consumer (for contractual reasons)— they should be notified that the service has changed and choose to “opt in”/”upgrade to” to the new version (or keep with the old one). This opt in should ideally be handled per each individual deployment. For example, if we have 10 active deployments from the same blueprint that was updated, is should be possible to choose to update each deployment independently or do them all in bulk. If there is an API to detect the inconsistency (version of blueprint) and make the change on one deployment, it should be possible to update them all programmatically.
- In the case of internal use, as an admin you may want to force a bulk update to reflect the change in a service to enforce standardization. Consumers may be notified that the change was applied but they will have no choice as to whether to accept it (i.e. security updates). The bulk update scenario is the primary one rather than updating deployments individually. This is more of a “push” update than a “pull” from the consumer side as described above, although this can theoretically be done through the same APIs.
VMware released a clear and straightforward strategy last year. Any Device, Any Application, Any Cloud. A big role in this strategy plays the vRealize Suite. One Suite to rule them all. More and more cloud endpoints are supported by vRealize and each new version introduced many new features.
A piece that is almost completely missing in the vRealize Suite is configuration management. VMware detected this gap and came with a management pack to solve at least the missing configuration management on the content side. Configuration Management of vRealize itself is a different story for another blog-post… The content management solution from VMware is called Management Pack for IT DEV Ops (aka Houdini). With the MS SQL Server and our preference towards GIT – fully-versioning and everything as code – in mind, let’s see what Houdini already delivers out of the box.
Houdini can be used to capture, store, test and release artefacts from the vRealize Suite (and some more). It stores the artefacts in a Xenon repository hosted on an own content server. The artefacts can be captured, tested and released on different vRealize stages and tenants (for vRealize Automation). Some of the top features are:
- capture/test/release of single packages (artefacts) our groups
- This includes a versioning mechanism
- manage groups
- add/remove packages to existing group etc
- detect and include dependencies
- day-2 operations on already capture packages
- visual pipeline of the capture/test/release operations
To see Houdini in action, this Youtube video gives you a good first insight:
Supported artefact types
In the list bellow one can see all supported artefacts bi Houdini. In bracelets () the Houdini term can be found.
- Converged Blueprints (Automation-CompositeBlueprint)
- XaaS Blueprints (Automation-XaaSBlueprint)
- Software (Automation-Software)
- Property Definitions (Automation-PropertyDefinition)
- Property Groups (Automation-PropertyGroup)
- Resource Actions (Automation-ResourceAction)
- Event Broker Subscriptions (Automation-Subscription)
- Codestream Pipeline (CodeStream-Pipeline)
- Custom Ressources (nA) → Included if captured with dependencies
- Icons (nA) → Included if captured with dependencies
It is also possible to handle vRealize Automation 6 artefacts, I have not listed them here since vRA 6 is not relevant in our case.
- Workflows (Orchestrator-Workflow)
- Actions (Orchestrator-Action)
- Configuration elements (Orchestrator-ConfigurationElement)
- Packages (Orchestrator-Package)
- Alert (Operations-Alert)
- Dashboard (Operations-Dashboard)
- Report (Operations-Report)
- Super Metric (Operations-SuperMetric)
- Metric Configuration of Adapters (Operations-ResourceKindMetricConfig)
- Widget Topography Structure (Operations-TopoWidgetConfig)
- Text Widget Content (Operations-TextWidgetContent)
- View (Operations-View)
- vSphere Guest Customization (vSphere-CustomSpec)
- VM Template (vSphere-Template)
- Files on Linux Host (Files-Linux)
- Files on CIFS share (Files-Windows)
- vApp Template (vCloud-vAppTemplate)
- Catalog Media (vCloud-Media)
In terms of configuration management the vRealize Suite itself does not deliver much built in functionality. The management pack for IT DevOps tries to change this, but (there is always a but, right? 🙂 ) it is not yet there. In the next chapter we will discuss in more detail the much needed features.
Obviously the main thing missing in Houdini is a native GIT integration. GIT is the preferred tool while developing any kind of content (also infrastructure as code). GIT is a must-have: in our CICD pipeline, which highly depends on GIT, Houdini does simply not integrate smoothly without integration to GIT. Merge-conflicts anyone?
It should be possible to exclude dependencies. With Houdini it is possible to include dependencies, but this will take all. Also system workflows or workflows used by a self written utils library. This can lead to huge issues. For example if a developer tests a new version of vendor Package in vRO in a DEV environment Houdini will automatically capture the new dependent artefacts and overwrite the old package piece in integration. Obviously this will in a long run lead to issues.
As we will have more then one customer on a vRA instance, tenant awareness is crucial for our cloud platform. It would be great if Houdini could introduce some sort of tenant profile/description. This can be as simple as a YAML file per tenant which describes tenant specific properties. If Houdini could take this file into consideration when moving content from one tenant to another this would be a huge help. I know, it is possible to hook in between the different states and write one workflows which will then exchange the tenant part, but an out-of-the-box experience is different, right?
With version 2.2.1 of Houdini the group management was introduced. This allows users to add and remove packages to existing groups. Great! As we planned to have a customized catalog per customer, it would be extremely useful to have a nested group functionality. Our vision here is to have a Houdini group per vRA product. Nested groups would allow us to create a group per Customer with only the product included he subscribed for. Maybe even with a clearly defined version – and not yet another copy?
Lets see, taking a look on my overview picture and compares it with the artefact list provided by Houdini, the missing bits and pieces can be derived mainly in the infrastructure department :
- reservation policies (placement, location, DRS)
- storage policies (quality, placement, location, DRS)
- NSX assets: networks, network profiles (IPAM definitions), security tags, security groups, security policies
- entitlements (configurable through naming conventions?)
- vRO Dynamic Types
Some of these configurations are very important for tenant onboarding and are – like machine prefixes – not tenanted at all. If these values are referenced in the blueprints – like a network profile – they have to be adapted during a preprocessing step before deployment. For a true SDDC in larger environments we should consider externalizing environment and tenant aware profiles and properties in a configuration management like spring cloud config is providing as an example. Of course any pre and post processing hooks are always very handy to adapt the packages to deploy just in time but there are better concepts to ensure environment dependent configurations without the need of touching already tested packages again.
For vCenter templates we need to detect changes (patching OS) to ensure that we are only using tested versions for our products.
Our collection of work arounds
- Currently GIT is not supported at all in the vRealize Suite. As we cannot accept this limitation we helped ourselves by doing a manual export and checking in the desired files in GIT.
- We had to implement a dependency detection that is very similar to the one provided by Houdini and are tracking vRA artefacts directly as vRO resources. Common or system libraries have to be filtered manually in order to prevent overwriting of files that do not belong to the desired functions. As every developer needs to maintain the dependencies in a vRO Package, export them and commit them to GIT, missing dependencies and merge conflicts happen way too often.
- Special attention has to be paid to references to workflows (uuids) that should be versioned (changed!).
- Furthermore we are developing a formal description of the dependencies each component can have (together with the interfaces and properties it offers).
A complete solution within Houdini should natively integrate with GIT, be configurable and filtering system libraries, should track all dependencies – loose dependencies have to be freely and hierarchically groupable. We need to be able to create feature branches as needed and want to have a merge view where merge conflicts can be resolved.
We are looking forward to collaborating closely with the team implementing vRealize Code Stream Management Pack for IT DevOps (Houdini) and hope, that we get the GIT integration as soon as possible.
Kudos to Marco Pillot helping me to write this article!