By Todd Cretacci

Introduction

Over the past several years, I’ve worked with traditional infrastructure teams, virtualization administrators, cloud engineers, network teams, security teams, and application owners to build on Google Distributed Cloud Virtual clusters.

This guide focuses on the operational side of GDCV: the lessons learned after clusters are deployed and organizations begin managing them at scale. The lessons in this guide can apply to both vmware and baremetal environments.

Consider this a dynamic blog post as I plan to add things as time goes on.

What Is Google Distributed Cloud Virtual?

Google Distributed Cloud Virtual (GDCV) allows organizations to run Google Kubernetes Engine (GKE) in customer-controlled environments while maintaining alignment with Google’s operational model.

For many organizations, this means:

• Kubernetes in VMware environments or bare-metal servers
• Modern application platforms without full cloud migration
• Hybrid cloud architectures
• Platform engineering initiatives
• AI-ready infrastructure
• Consistent operational models

The biggest advantage of GDCV is that it enables modernization while preserving existing datacenter investments. However, simply deploying a cluster does not guarantee success.

Lesson #1: Use Ansible as Your Primary Automation Platform

I am a strong believer in Infrastructure as Code, but GDCV environments require operational automation beyond initial provisioning. As discussed in my article Infrastructure as Code Beyond Terraform, infrastructure automation extends well beyond resource creation.

While Google’s Terraform modules can help create portions of the environment, they do not typically manage the complete lifecycle of:

• Cluster configuration files
• Operational metadata
• Upgrade workflows
• Environment standardization
• Day-two operations

Benefits to using Ansible include:

• Repeatable deployments
• Upgrade automation
• Configuration management
• Environment standardization
• Operational consistency

Use Jinja Templates

Store cluster configuration elements as templates:

• Cluster YAML files
• Network configurations
• DNS settings
• Registry settings
• Environment variables

Every playbook execution should regenerate configuration files from source templates.

This creates:

• Consistency
• Auditability
• Reduced configuration drift

A regenerated configuration is significantly easier to trust than a manually edited one.

Run Ansible playbooks in a python virtual environment

Every Ansible controller should have a dedicated Python virtual environment.

Benefits include:

• Isolated dependencies
• Version consistency
• Plug-in management
• Simplified upgrades
• Reduced operating system conflicts

This creates a repeatable automation environment that does not interfere with the host operating system.

When troubleshooting begins at 2 AM, predictable software versions matter.

Lesson #2: Use Static IP Addresses and Reserve Entire Subnets

Networking problems account for a surprising percentage of deployment and upgrade failures. While DHCP may seem easier initially, it introduces unnecessary risk. I strongly recommend:

• Reserving dedicated subnets
• Using static IP assignments
• Planning growth up front

Benefits include:

Predictability

Every node has a known address.

Easier Troubleshooting

Administrators can immediately identify:

• Control plane nodes
• Worker nodes
• Admin systems
• Infrastructure services

Preventing Address Exhaustion

The largest risk with shared DHCP scopes is accidental IP consumption.

Imagine:

• An upgrade starts
• New nodes need addresses
• Scope unexpectedly runs out of IPs
• Upgrade fails

The result could be a failed deployment or upgrade. By dedicating a subnet to GDCV, no unexpected devices can consume addresses intended for cluster infrastructure. This aligns with existing enterprise networking practices and minimizes operational risk.

Lesson #3: Treat Your Network Team as Strategic Partners

The network team is one of the most important stakeholders in a successful GDCV deployment. Far too often, infrastructure teams build clusters and then involve networking after problems occur.

Do the opposite. Include networking early. Work with them to:

• Reserve subnets
• Create routing standards
• Define DNS requirements
• Plan firewall strategy
• Review future growth plans

One recommendation I highly suggest:

Use Firewall Groups

Whenever possible, place cluster subnets into firewall groups.

Benefits:

• New nodes automatically inherit rules
• Reduced manual updates
• Consistent security policies
• Easier compliance management

Without this approach, it becomes very easy to miss addresses during cluster expansion.

Missing firewall rules can create:

• Application outages
• Connectivity failures
• Upgrade issues
• Long troubleshooting sessions

The simplest operational model is almost always the best one.

Lesson #4: Deploy Dedicated Linux Ansible Controllers

Do not use admin workstations or the admin cluster nodes as ansible controllers. These systems are regularly rebuilt, replaced, or upgraded. Do not tie long-term automation to short-lived systems. Deploy to dedicated Linux servers that function as permanent automation controllers.

Benefits:

• Consistent execution environment
• Persistent automation
• Centralized logging
• Shared operational practices
• Reduced dependency on individual administrators

This becomes incredibly important as environments grow. These machines could even double as jump boxes in a secure network environment.

Lesson #5: create a meaningful naming convention

It can become confusing when troubleshooting a GDCVV cluster. A valuable trick that I learned with using static IPs is it allows you to incorporate the last octet of its IP address into the name. Good ideas for naming conventions include:

• Location
• Environment type – development, test, pre-production, production, etc
• Cluster type / Purpose
• Network type
• IP address

For example, if I had a cluster node with the IP of 10.10.10.125, I could name that cluster node eastprod125. This naming convention would not only tell me the IP address of the node but also:

• Resides in an eastern area of the country
• This cluster is used for production
• Has an IP address that ends with 125

Currently in GDCV for VMWare the first three nodes of a user cluster are the control plane nodes – These nodes do not contain application workloads. When the nodes have the IP address in the hostname, I can quickly tell the role of each node.

Lesson #6: Standardize Cluster Folder Structures

This sounds simple but it isn’t. One of the easiest ways to create future problems is allowing every administrator to store files differently. Like documentation, it’s good to have a standard location for files. I recommend creating dedicated directories for every cluster:

clusters/
├── buffalo-prod-connected
├── buffalo-nonprod-connected
├── chicago-prod-disconnected

Within each cluster directory:

cluster-name/
├── kubeconfig
├── ssh-keys
├── logs
├── configs
├── backups
└── sensitive_secure files

Benefits:

• Easier administration
• Easier upgrades
• Better auditing
• Reduced mistakes
• Faster troubleshooting

Lesson #7: Design Around Variables, Not Environments

One of the biggest automation mistakes organizations make is writing separate playbooks for every environment. Use these resources to simplify your code.

• Inventories
• Variables
• Templates

A well-designed automation framework should support:

• Development
• Test
• Staging
• Production

with the same playbooks. The environment should change through variables, not duplicated code. This aligns with modern Platform Engineering principles where reusable patterns create operational consistency.

Lesson #8: Adopt GitOps Early

As discussed in Infrastructure as Code Beyond Terraform, GitOps significantly reduces configuration drift and improves governance.

Consider:

• Google Config Sync
• Argo CD
• Flux

Use them to manage:

• Namespaces
• RBAC
• RoleBindings
• Policies
• Platform standards

Benefits include:

• Version control
• Auditing
• Consistency
• Automatic reconciliation

These platforms can even prevent unauthorized changes from administrators. That may sound restrictive but in enterprise environments, that is often exactly what you want. These tools help companies rely on change management and approved access to cluster resources.

Lesson #9: Use Meaningful Cluster Names

Naming conventions matter more than many people realize. Cluster names should immediately identify:

• Location
• Environment
• Network type
• Purpose

Good naming reduces:

• Operational confusion
• Deployment mistakes
• Troubleshooting delays

Make names understandable to:

• Engineers
• Developers
• Security teams
• Leadership

Good examples are:

• buf-prod-shared
• roc-nonprod-sportsapps
• chi-preprod-pci

Lesson #10: Use Shell Aliases for Context Switching

If administrators manage several clusters, mistakes become inevitable.

Avoid commands like:

export KUBECONFIG=/path/to/file

Instead create aliases:

prod-cluster
nonprod-cluster
dr-cluster

Context switching becomes:

• Faster
• More consistent
• Less error prone

This simple improvement can prevent significant production mistakes.

Lesson #11: Plan Double the Resources You Think You’ll Need

Resource planning is one area where many organizations struggle. Cluster sizing must account for both applications and platform services. My recommendation is to plan for, at least, double the resources you initially expect. Take into account security and monitoring tools may be added at some point so these should be added to your initial assessment.

Include:

• Kubernetes system services
• Monitoring
• Logging
• Future workloads
• Cluster expansion
• Upgrade requirements

The biggest surprise for many organizations is that upgrades often require additional temporary resources. A cluster cannot expand if VMware lacks capacity.

Lesson #12: VMware Teams Are Critical to Success

As discussed in How Organizations Can Extend Existing VMware Investments with Google Distributed Cloud Virtual, VMware remains a foundational platform in many GDCV deployments.

Remember:

Behind the scenes:

• VMware provisions VMs
• VMware manages resources
• VMware provides infrastructure services

GDCV downloads the templates, but VMware does the heavy lifting. Like your network team, treat your VMware administrators as platform partners, not external dependencies.

Upgrade Best Practices

Run Diagnostics Before and after Upgrades

GDCV includes both cluster diagnostics and upgrade testing tools. Check out Google’s documentation on to run these and do it before and after every upgrade.

• How to diagnose a GDCVV cluster
• Running the pre-upgrade tool

Check Failed Pods and jobs

Check for failed pods not only in a user cluster but in the admin cluster as well. Each user cluster has a namespace in the admin cluster named after it.

Here’s a command to have kubectl only show pods that aren’t in Running or Completed status:

kubectl get pods -A | awk 'NR==1 || ($4 != "Running" && $4 != "Completed")'

Log Everything

When running gkectl output to a file so the log is saved for future reference with the –log_file switch.

gkectl upgrade cluster --kubeconfig $KUBECONFIG --config east-prod-cluster.yaml --log_file: /opt/clusters/east-prod-cluster/logs/cluster_upgrade_east_prod_10_05_2026.log

Learn What gkectl Is Doing

One of the most valuable skills a GDCV administrator can develop is understanding what happens behind the scenes.

Learn:

• Cluster lifecycle operations
• Kubernetes workflows
• Node replacement activities
• Validation stages

Kubernetes logs can be challenging. Force yourself to decipher the log and understand what is happening. This knowledge will them dramatically improves troubleshooting speed during an incident.

Build a Team, Not a Hero

One major difference between public cloud and GDCV is a company owns the entire stack. Potential issues may exist in:

• Hardware
• VMware
• Networking
• Storage
• DNS
• Kubernetes
• Security
• Applications

No individual can be an expert in all of these areas. Build a team. Create documentation. Share knowledge. Reduce operational silos.

The goal is resilience, not heroics. Take pride in training someone to be a superstar.

Stay Current on Updates

Google regularly addresses:

• Vulnerabilities
• Platform issues
• Security concerns
• Feature enhancements

Running outdated clusters creates risk. Older versions may eventually become unsupported. At some point, an organization may discover:

• Direct upgrade paths no longer exist
• A rebuild is required
• Applications must migrate

Those conversations are far harder than maintaining a regular upgrade cycle so practice upgrades as much as it takes to get good at it. Make sure your app owners are involved to provide solid feedback.

Maintain a Development Environment

I can’t stress the important of a development environment enough. If there isn’t money in the budget for it, build a small dev cluster on the VMWare environment being used for something else.

Always maintain:

• Development
• Test
• Validation environments

Before introducing:

• New GDCV versions
• Feature changes
• New integrations
• New operational tooling

As discussed in Running Kubernetes in Connected and Disconnected Environments, production environments rarely behave like lab environments. A dedicated development platform allows organizations to identify issues before production users are affected.

Final Thoughts

Google Distributed Cloud Virtual is one of the most practical hybrid cloud platforms available today because it acknowledges the reality most enterprises face: modernization is a journey, not a replacement project.

The technology itself is impressive. However, long-term success depends less on Kubernetes and more on operational discipline.

Use automation.

Standardize everything.

Invest in GitOps.

Partner with networking and VMware teams.

Document your environments.

Build dedicated automation platforms.

Stay current on upgrades.

Most importantly, create processes that survive personnel changes and platform upgrades.

The organizations that succeed with GDCV are rarely the ones with the most sophisticated technology. They are usually the ones with the most repeatable operational practices.

Related Articles

• Why Google Distributed Cloud Virtual Is the Most Overlooked Hybrid Cloud Platform
• How GDCV Helps Organizations Modernize Infrastructure Without Abandoning Existing Investments
• How Organizations Can Extend Existing VMware Investments with Google Distributed Cloud Virtual
• Running Kubernetes in Connected and Disconnected Environments
• Infrastructure as Code Beyond Terraform