Restarting a workload in one cluster is a focused task. Coordinating the same operation across a fleet means dealing with cluster discovery, authentication, network access, targeting and reporting—not just the restart itself.
I created A service account token pattern for coordinating workload restarts across multiple clusters on Azure Local. The mass-aks-pod-restart to bring those pieces together. Its AKS Arc Fleet Pod Restart implementation uses Kubernetes service account tokens stored in Azure Key Vault, Azure Automation for orchestration, and on-premises Hybrid Workers to reach the clusters.
The design targets fleets of 100’s of AKS Arc clusters on Azure Local. That is the scale the architecture is designed around, not a claim that every restart completes within a guaranteed time.
Why a service account token pattern?
The README describes an AKS Arc environment where Azure RBAC is not enabled by default and Azure AD or Managed Identity tokens are not accepted directly by the Kubernetes API. The solution separates Azure authentication from Kubernetes authentication.
The Hybrid Worker uses its managed identity to retrieve secrets from Key Vault. It then uses the target cluster’s own service account token to authenticate to that cluster’s Kubernetes API. Managed identity provides access to the stored credential; it does not replace the Kubernetes credential.
Each cluster has its own token and API server address. A token from one cluster cannot be reused to authenticate to another. This keeps the fleet-wide workflow consistent while retaining cluster-specific authentication.
How the fleet workflow fits together
The implementation splits setup from execution, and fleet orchestration from individual cluster operations.
- Bootstrap each cluster once. From the HCI host, use an admin kubeconfig to apply restart-sa.yaml. Store the service account token and API server address in Key Vault under satoken-<clustername> and apiserver-<clustername>.
- Discover and divide the fleet. Master-FleetRestart.ps1 uses Azure Resource Graph to enumerate connected clusters, applies the requested targeting, and divides the work into 50-cluster shards.
- Execute close to the clusters. Shard-RestartPods.ps1 runs on a Hybrid Worker on the same LAN, with 50 parallel threads. Worker-RestartSingleCluster.ps1 retrieves the relevant secrets and calls the Kubernetes API directly over REST.
- Record the outcome. Audit records go to the AKSArcRestart_CL table in Log Analytics, capturing cluster, namespace, workload, status, duration, initiatedBy and jobId.
The runtime restart path does not need kubectl or Az modules. Those dependencies are kept outside the per-cluster execution path; kubectl is still used for bootstrap and token rotation.
| The goal is one repeatable restart workflow across many clusters—not hundreds of separate manual operations. |
What multi-cluster management adds
This is a focused fleet restart tool, rather than a general-purpose cluster lifecycle platform. Within that scope, the benefits are practical:
- Consistent execution: the same discovery, credential retrieval, restart and audit workflow can be applied across the selected fleet.
- Targeted operations: filters let operators select one cluster, a named group, a name prefix or an environment tag instead of treating every request as fleet-wide.
- Parallel dispatch: sharding and concurrent workers reduce the need to process clusters in a single serial loop.
- Central credential storage: per-cluster Kubernetes credentials are retrieved from Key Vault rather than carried in each operator’s local workflow.
- Operational visibility: central audit data makes it easier to identify failed cluster operations and inspect what was requested.
- Pipeline integration: an optional Logic App HTTPS endpoint provides another way to trigger the Azure Automation job.
For example, an illustrative maintenance workflow could start with one non-production cluster, expand to a filtered non-production group, and only then target an approved production group. The filters support that targeting; staged approvals and health gates should not be assumed to be built-in features.
Choose the clusters—and the restart method
ClusterFilter accepts Resource Graph filter expressions. Examples from the project include:
- One cluster: | where name == ‘az-nuc-aks01’
- A specific group: | where name in (‘az-nuc-aks01’, ‘az-nuc-aks02’, ‘az-nuc-aks03’)
- A naming prefix: | where name startswith ‘az-nuc’
- An environment tag: | where tags.environment == ‘prod’
An empty filter targets all connected clusters. Confirm the discovered names and tags before submitting a broad restart request.
Controller-based restarts
For deployments, statefulsets and daemonsets, the worker Patches the named controller resource through the Kubernetes API. The request specifies Namespace, Workload Type and WorkloadName. This path triggers a controller-based rollout rather than directly deleting selected pods.
Pod-level restarts
Workload Type = pod uses a different approach: it finds pods by label selector and deletes matching managed pods. Here, Workload Name must be a selector—not a pod name. For example, app.kubernetes.io/name=dcgm-exporter targets pods carrying that label value.
Sequential deletion is the default, with PodDeleteDelayMs set to 2000 milliseconds between deletions. Parallel mode is also available. The worker skips already-terminating pods and pods with no owner References, and records warnings for unmanaged pods.
Important: direct pod deletion bypasses PodDisruptionBudget enforcement and the controller’s rollout strategy. The PDB check is advisory: a warning does not prevent deletion. A delay between deletions is not a readiness check, and this path provides no rollback. For quorum-sensitive StatefulSets, use a more specific selector to target one pod per run, as recommended in the README.
Security is part of the workflow
The service account permissions are scoped to the operations described in the project: get, list and patch for deployments, statefulsets and daemonsets; get, list and delete for namespaces and pods; and get and list for poddisruptionbudgets.
The supplied permissions do not allow creating resources, reading Kubernetes secrets, executing inside pods or modifying RBAC. However, delete permission is consequential and should not be described as harmless. Review the manifest and its scope before deploying it.
The Hybrid Worker’s managed identity needs Key Vault Secrets User access for token retrieval. Bootstrap requires an admin kubeconfig, and the project includes Rotate-ServiceAccountTokens.ps1 for annual token rotation.
If you enable the Logic App trigger, treat its URL as a secret because it contains a SAS token. The README suggests storing it in Key Vault or putting the endpoint behind API Management. Caller attribution also depends on how the request is submitted, so verify initiated By values in your audit workflow.
Getting started: a step-by-step how-to
The following walkthrough follows the project’s setup sequence. Run repository scripts from SA-Token-Pattern unless another location is noted. The resource, cluster and workload names are examples: adapt them to your environment. Start with a non-production target and confirm the intended scope before making changes.
Set your subscription ID in the PowerShell session used for the cleanup and infrastructure examples. This avoids publishing an environment-specific subscription ID in the post.
$subscriptionId = Read-Host “Enter your Azure subscription ID”
Step 0 — Clean up previous approach (if needed)
If you are replacing an earlier approach, preview the cleanup before proceeding. This command includes role-assignment removal, so review what it would affect.
.\scripts\Cleanup-AzureResources.ps1 `
-SubscriptionId $subscriptionId `
-RemoveRoleAssignments `
-WhatIf # preview first
Keep -WhatIf for the preview. Only remove it and rerun after confirming that the proposed changes are intended. Skip this step if there is no previous deployment to clean up.
Step 1 — Deploy Azure infrastructure
Deploy the Azure Automation account, Key Vault, Log Analytics workspace and supporting role assignments using the infrastructure script. This example uses the same subscription for infrastructure and managed-cluster discovery; set ManagedClusterSubs to your intended discovery scope if it differs.
.\infra\Deploy-Infrastructure.ps1 `
-SubscriptionId $subscriptionId `
-ResourceGroup “rg-aksarc-automation” `
-Location “australiaeast” `
-AutomationName “aa-aksarc-restart” `
-KeyVaultName “kv-aksarc-automation” `
-WorkspaceName “law-aksarc-restart” `
-HybridWorkerGroup “AKSArcWorkers” `
-ManagedClusterSubs $subscriptionId
Use resource names appropriate for your deployment. Deploy-Infrastructure.ps1 writes infra-state.json, which the optional Logic App deployment reads later.
Step 2 — Register Hybrid Worker(s)
Use an on-premises Windows machine on the same LAN as the clusters. Install the Azure Connected Machine agent and Arc-enroll the machine, then register it as a Hybrid Runbook Worker. Repeat this setup for each worker you intend to use.
$workerName = Read-Host “Enter the Hybrid Worker name”
az automation hrwg hrw create `
–resource-group rg-aksarc-automation `
–automation-account-name aa-aksarc-restart `
–hybrid-runbook-worker-group-name AKSArcWorkers `
–name $workerName `
–worker-type HybridV2
Grant Key Vault Secrets User to that worker’s managed identity. Supply the actual managed identity object ID and the full resource ID of your deployed vault—not a subscription-wide scope.
$workerIdentityObjectId = Read-Host “Enter the worker managed identity object ID”
$keyVaultScope = Read-Host “Enter the full Azure resource ID of the Key Vault”
az role assignment create `
–role “Key Vault Secrets User” `
–assignee-object-id $workerIdentityObjectId `
–scope $keyVaultScope
The worker identity retrieves credentials from Key Vault; the per-cluster service account token authenticates the Kubernetes API request.
Step 3 — Bootstrap service accounts (from Azure Local host)
Run the bootstrap script on the Azure Local Cluster host that manages the clusters. Choose either the single-cluster example or Auto Discover for the clusters on that host; you do not need to run both for the same setup.
# Single cluster: requires its admin kubeconfig
.\scripts\Bootstrap-ServiceAccount.ps1 `
-KeyVaultName “kv-aksarc-automation” `
-ClusterName “az-nuc-aks01” `
-KubeconfigPath “C:\admin-kubeconfigs\az-nuc-aks01.yaml”
# Alternative: discover all clusters managed by this HCI host
.\scripts\Bootstrap-ServiceAccount.ps1 `
-KeyVaultName “kv-aksarc-automation” `
-AutoDiscover
Bootstrap applies restart-sa.yaml and stores the service account token and API server address in Key Vault. Repeat the process on each relevant host. Keep admin kubeconfigs and service account tokens out of published examples and logs.
Step 4 — Verify on Hybrid Worker
Run the access test from the Hybrid Worker against one known cluster. Use the Key Vault and cluster names from your deployment.
.\scripts\Test-SATokenAccess.ps1 `
-KeyVaultName “kv-aksarc-automation” `
-ClusterName “az-nuc-aks01” `
-Namespace “default”
After verifying access, test an actual restart against a workload you have deliberately selected. The next command changes the target workload; replace the sample names with an appropriate non-production test target.
.\scripts\Test-SATokenAccess.ps1 `
-KeyVaultName “kv-aksarc-automation” `
-ClusterName “az-nuc-aks01” `
-Namespace “default” `
-TestRestart `
-WorkloadType “deployment” `
-WorkloadName “my-app”
Inspect the result and confirm workload health before moving beyond the test cluster.
Step 5 — Import runbooks and trigger fleet restart
Import the project’s runbooks through the Azure CLI or Azure Portal. Master-FleetRestart.ps1 orchestrates discovery and sharding; Shard-RestartPods.ps1 and Worker-RestartSingleCluster.ps1 execute the cluster operations on Hybrid Workers, using Shared-Functions.ps1.
The following illustrative trigger retains the production StatefulSet values from the README but adds a single-cluster filter. Change the namespace and workload to your approved target before running it. Initiated By is an illustrative audit label, not proof of an authenticated caller.
Start-AzAutomationRunbook `
-AutomationAccountName “aa-aksarc-restart” `
-ResourceGroupName “rg-aksarc-automation” `
-Name “Master-FleetRestart” `
-Parameters @{
Namespace = “production”
WorkloadType = “statefulset”
WorkloadName = “my-app”
ClusterFilter = “| where name == ‘az-nuc-aks01′”
InitiatedBy = “jasetech-blog-example”
}
Scope check: leaving ClusterFilter empty targets all connected clusters. For a broader approved run, replace the single-cluster expression with a specific cluster group or environment-tag filter from the targeting section. Confirm the discovery results before submission.
Step 6 — Option A (Optional) — Azure Workbook: select clusters and restart workloads from the portal
infra/workbook-definition.json is an Azure Monitor Workbook that gives operators a point-and-click interface for fleet restarts. No Logic App is required: the Start Fleet Restart button issues an ARM PUT that creates a Master-FleetRestart Automation job directly, using the signed-in user’s own permissions.
Deploy it
- In Azure Portal, open Monitor → Workbooks → New → Advanced Editor (</>).
- Paste the contents of infra/workbook-definition.json using Gallery Template type, click Apply, then Save to a resource group.
What it does
| Section | Purpose |
| Connection Settings | One-time entry of the Automation Account subscription, resource group, name, and the Log Analytics workspace. Save the workbook afterwards so the values persist. |
| Workload Target | Choose WorkloadType (Deployment, StatefulSet, DaemonSet, Pod by label selector), Namespace, WorkloadName, PodDeleteMode (Sequential / Parallel) and an optional InitiatedBy audit label. |
| Step 1 — Select Clusters | Resource Graph grid of all Connected microsoft.kubernetes/connectedclusters: name, resource group, subscription, environment tag, location and Kubernetes version. Filter and multi-select rows. Leave all rows unselected to target the whole fleet. |
| Step 2 — Review & Launch | Live summary of every parameter and the generated cluster filter, then ▶ Start Fleet Restart with a confirmation dialog. |
| Step 3 — Monitor Active Jobs | Per-job summary (clusters, OK, Failed, Skipped, Running) for the last 2 hours from AKSArcRestart_CL. |
| Step 4 — Audit History | Per-cluster results with time range and status filters, colour-coded status/HTTP code, duration heat-bar and error details. |
How selection works
- The selected cluster names are exported as SelectedClusterNames and converted into a Resource Graph fragment, for example: | where name in~ (‘az-nuc-aks01′,’az-nuc-aks02’).
- That fragment is passed to the runbook as -ClusterFilter. An empty selection produces an empty filter and targets the requested workload across all connected clusters.
- A fresh job GUID is generated for each run. Click the refresh icon on the parameter between restarts in the same session.
- Parameters sent: Namespace, WorkloadType, WorkloadName, ClusterFilter, InitiatedBy, PodDeleteMode.
Requirements
- Automation Operator (or higher) on the Automation Account for anyone who starts restarts.
- Reader on the subscriptions containing the clusters for the Resource Graph grid, and on the Log Analytics workspace.
- Results appear in AKSArcRestart_CL roughly 2 minutes after the job completes. Allow for this delay when checking the workbook.
First test
For the suggested first test, select a single cluster and restart the coredns Deployment in kube-system. Use a non-production cluster where this change is approved, review the generated filter and workload parameters, and confirm the single-cluster selection before launching.
For Pod by label selector, retain the pod-deletion safeguards discussed earlier: direct deletion bypasses PDB enforcement, and Sequential mode is not a readiness gate. Review the confirmation dialog and check job output, audit records and workload health after the run.
Option A launches the job directly under the signed-in user’s permissions. If you instead want an HTTP endpoint for operators or pipelines, use the existing Logic App approach in Option B below.
Step 6 — Option B (Optional) — Deploy Logic App HTTP trigger
The optional Logic App provides an HTTPS endpoint for operators or pipelines to trigger the Azure Automation workflow without opening the portal. Choose the default deployment or the custom-name alternative below.
# Run from SA-Token-Pattern/infra/
.\Deploy-LogicApp.ps1
# Alternative: deploy with a custom name
.\Deploy-LogicApp.ps1 -LogicAppName “logic-aksarc-restart-prod”
The script reads infra-state.json automatically. After deployment, the trigger URL is printed to the console and saved in infra/logic-app-state.json.
Protect the endpoint: the trigger URL contains a SAS token. Treat it as a secret, store it in Key Vault or protect access through API Management, and never publish the live URL in your blog or example scripts.
Example call
This PowerShell example submits the same single-cluster request used above. Supply your deployed trigger URL privately and replace the sample workload values with your approved target.
$triggerUrl = Read-Host “Enter the secret Logic App trigger URL”
$body = @{
namespace = “production”
workloadType = “statefulset”
workloadName = “my-app”
clusterFilter = “| where name == ‘az-nuc-aks01′”
initiatedBy = “jasetech-blog-example”
} | ConvertTo-Json
Invoke-RestMethod `
-Method Post `
-Uri $triggerUrl `
-ContentType “application/json” `
-Body $body
The response includes jobGuid, jobStatus, jobStartTime, jobEndTime, the requested namespace and workload, and the runbook output stream. The documented endpoint returns HTTP 200 when the Automation job completes, or HTTP 500 when it fails, stops or times out. The Logic App waits up to 35 minutes.
A completed Automation job is not, by itself, a guarantee that every application is healthy. Review the output and audit records, investigate failed cluster operations, and verify readiness before expanding the scope.
Start with a small, known target and inspect the Log Analytics records before expanding. For pod-delete runs, the audit queries include pods found, deleted and skipped, alongside status and duration.
Fast dispatch is not the same as recovery
The README includes performance estimates for REST calls and parallel fleet dispatch. These are estimates, not published benchmark results or an end-to-end recovery guarantee. Dispatching a restart request, recording a successful API operation and confirming application health are different outcomes.
Use the audit trail to inspect execution results, but verify workload readiness and application health separately before considering a maintenance operation complete. That distinction matters even more when a request spans many clusters.
One workflow, many clusters
mass-aks-pod-restart brings a specific operational task into a fleet-oriented workflow: discover the clusters, select the targets, authenticate per cluster, execute locally and report centrally.
For teams running AKS Arc on Azure Local, the value is not just being able to restart workloads at scale. It is making the operation more repeatable and visible, while keeping the differences between controller rollouts and direct pod deletion explicit.
If this matches the kind of fleet you manage, explore the implementation, review the permissions and start with a controlled test.
Repository: https://github.com/jasontayler/mass-aks-pod-restart
No responses yet