Stateful Counter

Preserve in-memory, root filesystem, and durable volume state across actor suspend and resume cycles.

⏱️ 5 mins · 6 min read ·

This demo runs a stateful HTTP server (counter.go) on Agent Substrate and shows how actor snapshots preserve state across suspend and resume cycles.

On every HTTP request, counter.go increments three counters:

  • preserved memory count in process memory
  • preserved file counter in /home/counter/a.txt on the data durableDir volume
  • preserved rootfs counter in /rootfs-counter.txt on the container root filesystem

The demo ActorTemplate sets preferredFidelity to SNAPSHOT_FIDELITY_MEMORY, which captures process memory, root filesystem writes, and durable volumes when an actor suspends. When the actor resumes, all three counters continue from where they left off. Under SNAPSHOT_FIDELITY_ROOTFS, the durable volume and root filesystem counters survive while the in-memory counter resets to 1. Under SNAPSHOT_FIDELITY_VOLUMES, only the durable volume counter survives.

The demo creates two resources:

  • WorkerPool (counter), a Kubernetes custom resource in the ate-demo-counter namespace.
  • ActorTemplate (counter), a Substrate API resource in the ate-demo-counter atespace, managed through kubectl ate.

Prerequisites

  • A Kubernetes cluster with Agent Substrate installed as in the Quickstart or Operator Install Guide. On kind, keep export ATE_INSTALL_KIND=true set in your shell so ate-setup targets the local cluster and registry.
  • atenet-router configured with --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client (ATE_INGRESS_ALLOWED_SPIFFE_IDS) so kubectl ate proxy actor can authenticate to the router.
  • A snapshot location set at install (SNAPSHOT_LOCATION). ATE_INSTALL_KIND=true sets this automatically to gs://ate-snapshots/.

How to Run on Agent Substrate

This walkthrough deploys the default gvisor sandbox variant (gvisor-default). See Running on microVM for the microvm variant.

The Quickstart already deploys this demo, creates my-counter-1, and starts its proxy on port 8000. If you ran it, skip to How to Use.

1. Build and Deploy

Note

Do not manually edit demos/counter/counter.yaml.tmpl or demos/counter/counter-template.yaml.tmpl. ate-setup fills in their ${...} placeholders during deployment.

Use ate-setup to build the image and deploy the demo to your cluster:

go run ./cmd/ate-setup deploy demo counter

This command:

  • Builds the counter server image using ko.
  • Applies counter.yaml.tmpl to create the ate-demo-counter namespace and the counter WorkerPool, then waits for the worker rollout.
  • Creates the ate-demo-counter atespace.
  • Creates the counter actor template through the Substrate API from counter-template.yaml.tmpl. The manifest uses the message’s protojson form, which matches kubectl ate get actor-template counter -a ate-demo-counter -o yaml.
  • Waits until the template’s golden snapshot is ready.

Inspect the deployed template:

kubectl ate get actor-templates -a ate-demo-counter
kubectl ate get actor-template counter -a ate-demo-counter -o yaml

2. Create a Counter Actor

Install the kubectl-ate CLI plugin and create an actor named my-counter-1 in the ate-demo-counter atespace from the counter template:

go install ./cmd/kubectl-ate
kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter

3. Proxy to the Actor

Start a local proxy that authenticates with atenet-router and forwards local port 8000 to port 80 on the actor:

kubectl ate proxy actor my-counter-1 -a ate-demo-counter

How to Use

When you send an HTTP request through the proxy, atenet-router resumes the actor onto an available worker pod and routes the traffic.

  1. In a separate terminal, send an HTTP POST request to increment the counters:
curl -X POST http://localhost:8000

Expected output:

preserved memory count: 1 | preserved file counter: 1 | preserved rootfs counter: 1
  1. Verify that the actor is in ACTOR_STATE_RUNNING and assigned to a worker pod:
kubectl ate get actor my-counter-1 -a ate-demo-counter
  1. Manually suspend the actor:
kubectl ate suspend actor my-counter-1 -a ate-demo-counter

The snapshot stays on the worker’s node and uploads to snapshot storage in the background. Pass --mode=sync to wait for the upload before returning.

Repeat the curl command from step 1:

curl -X POST http://localhost:8000

Expected output:

preserved memory count: 2 | preserved file counter: 2 | preserved rootfs counter: 2

The actor resumes from its snapshot, potentially on a different worker pod, and all three counters continue from 1 to 2. Run kubectl ate get actor my-counter-1 -a ate-demo-counter again and check the WORKER POD column to see which worker pod it resumed on.

Reaching a non-default port

counter.go also listens on a second configurable port (--extra-port=9090 in this demo’s template) to exercise arbitrary-port ingress through atenet-router. Pass [local-port:]remote-port arguments to kubectl ate proxy actor to forward additional ports. Stop any earlier proxy on port 8000 first:

kubectl ate proxy actor my-counter-1 8000:80 9090:9090 -a ate-demo-counter

Send a request to local port 9090:

curl http://localhost:9090/

Expected output:

hello from extra port 9090

This reaches the same actor’s second listener and resumes the actor if it is suspended.

Current limitation: only HTTP(S) traffic over the tunnel is supported. Raw TCP or other non-HTTP protocols on a non-default port are not reachable through atenet-router yet.

Delete the actor when finished:

kubectl ate delete actor my-counter-1 -a ate-demo-counter

Running on microVM

You can also run the counter demo on the microvm sandbox class (ateworker-microvm, which runs a Kata guest on Cloud Hypervisor and requires /dev/kvm on worker nodes). This variant uses:

deploy ate-system stages the microVM guest assets and applies the cluster-wide microvm SandboxConfig by default (--sandbox-classes=gvisor,microvm), so on a cluster installed as in the Quickstart, deploy the demo with:

go run ./cmd/ate-setup deploy demo counter-microvm

Inspect the deployed template:

kubectl ate get actor-templates -a ate-demo-counter-microvm
kubectl ate get actor-template counter-microvm -a ate-demo-counter-microvm -o yaml

Create, proxy, suspend, and delete the microVM actor in the ate-demo-counter-microvm atespace:

kubectl ate create actor my-counter-1 -a ate-demo-counter-microvm --template counter-microvm
kubectl ate proxy actor my-counter-1 -a ate-demo-counter-microvm

In a second terminal:

curl -X POST http://localhost:8000
kubectl ate get actor my-counter-1 -a ate-demo-counter-microvm
kubectl ate suspend actor my-counter-1 -a ate-demo-counter-microvm
curl -X POST http://localhost:8000
kubectl ate delete actor my-counter-1 -a ate-demo-counter-microvm

Attaching an External Volume

Pass --with-external-volume to attach a CSI-backed external volume mounted at /external-data. That flag sets --validate-existing-file-path=/external-data/test.txt on counter.go, so curl returns failed to read test file: open /external-data/test.txt: no such file or directory until /external-data/test.txt exists on the volume.

Important

External volumes are a preview feature and require the ExternalVolumes preview feature gate plus an adapted CSI driver with a CSIDriverConfig (see CSI Volumes and Deploying CSI Drivers):

# 1. Deploy ate-system with the ExternalVolumes preview gate enabled
go run ./cmd/ate-setup deploy ate-system \
  --preview=ExternalVolumes \
  --credential-provider='{"name":"k8s.io"}' \
  --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client

# 2. Install a bundled CSI driver and its CSIDriverConfig:
#    - NFS (creates StorageClass csi-nfs-sc, requires the nfsd kernel module on nodes):
go run ./cmd/ate-setup setup csi nfs
#    - Or hostpath on kind clusters (creates StorageClass csi-hostpath-sc):
# go run ./cmd/ate-setup setup csi hostpath

# 3. Delete any existing counter demo (templates are immutable), then deploy with the external volume
go run ./cmd/ate-setup delete demo counter
kubectl wait --for=delete namespace/ate-demo-counter --timeout=60s
go run ./cmd/ate-setup deploy demo counter \
  --with-external-volume \
  --storage-class=csi-nfs-sc \
  --preview=ExternalVolumes

# 4. Create the actor with --preview enabled on kubectl ate
kubectl ate --preview create actor my-counter-1 -a ate-demo-counter --template counter

Settings

The installer reads these settings in addition to those in docs/guides/operator-install.md, through the same flag, environment variable, and configuration document channels.

KeyEnvironment variableFlagDefaultApplies to
demo.counter.storageClassSTORAGE_CLASS--storage-classstandarddeploy demo counter
demo.counter.withExternalVolumeATE_DEMO_COUNTER_WITH_EXTERNAL_VOLUME--with-external-volumefalsedeploy demo counter

How to Uninstall

Remove the counter demo resources from your cluster:

go run ./cmd/ate-setup delete demo counter

# Or for the microVM variant:
go run ./cmd/ate-setup delete demo counter-microvm

This deletes the actors created from the demo templates, the templates themselves (along with their golden actors and golden snapshots), the demo atespaces, and the worker pools.

Was this page helpful?