What an Ansible Task Really Costs

The playbook is comfortable at 10 hosts. At 500, the same task list starts to crawl. The YAML did not get longer. The distance between the controller and each host did, and the scheduler now has 500 copies of every task to place.

Two questions explain most of the gap:

1. How many SSH operations does one module task require on one host?
2. When can the next host or task start?

The short answer is that a normal module task uses five SSH operations without pipelining and one with it. OpenSSH multiplexing makes each operation cheaper by reusing an authenticated connection. Forks, batches, per-task caps, and strategy plugins decide how those costs overlap across the fleet.

First, define the thing being counted

“Round trip” can mean three different things:

• A network RTT is one packet exchange between controller and host.
• A fresh SSH session includes TCP setup, SSH version exchange, key exchange, authentication, channel open, and an exec request. With pubkey authentication and one auth attempt, that is roughly 5 to 7 network RTTs before useful command output. The exact count depends on auth, OpenSSH, GSSAPI, proxies, and retries.
• An SSH operation is one ssh, sftp, or scp process launched by Ansible. A multiplexed operation opens a channel through a local Unix socket instead of repeating TCP, key exchange, and authentication.

This post counts SSH operations exactly from the ansible-core code path. RTT estimates use a stated model: 6 RTTs for a fresh session and 1.5 RTTs for a multiplexed operation. Those are protocol approximations, not captured timings.

Anatomy of one module task

For a normal new-style Python module, with pipelining off, no become, no async wrapper, and no extra payload file, ActionBase._execute_module takes this path:

That is 4 ssh processes plus 1 sftp process, per task, per host. The source trail runs from _execute_module through put_file and back through cleanup.

Watch the operation stack fill, then enable pipelining to collapse the task and its fleet cost.

The result being ok or changed does not select this transport path. The module has to run to discover that result. A clean file task and a changed file task both pay the module-delivery cost. Action plugins such as copy, template, unarchive, and fetch can add payload transfers of their own. Pipelining removes the module-code transfer, not a file that the task exists to copy.

The default SSH transfer method is smart: try SFTP, then SCP, then a piped dd transfer. SCP on new OpenSSH versions may require legacy mode. transfer_method = piped is useful for transfers that remain because it sends bytes through an SSH exec channel instead of negotiating the SFTP subsystem.

Look for sftp> put in the smart run and dd of= in the piped run.

The before and after count

Assume a warm ControlPersist master where the table says “steady state.”

The first operation for each host still establishes the master. Across a run with N hosts and M normal module tasks:

• No pipelining: 5 × N × M controller-side SSH/SFTP process launches.
• Pipelining: N × M launches.
• No multiplexing: every launch can pay a full handshake.
• ControlPersist: about N full handshakes while the masters stay alive.

At 500 hosts and 20 tasks, the operation count is 50,000 without pipelining and 10,000 with it. With multiplexing kept warm, the full-handshake count is about 500, one per host, rather than one per operation.

Use the calculator to scale the exact per-task operation count across a fleet and compare full handshakes with and without a warm ControlPersist master. It deliberately leaves scheduling and network-latency models to the sections that follow.

Look for the five highlighted SSH: EXEC operations, one per table row: the temp dir, the sftp put, chmod, the module run, and cleanup. The unhighlighted echo ~ before them is a one-time home-directory probe, not part of the per-task count.

This -vvvv run against a real node makes the five operations countable without rewriting Ansible’s output. Each highlighted SSH: EXEC line is one operation: the temp dir, the SFTP put, chmod, the Python execution, and cleanup. Without a warm master, every one of them opens its own SSH connection.

Reusing the connection

Ansible’s SSH plugin does not keep a Python connection object open. Its own source says connection management is unnecessary because exec_command, put_file, and fetch_file run external ssh/scp/sftp processes. OpenSSH owns reuse.

The default SSH arguments are:

-C -o ControlMaster=auto -o ControlPersist=60s

The first client for a (host, port, user) tuple becomes the master. Later clients connect to its Unix socket and request another SSH channel over the existing encrypted connection. TCP setup, key exchange, and authentication do not repeat.

When ControlPersist appears without a ControlPath, Ansible creates ~/.ansible/cp and hashes the host, port, and user into a 10-character socket name. The hash avoids the Unix socket path-length limit that used to break long hostnames. If you set a path yourself, keep it short. OpenSSH’s %C token is the safe default shape:

[ssh_connection]
control_path = %(directory)s/%%C

Two details cause expensive surprises:

• Setting ssh_args replaces Ansible’s default string. Adding only -o ServerAliveInterval=30 silently removes ControlMaster and ControlPersist.
• ControlPersist=60s expires 60 seconds after the last channel closes. A long controller-side pause can make every host pay another handshake at the next task.

For repeated development runs, a longer value can keep masters useful:

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=10m

One master exists per host, port, and user tuple. A very large inventory therefore consumes file descriptors and processes on the controller. A bastion can also become the choke point: its MaxStartups limit applies while a wave of new masters is still unauthenticated.

Look for the highlighted auto-mux: Trying existing master and mux_client_request_session on the second run, then the socket listing and the visible ssh -O check result.

The two visible ping commands run against the same real node. The second keeps OpenSSH’s auto-mux: Trying existing master and mux_client_request_session evidence. The final ls -la and ssh -O check steps show the socket and confirm that its master is live.

Pipelining removes the file path

With pipelining enabled, ansible-core sends the AnsiballZ payload to remote Python over standard input. The temp directory, module transfer, chmod, and cleanup branches are skipped. One SSH exec operation remains.

Enable it in any of these scopes:

[defaults]
pipelining = True

[ssh_connection]
pipelining = True

web:
  vars:
    ansible_pipelining: true

The environment forms are ANSIBLE_PIPELINING=1 and the older ANSIBLE_SSH_PIPELINING=1 alias.

Pipelining is off by default because it can conflict with privilege escalation on hosts whose sudo policy requires a TTY. Ansible cannot force a TTY while also feeding the Python module through standard input. Most current distributions do not enable requiretty, but the fleet’s sudo policy is the fact that matters.

become by itself does not disable pipelining. The selected connection and become plugins must permit it, and the remote sudo policy must accept a non-TTY invocation. Escalating to another unprivileged user can also add permission-fix operations: setfacl, chown, macOS ACL syntax, or chgrp fallbacks may each require another remote exec when earlier choices fail.

Look for one highlighted SSH: EXEC and no separate SFTP, chmod, or cleanup.

The paired capture keeps the single SSH establishment line and the module command reading from stdin. Next to the cold run there is no SFTP put, no AnsiballZ temp-file execution, no chmod, and no cleanup.

These are measured seconds, not a model. The same 12-task play runs against the same five hosts with five forks, changing only ControlPersist and pipelining. Multiplexing and pipelining each cut the time, and together they turn a 17 second run into under 5 seconds. A second pass held the same ordering, so read the numbers as a stable range rather than a single figure.

Scheduling the fleet

Transport cost answers how much work exists. Scheduling answers how much can overlap.

forks is the controller’s worker-pool limit. The default is 5. Each worker handles one (host, task) pair at a time, so forks = 50 means at most 50 simultaneous host-task executions, not 50 threads per host. Raising it consumes more controller memory and can create a burst of SSH handshakes when no masters are warm.

Four other controls narrow or reorder that pool:

• serial divides the play into rolling host batches. It accepts a number, a percentage, or a list such as [1, 5, "25%"]. The whole play runs once per batch. Handlers and run_once tasks therefore have batch scope, and failure scope shrinks to the batch.
• throttle caps one task or block below the current forks and batch size. It cannot add capacity. Use it around an API, package mirror, load balancer, or database operation that cannot absorb a full fleet wave.
• run_once schedules a task on the first live host in the current batch and copies the result to the other hosts in that batch. With serial, it runs once per batch. This is a linear behavior: under free or host_pinned, ansible-core warns that run_once is not supported and runs the task on every host.
• delegate_to chooses where that one execution happens. A database migration or API call often belongs on localhost or a specific utility host.

For an action that must run once across the entire play, regardless of serial batches, make the condition explicit:

- name: Apply the schema migration once for the whole play
  ansible.builtin.command: /opt/app/migrate
  delegate_to: db-admin-01
  when: inventory_hostname == ansible_play_hosts_all[0]

The visualizer’s units are schematic. Turn on the slow web-07 model, then switch strategies:

• linear is the default. Every host in the batch reaches the current task barrier before any host advances. One slow host can stretch every barrier.
• free lets a ready host take its next task. The slow host delays itself while other hosts keep using available forks. Output order becomes harder to read, and cross-host assumptions need scrutiny.
• host_pinned behaves like free with slot ownership. Up to forks hosts start, each keeps its slot through the whole play, and a completed host admits the next waiting host. This favors per-host locality and keeps a host’s connection active through adjacent tasks.

The two useful mental equations are:

linear total       ≈ sum over tasks(max host time for that task)
free total         ≈ max over hosts(sum of that host's task times)

They describe the barrier shape, not exact Ansible wall time. serial wraps either strategy in batch boundaries, and effective concurrency is always bounded by the smallest active limit:

min(forks, current serial batch size, task throttle)

async deserves a separate warning. It is useful when a long remote job should release a fork, especially with poll: 0. On SSH it disables pipelining and transfers an async wrapper, so it is not a shortcut for cheap tasks.

These are real SSH runs across ten hosts, so the schedule carries true per-host connection cost. Under linear the fleet moves in lockstep, and the slow web-07 stretches every barrier. Under free, ready hosts take their next task and the banners interleave.

Each PLAY banner is one serial batch: the play re-runs for one host, then two, then the rest. run_once and throttle narrow concurrency inside a play instead of across batches.

Facts are work too

Implicit fact gathering runs the setup module at the start of every play for every host. That is a module transport plus remote probing of hardware, mounts, packages, and networking. It can be one of the largest AnsiballZ payloads in a common play.

Start by asking whether the play uses facts:

- name: Restart a service without host facts
  hosts: web
  gather_facts: false

When templates need only a small set, narrow the probe and bound it:

gather_facts: true
gather_subset:
  - '!all'
  - '!min'
  - network
gather_timeout: 10

For repeated runs, gathering = smart skips hosts already gathered within the current run. A persistent cache can carry facts between runs:

[defaults]
gathering = smart
fact_caching = jsonfile
fact_caching_connection = ~/.ansible/factcache
fact_caching_timeout = 86400

The default memory cache does not survive the process. Persistent facts also become stale data, so the timeout and refresh schedule should match how quickly the fleet changes.

These seconds are capture data, not estimates. Gathering disabled is the floor. A narrowed network subset costs less than the full probe. The first gathering = smart run pays to fill the cache, and the next run reads it back and lands close to the no-facts floor.

A tuning order for many nodes

1. Measure normal runs without -vvvv. Enable ansible.posix.profile_tasks and ansible.posix.timer. High verbosity creates its own controller work.
2. Raise forks until a real limit appears. Watch controller CPU and memory, network capacity, bastion startup limits, and service-side rate limits.
3. Enable pipelining where sudo policy permits it. This is the direct five-to-one reduction for ordinary module tasks.
4. Keep multiplexing. Do not replace ssh_args without restoring ControlMaster and ControlPersist. Lengthen persistence when gaps or repeated runs justify it.
5. Reduce fact work. Disable unused gathering, trim subsets, or use a cache with an honest expiration policy.
6. Pin the Python interpreter when the path is known. This removes the first-contact discovery exec and its warning.
7. Choose strategy and batch size from dependency and risk. Use linear where hosts must meet at barriers. Use free or host_pinned when hosts can progress independently. Keep serial as wide as the rollout can safely tolerate.
8. Cap only the sensitive task. A task-level throttle is cheaper than shrinking the whole play.
9. Remove tasks. Every ok module invocation still costs at least one SSH operation. Pass lists to package modules and prefer one complete template over many line edits when the ownership model allows it.

[defaults]
forks = 50
pipelining = True
gathering = smart
fact_caching = jsonfile
fact_caching_connection = ~/.ansible/factcache
fact_caching_timeout = 86400
callbacks_enabled = ansible.posix.profile_tasks, ansible.posix.timer

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=10m
transfer_method = piped

The inventory can remove interpreter discovery when the path is stable:

web:
  vars:
    ansible_python_interpreter: /usr/bin/python3

Connection reuse and pipelining solve different halves of the same equation. ControlPersist lowers the cost of each operation. Pipelining removes operations. Scheduling decides whether one slow host becomes the fleet’s clock.

The next time a 10-host play turns into a 500-host wait, count the operations first. Then look for the barrier.

Source trail

The code-path claims in this post use ansible-core 2.22.0.dev0 at commit 7d7b8d9. Shipping versions can differ, especially around SSH verbosity and removed compatibility options.

• SSH connection defaults and command construction: lib/ansible/plugins/connection/ssh.py
• Module staging, pipelining, permission fixes, execution, and cleanup: lib/ansible/plugins/action/__init__.py
• Worker queue and task throttle: lib/ansible/plugins/strategy/__init__.py
• Strategy implementations: linear.py, free.py, and host_pinned.py
• Serial batch construction: lib/ansible/executor/playbook_executor.py