Failed to create temporary directory when running win_template module against Windows Server 2019

Question

Failed to create temporary directory when running win_template module against Windows Server 2019

dwierima-aspentech opened this issue 4 months ago · 10 comments

dwierima-aspentech commented 4 months ago

SUMMARY

Running win_template module against a Windows 2019 Server results in a 'Failed to create temporary directory' error. This same behavior is not exhibited by Windows Server 2022, Windows desktop 10 or 11, when running the exact same play. I believe this is due to win_template attempting to create a temporary directory that was already created in a previous play.
For instance in my most recent testing I see the following path in stdout of the win_template play, and the previous win_path play: "C:\Users\username\AppData\Local\Temp\ansible-tmp-1714509461.069503-301-253568139436398"
However, when I decode the same play when run against Windows server 2022, ansible doesn't even appear to be creating a temporary directory.

I have a lot of plays in this playbook, and I have attempted shuffling around the order that the plays in the playbook get called, and adding a 5 second pause prior to running the win_template play, however the win_template play always throws the error. Which is why I believe that this error is caused by win_template.

ISSUE TYPE

Bug Report

COMPONENT NAME

win_template

ANSIBLE VERSION

ansible [core 2.16.5]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/var/jenkins_home/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /var/jenkins_home/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

Collection      Version
--------------- -------
ansible.windows 2.3.0

CONFIGURATION

ANSIBLE_FORCE_COLOR = False
ANSIBLE_NOCOLOR = True
DEFAULT_STDOUT_CALLBACK = yaml
DISPLAY_ARGS_TO_STDOUT = False
DISPLAY_SKIPPED_HOSTS = False
HOST_KEY_CHECKING = False
PAGER(env: PAGER) = less
PERSISTENT_COMMAND_TIMEOUT = 120
PERSISTENT_CONNECT_TIMEOUT = 120

OS / ENVIRONMENT

Ansible Controller: Debian 12
Target server: Windows Server 2019

STEPS TO REPRODUCE

- name: Setup ntp.conf
  ansible.windows.win_template:
    src: ntp_server.conf
    dest: C:\Program Files (x86)\NTP\etc\ntp.conf

EXPECTED RESULTS

Running the above play produces the following output on Windows Server 2022, where the file is templeted out correctly.

TASK [Setup ntp.conf] ***********
task path: 
<path>/install_MeinbergNTP.yml:45
ok: [<hostname>] => changed=false 
  checksum: 200e5d180dcc7281386bc4d14552bce6fce8ef11
  dest: C:\Program Files (x86)\NTP\etc\ntp.conf
  operation: file_copy
  original_basename: ntp_server.conf
  size: 500
  src: /var/jenkins_home/.ansible/tmp/ansible-local-312oknt2381/tmp41by5m38/ntp_server.conf
Using module file /usr/lib/python3/dist-packages/ansible_collections/ansible/windows/plugins/modules/win_service.ps1
Pipelining is enabled.
<ip> ESTABLISH SSH CONNECTION FOR USER: <username>
<ip> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="<username>"' -o ConnectTimeout=10 -o 'ControlPath="/var/jenkins_home/.ansible/cp/c3a9fef035"' <ip> 'chcp.com 65001 >nul 2>&1 && PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -EncodedCommand UABvAHcAZQByAFMAaABlAGwAbAAgAC0ATgBvAFAAcgBvAGYAaQBsAGUAIAAtAE4AbwBuAEkAbgB0AGUAcgBhAGMAdABpAHYAZQAgAC0ARQB4AGUAYwB1AHQAaQBvAG4AUABvAGwAaQBjAHkAIABVAG4AcgBlAHMAdAByAGkAYwB0AGUAZAAgAC0ARQBuAGMAbwBkAGUAZABDAG8AbQBtAGEAbgBkACAASgBnAEIAagBBAEcAZwBBAFkAdwBCAHcAQQBDADQAQQBZAHcAQgB2AEEARwAwAEEASQBBAEEAMgBBAEQAVQBBAE0AQQBBAHcAQQBEAEUAQQBJAEEAQQArAEEAQwBBAEEASgBBAEIAdQBBAEgAVQBBAGIAQQBCAHMAQQBBAG8AQQBhAFEAQgBtAEEAQwBBAEEASwBBAEEAawBBAEYAQQBBAFUAdwBCAFcAQQBHAFUAQQBjAGcAQgB6AEEARwBrAEEAYgB3AEIAdQBBAEYAUQBBAFkAUQBCAGkAQQBHAHcAQQBaAFEAQQB1AEEARgBBAEEAVQB3AEIAVwBBAEcAVQBBAGMAZwBCAHoAQQBHAGsAQQBiAHcAQgB1AEEAQwBBAEEATABRAEIAcwBBAEgAUQBBAEkAQQBCAGIAQQBGAFkAQQBaAFEAQgB5AEEASABNAEEAYQBRAEIAdgBBAEcANABBAFgAUQBBAGkAQQBEAE0AQQBMAGcAQQB3AEEAQwBJAEEASwBRAEEAZwBBAEgAcwBBAEMAZwBBAG4AQQBIAHMAQQBJAGcAQgBtAEEARwBFAEEAYQBRAEIAcwBBAEcAVQBBAFoAQQBBAGkAQQBEAG8AQQBkAEEAQgB5AEEASABVAEEAWgBRAEEAcwBBAEMASQBBAGIAUQBCAHoAQQBHAGMAQQBJAGcAQQA2AEEAQwBJAEEAUQBRAEIAdQBBAEgATQBBAGEAUQBCAGkAQQBHAHcAQQBaAFEAQQBnAEEASABJAEEAWgBRAEIAeABBAEgAVQBBAGEAUQBCAHkAQQBHAFUAQQBjAHcAQQBnAEEARgBBAEEAYgB3AEIAMwBBAEcAVQBBAGMAZwBCAFQAQQBHAGcAQQBaAFEAQgBzAEEARwB3AEEASQBBAEIAMgBBAEQATQBBAEwAZwBBAHcAQQBDAEEAQQBiAHcAQgB5AEEAQwBBAEEAYgBnAEIAbABBAEgAYwBBAFoAUQBCAHkAQQBDAEkAQQBmAFEAQQBuAEEAQQBvAEEAWgBRAEIANABBAEcAawBBAGQAQQBBAGcAQQBEAEUAQQBDAGcAQgA5AEEAQQBvAEEASgBBAEIAbABBAEgAZwBBAFoAUQBCAGoAQQBGADgAQQBkAHcAQgB5AEEARwBFAEEAYwBBAEIAdwBBAEcAVQBBAGMAZwBCAGYAQQBIAE0AQQBkAEEAQgB5AEEAQwBBAEEAUABRAEEAZwBBAEMAUQBBAGEAUQBCAHUAQQBIAEEAQQBkAFEAQgAwAEEAQwBBAEEAZgBBAEEAZwBBAEUAOABBAGQAUQBCADAAQQBDADAAQQBVAHcAQgAwAEEASABJAEEAYQBRAEIAdQBBAEcAYwBBAEMAZwBBAGsAQQBIAE0AQQBjAEEAQgBzAEEARwBrAEEAZABBAEIAZgBBAEgAQQBBAFkAUQBCAHkAQQBIAFEAQQBjAHcAQQBnAEEARAAwAEEASQBBAEEAawBBAEcAVQBBAGUAQQBCAGwAQQBHAE0AQQBYAHcAQgAzAEEASABJAEEAWQBRAEIAdwBBAEgAQQBBAFoAUQBCAHkAQQBGADgAQQBjAHcAQgAwAEEASABJAEEATABnAEIAVABBAEgAQQBBAGIAQQBCAHAAQQBIAFEAQQBLAEEAQgBBAEEAQwBnAEEASQBnAEIAZwBBAEQAQQBBAFkAQQBBAHcAQQBHAEEAQQBNAEEAQgBnAEEARABBAEEASQBnAEEAcABBAEMAdwBBAEkAQQBBAHkAQQBDAHcAQQBJAEEAQgBiAEEARgBNAEEAZABBAEIAeQBBAEcAawBBAGIAZwBCAG4AQQBGAE0AQQBjAEEAQgBzAEEARwBrAEEAZABBAEIAUABBAEgAQQBBAGQAQQBCAHAAQQBHADgAQQBiAGcAQgB6AEEARgAwAEEATwBnAEEANgBBAEYASQBBAFoAUQBCAHQAQQBHADgAQQBkAGcAQgBsAEEARQBVAEEAYgBRAEIAdwBBAEgAUQBBAGUAUQBCAEYAQQBHADQAQQBkAEEAQgB5AEEARwBrAEEAWgBRAEIAegBBAEMAawBBAEMAZwBCAEoAQQBHAFkAQQBJAEEAQQBvAEEAQwAwAEEAYgBnAEIAdgBBAEgAUQBBAEkAQQBBAGsAQQBIAE0AQQBjAEEAQgBzAEEARwBrAEEAZABBAEIAZgBBAEgAQQBBAFkAUQBCAHkAQQBIAFEAQQBjAHcAQQB1AEEARQB3AEEAWgBRAEIAdQBBAEcAYwBBAGQAQQBCAG8AQQBDAEEAQQBMAFEAQgBsAEEASABFAEEASQBBAEEAeQBBAEMAawBBAEkAQQBCADcAQQBDAEEAQQBkAEEAQgBvAEEASABJAEEAYgB3AEIAMwBBAEMAQQBBAEkAZwBCAHAAQQBHADQAQQBkAGcAQgBoAEEARwB3AEEAYQBRAEIAawBBAEMAQQBBAGMAQQBCAGgAQQBIAGsAQQBiAEEAQgB2AEEARwBFAEEAWgBBAEEAaQBBAEMAQQBBAGYAUQBBAEsAQQBGAE0AQQBaAFEAQgAwAEEAQwAwAEEAVgBnAEIAaABBAEgASQBBAGEAUQBCAGgAQQBHAEkAQQBiAEEAQgBsAEEAQwBBAEEATABRAEIATwBBAEcARQBBAGIAUQBCAGwAQQBDAEEAQQBhAGcAQgB6AEEARwA4AEEAYgBnAEIAZgBBAEgASQBBAFkAUQBCADMAQQBDAEEAQQBMAFEAQgBXAEEARwBFAEEAYgBBAEIAMQBBAEcAVQBBAEkAQQBBAGsAQQBIAE0AQQBjAEEAQgBzAEEARwBrAEEAZABBAEIAZgBBAEgAQQBBAFkAUQBCAHkAQQBIAFEAQQBjAHcAQgBiAEEARABFAEEAWABRAEEASwBBAEMAUQBBAFoAUQBCADQAQQBHAFUAQQBZAHcAQgBmAEEASABjAEEAYwBnAEIAaABBAEgAQQBBAGMAQQBCAGwAQQBIAEkAQQBJAEEAQQA5AEEAQwBBAEEAVwB3AEIAVABBAEcATQBBAGMAZwBCAHAAQQBIAEEAQQBkAEEAQgBDAEEARwB3AEEAYgB3AEIAagBBAEcAcwBBAFgAUQBBADYAQQBEAG8AQQBRAHcAQgB5AEEARwBVAEEAWQBRAEIAMABBAEcAVQBBAEsAQQBBAGsAQQBIAE0AQQBjAEEAQgBzAEEARwBrAEEAZABBAEIAZgBBAEgAQQBBAFkAUQBCAHkAQQBIAFEAQQBjAHcAQgBiAEEARABBAEEAWABRAEEAcABBAEEAbwBBAEoAZwBBAGsAQQBHAFUAQQBlAEEAQgBsAEEARwBNAEEAWAB3AEIAMwBBAEgASQBBAFkAUQBCAHcAQQBIAEEAQQBaAFEAQgB5AEEAQQA9AD0A'
<ip> (0, b'{"changed":true,"invocation":{"module_args":{"failure_reset_period_sec":null,"desktop_interact":false,"failure_command":null,"dependency_action":"set","username":null,"state":"restarted","path":null,"description":null,"sid_info":null,"start_mode":null,"load_order_group":null,"dependencies":null,"required_privileges":null,"force_dependent_services":false,"error_control":null,"failure_actions":null,"failure_actions_on_non_crash_failure":null,"pre_shutdown_timeout_ms":null,"password":null,"service_type":null,"display_name":null,"failure_reboot_msg":null,"name":"ntp","update_password":null}},"exists":true,"dependencies":["Tcpip"],"description":"Synchronizes the local system clock to a reference time source and (eventually) provide this time to NTP and/or SNTP clients","desktop_interact":false,"display_name":"Network Time Protocol Daemon","name":"NTP","path":"\\"C:\\\\Program Files (x86)\\\\NTP\\\\bin\\\\ntpd.exe\\" -U 3 -M -g -c \\"C:\\\\Program Files (x86)\\\\NTP\\\\etc\\\\ntp.conf\\"","start_mode":"auto","username":"LocalSystem","state":"running","can_pause_and_continue":false,"depended_by":[]}\r\n\r\n', b'OpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname <ip> is address\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/var/jenkins_home/.ssh/known_hosts\'\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/var/jenkins_home/.ssh/known_hosts2\'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 325\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\n#< CLIXML\r\n<Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><Obj S="progress" RefId="0"><TN RefId="0"><T>System.Management.Automation.PSCustomObject</T><T>System.Object</T></TN><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj><Obj S="progress" RefId="1"><TNRef RefId="0" /><MS><I64 N="SourceId">2</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj></Objs>debug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')

ACTUAL RESULTS

Running the same play as above produces the following output on Windows Server 2019, where an unreachable error occurs.
I included both the play which produces an error and the previous play to show that they both create the same temp directory.

TASK [Set path variable] ********
ok: [<hostname>] => changed=false 
  invocation:
    module_args:
      elements:
      - '%ProgramFiles(x86)%\NTP\bin'
      name: PATH
      scope: machine
      state: present
  path_value: '%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;%SYSTEMROOT%\System32\OpenSSH\;%ProgramFiles(x86)%\NTP\bin;C:\ProgramData\chocolatey\bin'
<ip> ESTABLISH SSH CONNECTION FOR USER: <username>
<ip> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="<username>"' -o ConnectTimeout=10 -o 'ControlPath="/var/jenkins_home/.ansible/cp/62d3c0d32f"' <ip> 'chcp.com 65001 >nul 2>&1 && PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -EncodedCommand UABvAHcAZQByAFMAaABlAGwAbAAgAC0ATgBvAFAAcgBvAGYAaQBsAGUAIAAtAE4AbwBuAEkAbgB0AGUAcgBhAGMAdABpAHYAZQAgAC0ARQB4AGUAYwB1AHQAaQBvAG4AUABvAGwAaQBjAHkAIABVAG4AcgBlAHMAdAByAGkAYwB0AGUAZAAgAC0ARQBuAGMAbwBkAGUAZABDAG8AbQBtAGEAbgBkACAAVQB3AEIAbABBAEgAUQBBAEwAUQBCAFQAQQBIAFEAQQBjAGcAQgBwAEEARwBNAEEAZABBAEIATgBBAEcAOABBAFoAQQBCAGwAQQBDAEEAQQBMAFEAQgBXAEEARwBVAEEAYwBnAEIAegBBAEcAawBBAGIAdwBCAHUAQQBDAEEAQQBUAEEAQgBoAEEASABRAEEAWgBRAEIAegBBAEgAUQBBAEMAZwBBAGsAQQBIAFEAQQBiAFEAQgB3AEEARgA4AEEAYwBBAEIAaABBAEgAUQBBAGEAQQBBAGcAQQBEADAAQQBJAEEAQgBiAEEARgBNAEEAZQBRAEIAegBBAEgAUQBBAFoAUQBCAHQAQQBDADQAQQBSAFEAQgB1AEEASABZAEEAYQBRAEIAeQBBAEcAOABBAGIAZwBCAHQAQQBHAFUAQQBiAGcAQgAwAEEARgAwAEEATwBnAEEANgBBAEUAVQBBAGUAQQBCAHcAQQBHAEUAQQBiAGcAQgBrAEEARQBVAEEAYgBnAEIAMgBBAEcAawBBAGMAZwBCAHYAQQBHADQAQQBiAFEAQgBsAEEARwA0AEEAZABBAEIAVwBBAEcARQBBAGMAZwBCAHAAQQBHAEUAQQBZAGcAQgBzAEEARwBVAEEAYwB3AEEAbwBBAEMAYwBBAEoAUQBCAFUAQQBFAFUAQQBUAFEAQgBRAEEAQwBVAEEASgB3AEEAcABBAEEAbwBBAEoAQQBCADAAQQBHADAAQQBjAEEAQQBnAEEARAAwAEEASQBBAEIATwBBAEcAVQBBAGQAdwBBAHQAQQBFAGsAQQBkAEEAQgBsAEEARwAwAEEASQBBAEEAdABBAEYAUQBBAGUAUQBCAHcAQQBHAFUAQQBJAEEAQgBFAEEARwBrAEEAYwBnAEIAbABBAEcATQBBAGQAQQBCAHYAQQBIAEkAQQBlAFEAQQBnAEEAQwAwAEEAVQBBAEIAaABBAEgAUQBBAGEAQQBBAGcAQQBDAFEAQQBkAEEAQgB0AEEASABBAEEAWAB3AEIAdwBBAEcARQBBAGQAQQBCAG8AQQBDAEEAQQBMAFEAQgBPAEEARwBFAEEAYgBRAEIAbABBAEMAQQBBAEoAdwBCAGgAQQBHADQAQQBjAHcAQgBwAEEARwBJAEEAYgBBAEIAbABBAEMAMABBAGQAQQBCAHQAQQBIAEEAQQBMAFEAQQB4AEEARABjAEEATQBRAEEAMABBAEQAVQBBAE0AQQBBADUAQQBEAFEAQQBOAGcAQQB4AEEAQwA0AEEATQBBAEEAMgBBAEQAawBBAE4AUQBBAHcAQQBEAE0AQQBMAFEAQQB6AEEARABBAEEATQBRAEEAdABBAEQASQBBAE4AUQBBAHoAQQBEAFUAQQBOAGcAQQA0AEEARABFAEEATQB3AEEANQBBAEQAUQBBAE0AdwBBADIAQQBEAE0AQQBPAFEAQQA0AEEAQwBjAEEAQwBnAEIAWABBAEgASQBBAGEAUQBCADAAQQBHAFUAQQBMAFEAQgBQAEEASABVAEEAZABBAEIAdwBBAEgAVQBBAGQAQQBBAGcAQQBDADAAQQBTAFEAQgB1AEEASABBAEEAZABRAEIAMABBAEUAOABBAFkAZwBCAHEAQQBHAFUAQQBZAHcAQgAwAEEAQwBBAEEASgBBAEIAMABBAEcAMABBAGMAQQBBAHUAQQBFAFkAQQBkAFEAQgBzAEEARwB3AEEAVABnAEIAaABBAEcAMABBAFoAUQBBAEsAQQBFAGsAQQBaAGcAQQBnAEEAQwBnAEEATABRAEIAdQBBAEcAOABBAGQAQQBBAGcAQQBDAFEAQQBQAHcAQQBwAEEAQwBBAEEAZQB3AEEAZwBBAEUAawBBAFoAZwBBAGcAQQBDAGcAQQBSAHcAQgBsAEEASABRAEEATABRAEIAVwBBAEcARQBBAGMAZwBCAHAAQQBHAEUAQQBZAGcAQgBzAEEARwBVAEEASQBBAEIATQBBAEUARQBBAFUAdwBCAFUAQQBFAFUAQQBXAEEAQgBKAEEARgBRAEEAUQB3AEIAUABBAEUAUQBBAFIAUQBBAGcAQQBDADAAQQBSAFEAQgB5AEEASABJAEEAYgB3AEIAeQBBAEUARQBBAFkAdwBCADAAQQBHAGsAQQBiAHcAQgB1AEEAQwBBAEEAVQB3AEIAcABBAEcAdwBBAFoAUQBCAHUAQQBIAFEAQQBiAEEAQgA1AEEARQBNAEEAYgB3AEIAdQBBAEgAUQBBAGEAUQBCAHUAQQBIAFUAQQBaAFEAQQBwAEEAQwBBAEEAZQB3AEEAZwBBAEcAVQBBAGUAQQBCAHAAQQBIAFEAQQBJAEEAQQBrAEEARQB3AEEAUQBRAEIAVABBAEYAUQBBAFIAUQBCAFkAQQBFAGsAQQBWAEEAQgBEAEEARQA4AEEAUgBBAEIARgBBAEMAQQBBAGYAUQBBAGcAQQBFAFUAQQBiAEEAQgB6AEEARwBVAEEASQBBAEIANwBBAEMAQQBBAFoAUQBCADQAQQBHAGsAQQBkAEEAQQBnAEEARABFAEEASQBBAEIAOQBBAEMAQQBBAGYAUQBBAD0A'
<ip> (1, b'C:\\Users\\<username>\\AppData\\Local\\Temp\\ansible-tmp-1714509461.069503-301-253568139436398\r\n', b'OpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname <ip> is address\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/var/jenkins_home/.ssh/known_hosts\'\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/var/jenkins_home/.ssh/known_hosts2\'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 268\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\n#< CLIXML\r\n<Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><Obj S="progress" RefId="0"><TN RefId="0"><T>System.Management.Automation.PSCustomObject</T><T>System.Object</T></TN><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj><Obj S="progress" RefId="1"><TNRef RefId="0" /><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj></Objs>#< CLIXML\r\n<Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><S S="Error">Program \'powershell.exe\' failed to run: Win32 internal error "Access is denied" 0x5 occurred while reading the console _x000D__x000A_</S><S S="Error">output buffer. Contact Microsoft Customer Support Services.At line:1 char:1_x000D__x000A_</S><S S="Error">+ PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted - ..._x000D__x000A_</S><S S="Error">+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~._x000D__x000A_</S><S S="Error">At line:1 char:1_x000D__x000A_</S><S S="Error">+ PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted - ..._x000D__x000A_</S><S S="Error">+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~_x000D__x000A_</S><S S="Error">    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException_x000D__x000A_</S><S S="Error">    + FullyQualifiedErrorId : NativeCommandFailed_x000D__x000A_</S><S S="Error"> _x000D__x000A_</S></Objs>debug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n')
<ip> Failed to connect to the host via ssh: OpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname <ip> is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/var/jenkins_home/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/var/jenkins_home/.ssh/known_hosts2'
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 268
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
#< CLIXML
<Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><Obj S="progress" RefId="0"><TN RefId="0"><T>System.Management.Automation.PSCustomObject</T><T>System.Object</T></TN><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj><Obj S="progress" RefId="1"><TNRef RefId="0" /><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj></Objs>#< CLIXML
<Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><S S="Error">Program 'powershell.exe' failed to run: Win32 internal error "Access is denied" 0x5 occurred while reading the console _x000D__x000A_</S><S S="Error">output buffer. Contact Microsoft Customer Support Services.At line:1 char:1_x000D__x000A_</S><S S="Error">+ PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted - ..._x000D__x000A_</S><S S="Error">+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~._x000D__x000A_</S><S S="Error">At line:1 char:1_x000D__x000A_</S><S S="Error">+ PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted - ..._x000D__x000A_</S><S S="Error">+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~_x000D__x000A_</S><S S="Error">    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException_x000D__x000A_</S><S S="Error">    + FullyQualifiedErrorId : NativeCommandFailed_x000D__x000A_</S><S S="Error"> _x000D__x000A_</S></Objs>debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 1

TASK [Setup ntp.conf] ***********
fatal: [hostname]: UNREACHABLE! => changed=false 
  msg: |-
    Failed to create temporary directory. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in "/tmp", for more error information use -vvv. Failed command was: PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -EncodedCommand UwBlAHQALQBTAHQAcgBpAGMAdABNAG8AZABlACAALQBWAGUAcgBzAGkAbwBuACAATABhAHQAZQBzAHQACgAkAHQAbQBwAF8AcABhAHQAaAAgAD0AIABbAFMAeQBzAHQAZQBtAC4ARQBuAHYAaQByAG8AbgBtAGUAbgB0AF0AOgA6AEUAeABwAGEAbgBkAEUAbgB2AGkAcgBvAG4AbQBlAG4AdABWAGEAcgBpAGEAYgBsAGUAcwAoACcAJQBUAEUATQBQACUAJwApAAoAJAB0AG0AcAAgAD0AIABOAGUAdwAtAEkAdABlAG0AIAAtAFQAeQBwAGUAIABEAGkAcgBlAGMAdABvAHIAeQAgAC0AUABhAHQAaAAgACQAdABtAHAAXwBwAGEAdABoACAALQBOAGEAbQBlACAAJwBhAG4AcwBpAGIAbABlAC0AdABtAHAALQAxADcAMQA0ADUAMAA5ADQANgAxAC4AMAA2ADkANQAwADMALQAzADAAMQAtADIANQAzADUANgA4ADEAMwA5ADQAMwA2ADMAOQA4ACcACgBXAHIAaQB0AGUALQBPAHUAdABwAHUAdAAgAC0ASQBuAHAAdQB0AE8AYgBqAGUAYwB0ACAAJAB0AG0AcAAuAEYAdQBsAGwATgBhAG0AZQAKAEkAZgAgACgALQBuAG8AdAAgACQAPwApACAAewAgAEkAZgAgACgARwBlAHQALQBWAGEAcgBpAGEAYgBsAGUAIABMAEEAUwBUAEUAWABJAFQAQwBPAEQARQAgAC0ARQByAHIAbwByAEEAYwB0AGkAbwBuACAAUwBpAGwAZQBuAHQAbAB5AEMAbwBuAHQAaQBuAHUAZQApACAAewAgAGUAeABpAHQAIAAkAEwAQQBTAFQARQBYAEkAVABDAE8ARABFACAAfQAgAEUAbABzAGUAIAB7ACAAZQB4AGkAdAAgADEAIAB9ACAAfQA=, exited with result 1, stdout output: C:\Users\<username>\AppData\Local\Temp\ansible-tmp-1714509461.069503-301-253568139436398
    , stderr output: OpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
    debug1: /etc/ssh/ssh_config line 21: Applying options for *
    debug2: resolve_canonicalize: hostname ip is address
    debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/var/jenkins_home/.ssh/known_hosts'
    debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/var/jenkins_home/.ssh/known_hosts2'
    debug1: auto-mux: Trying existing master
    debug2: fd 3 setting O_NONBLOCK
    debug2: mux_client_hello_exchange: master version 4
    debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
    debug3: mux_client_request_session: entering
    debug3: mux_client_request_alive: entering
    debug3: mux_client_request_alive: done pid = 268
    debug3: mux_client_request_session: session request sent
    debug1: mux_client_request_session: master session id: 2
    #< CLIXML
    <Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><Obj S="progress" RefId="0"><TN RefId="0"><T>System.Management.Automation.PSCustomObject</T><T>System.Object</T></TN><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj><Obj S="progress" RefId="1"><TNRef RefId="0" /><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj></Objs>#< CLIXML
    <Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><S S="Error">Program 'powershell.exe' failed to run: Win32 internal error "Access is denied" 0x5 occurred while reading the console _x000D__x000A_</S><S S="Error">output buffer. Contact Microsoft Customer Support Services.At line:1 char:1_x000D__x000A_</S><S S="Error">+ PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted - ..._x000D__x000A_</S><S S="Error">+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~._x000D__x000A_</S><S S="Error">At line:1 char:1_x000D__x000A_</S><S S="Error">+ PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted - ..._x000D__x000A_</S><S S="Error">+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~_x000D__x000A_</S><S S="Error">    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException_x000D__x000A_</S><S S="Error">    + FullyQualifiedErrorId : NativeCommandFailed_x000D__x000A_</S><S S="Error"> _x000D__x000A_</S></Objs>debug3: mux_client_read_packet: read header failed: Broken pipe
    debug2: Received exit status from master 1
  unreachable: true

Answer 1 · 2024-04-30T22:23:43.000Z

The problem here doesn't look like anything to do with win_template but rather trying to start PowerShell. You can see the decoded CLIXML message is:

Program 'powershell.exe' failed to run: Win32 internal error "Access is denied" 0x5 occurred while reading the console output buffer. Contact Microsoft Customer Support Services.At line:1 char:1

+ PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.
At line:1 char:1
+ PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

Something is breaking internally in the ssh daemon that is causing an access denied failure. Unfortunately for an error such as this it's going to be something environment specific as it's a fundamental issue with trying to start powershell.exe to run our code.

Answer 2 · 2024-04-30T22:38:02.000Z

I thought that initially as well, however, in the Set Path Variable play I see the same access denied error, which leads me to believe it's a non-fatal error.

It's a separate issue, but when researching that error message I found multiple threads reporting that that access denied error occurs when running powershell commands from a non-interactive ssh session. This thread explains that it's an issue with specific versions of OpenSSH, where powershell doesn't handle data sent to stdout in a non-interactive ssh session. And when decoding the encoded powershell command, there is a write-output command, and it would make sense to me if that triggered the same issue.
But again, that error shows up multiple times in my logs on plays that succeeded, so I don't believe it's causing the 'Unable to create Temporary Directory' error.

Answer 3 · 2024-05-01T00:45:49.000Z

Ah I see, the temp directory is created and outputted as expected but as the return value of the process to create the temporary directory is not 0 the connection plugin sees the command as a failure.

Unfortunately at the point in time the rc is being checked is not something this collection can control, it's part of shared code inside Ansible which validates whether the return code was 0 or not. This would need to fixed on the transport layer to ensure that a non-zero rc is not returned for scenarios like this unfortunately.

Answer 4 · 2024-05-01T13:49:22.000Z

From what I can tell the error is being caused by attempting to create a temp directory that already exists. Decoding the temp directory path, it seems that a timestamp is included in the temp directory name, most likely to avoid the issue of creating duplicate directory names. However, because the same directory gets created twice, it seems that it's not generating a new directory name for the win_template play.
Is it possible to either re-generate the temp directory name, or skip creating a directory at all? When I decode the command that gets run on Windows 2022, it seems that a temp directory isn't created at all.

Answer 5 · 2024-05-01T18:55:39.000Z

From what I can tell the error is being caused by attempting to create a temp directory that already exists

That can't happen as the temporary directory has enough randomness in it to avoid conflicts. The shell plugin uses _generate_temp_dir_name to create the directory name which uses both time, the pid, and a random integer between 0 and 2^48. The temporary directory is also scoped to each task so doing subsequent win_template tasks will use their own temp path. The only exception to this is using multiple tasks in a loop but Ansible is smart enough to only create the temp path once. When you decode the -EncodedCommand being run you can see that the PowerShell script being run by the connection plugin is

Set-StrictMode -Version Latest
$tmp_path = [System.Environment]::ExpandEnvironmentVariables('%TEMP%')
$tmp = New-Item -Type Directory -Path $tmp_path -Name 'ansible-tmp-1714509461.069503-301-253568139436398'
Write-Output -InputObject $tmp.FullName
If (-not $?) { If (Get-Variable LASTEXITCODE -ErrorAction SilentlyContinue) { exit $LASTEXITCODE } Else { exit 1 } }

One of the last step is to output the temporary directory which if we look at the stdout we can see that is the case:

exited with result 1, stdout output: C:\Users\<username>\AppData\Local\Temp\ansible-tmp-1714509461.069503-301-253568139436398

If it was an issue where the same directory is created twice stderr would contain the error

New-Item: An item with the specified name ... already exists

There's nothing about the output that would indicate it's being created twice, it seems like it's being created but because of the internal error in PowerShell about the output stream it is exiting with 1 causing the subsequent failures. You need to solve the problem with ssh here to ensure it doesn't return the rc of 1.

Answer 6 · 2024-05-01T19:12:47.000Z

The reason I say that the same temp directory is getting created twice is because if I decode the command for the Set Path Variable it results in:

Set-StrictMode -Version Latest
$tmp_path = [System.Environment]::ExpandEnvironmentVariables('%TEMP%')
$tmp = New-Item -Type Directory -Path $tmp_path -Name 'ansible-tmp-1714509461.069503-301-253568139436398'
Write-Output -InputObject $tmp.FullName
If (-not $?) { If (Get-Variable LASTEXITCODE -ErrorAction SilentlyContinue) { exit $LASTEXITCODE } Else { exit 1 } }

Which is the exact same command down to the temp path that gets sent during the Setup ntp.conf play.

Answer 7 · 2024-05-01T19:35:40.000Z

I think your output might be mixed and the messages you see is actually for the template task and not the win_path one. I say that because the output comes back after the win_path result is shown so that module has already run and Ansible has processed the result. Also unless you've opted into preserving the temp files Ansible won't create a temp directory for normal module invocations as it pipelines the data, win_template is only doing it because it needs to create the temp directory to store the file that is being copied.

Answer 8 · 2024-05-01T19:58:23.000Z

Ahh, that makes a lot of sense. The log output format is a bit weird, and that definetly explains why it looks like it's creating the same folder twice.
Looks like this issue isn't what I thought it was, so I'm going to close this ticket.

Thanks for all your help.

Answer 9 · 2024-05-02T18:53:47.000Z

For anyone else that runs into this issue, I solved it. It's specifically caused by PowerShell transcription getting turned on (HKLM:\SoftwarePolicies\Microsoft\Windows\PowerShell\Transcription EnableTranscripting). This setting seems to interact weirdly with the version of OpenSSH 7.7p1, which was causing the Access is Denied error that I was seeing above.
This same bug does not appear to be occurring on Windows 2022 which has is running OpenSSH 8.1p1 with PowerShell transcription enabled.

Answer 10 · 2024-05-02T19:29:52.000Z

Awesome glad you figured what the underlying cause was. A pity the OpenSSH server is causing issue with it though.