Hey all,
i've come across an abundance of things that can cause guest OS customization to time out. Especially if your template VMs are prepared with Guest User Agent (btw: to install Guest User Agent on windows you're best off using this script: The missing prepare_vra_template » Extending Clouds )
This is all windows guests, but might help for linux troubleshooting as well ...
typically the procedure is
a) vsphere customization is invoked -- Log A in guest OS: C:\Windows\Temp\vmware-imc\guestcust.log
b) then it tries to connect to Manager/IaaS (SSL handshake), which can be found in Log B: C:\VRMGuestAgent\axis2\logs\gugent-axis.log
c) once that succeds the agent logs here: Log C: C:\opt\vmware-appdirector\agent\logs\agent_bootstrap.log
1. if using vSphere 5.5, upgrade your ESXi to latest (at least VMware ESXi 5.5 Update 3)
in our case sometimes a machine failed to customize as it tried to join the domain. But all network connectors were disconnected on the VM at this stage, so the domain join would never happen.
In Log A: you find typically:
Its trying to connect interface
[2016-07-21T02:01:47 DEBUG] Got VMX response 'disconnected'
[2016-07-21T02:01:49 DEBUG] Trying to connect network interfaces, attempt 5
[2016-07-21T02:01:49 DEBUG] Rpci: Sending request='deployPkg.update.state 4 103 C:\Windows\TEMP\vmware-imc\guestcust.log@4000'
But fails after 5 tries:
[2016-07-21T02:01:54 ERROR] Can't connect network interfaces after 5 attempts, giving up
[2016-07-21T02:01:54 INFO] GuestCustUtil exiting.
when you try to connect the adapters manually in vCenter, this also fails:
ESXi logs similar to this:
2016-07-29T14:53:37.987Z [69CC1B70 info 'Libs' opID=70468395-37 user=vpxuser] Vix: [311413 foundryHandles.c:1526]: Error VIX_E_FAIL in Vix_AddRefHandleImpl(): Add ref a destroyed/recycled handle=-1, file=(null), line=0.
2016-07-29T14:53:37.987Z [69CC1B70 info 'Libs' opID=70468395-37 user=vpxuser] Vix: [311413 foundryHandles.c:947]: Error VIX_E_FAIL in Vix_ReleaseHandleImpl(): Releasing a recycled or destroyed handle=-1, file=(null), line=0.
2016-07-29T14:53:38.136Z [687C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/54c3a8e8-15ba7ec7-2889-c81f66ec975c/guestVmName/guestVmName.vmx'] NIC: connection control message: Failed to connect virtual device ethernet0.
totally different issue but exact same logging: Changing the boot delay on a powered on Virtual Machine on ESXi 5.5 fails (2138218) | VMware KB --> Update vSphere layer, resolved the issue for me.
If you are able to connect the Networking adapters, make sure your virtual machines HW version is upodated as well as VMware Tools
2. appd.properties not found in Log C
check first Log B if the agent can actually connect to the Manager/IaaS
Problem 1:
[Fri Aug 05 05:48:53 2016] [debug] ..\..\src\core\transport\http\sender\http_transport_sender.c(246) ctx_epr:https://yourManagerFQDN:443/VMPS2
[Fri Aug 05 05:48:53 2016] [debug] ..\..\src\core\transport\http\sender\http_transport_sender.c(805) using axis2 native http sender.
[Fri Aug 05 05:48:53 2016] [debug] ..\..\src\core\transport\http\sender\http_sender.c(416) msg_ctx_id:urn:uuid:d014410f-69f5-4c19-9705-b3751d963467
[Fri Aug 05 05:48:53 2016] [error] ..\..\src\core\transport\http\sender\http_client.c(238) Error creating a socket. Most probably error returned by OS
[Fri Aug 05 05:48:53 2016] [error] ..\..\src\core\transport\http\sender\http_client.c(239) Socket Creation failed.
If you're using load-balanced vRA, check that LB is forwarding to the correct (active) manager. Best is to simply try in browser what is reported as URL in first line (in the logs above)
Problem 2:
Fri Aug 05 05:54:31 2016] [debug] ..\..\src\core\transport\http\sender\http_transport_sender.c(246) ctx_epr:https://yourManagerFQDN:443/VMPS2
[Fri Aug 05 05:54:31 2016] [debug] ..\..\src\core\transport\http\sender\http_transport_sender.c(805) using axis2 native http sender.
[Fri Aug 05 05:54:31 2016] [debug] ..\..\src\core\transport\http\sender\http_sender.c(416) msg_ctx_id:urn:uuid:4a4aaa32-7e5d-4e76-aee4-976715fbf36d
[Fri Aug 05 05:54:32 2016] [info] [ssl client] Client certificate chain filenot specified
This indicates certificate problem, but did not make sense as the last few days deployment was ok. So I checked and the load balancer was pointing to the wrong INF server (need to check my monitoring rules)
3. Misconfigured Guest user agent
make sure to use script The missing prepare_vra_template » Extending Clouds
when running the script it requires three parameters
1. inf server where to download components. Use your infra server FQDN (not loadbalancer). if you used load balancer, simply use 1st INF server
2. IaaS server for agent communication: In loadbalanced environ,ment I use FQDN of Manager on loadbalancer. In small environment FQDN of infra server
3. Darwin password: create a new password (strong!!)
I'll probably add to it whatever else comes to mind., Please feel free to comment I'll modify accordingly. Those problems have taken too much time of my precious life ...