Some virtual machines are stopped but not deallocated

[JosefBackovsky]

We are using TC completely in azure. We have one VM as leader without agent.
We are using Auzer Resource Manager cloud type.
Terminate instance idle time 60
Max 10 VM from vhd image.
OS windows with reuse allocated virtual machines.
They are part of default pool.
All VMs are form same vhd image.
Plugin version 0.5.3

When there is no load and all cloud agents are displayed as stopped in TC. We still have 6 vm in azure as stopped not deallocated.
When new build is required TC will start agent. In azure portal one of VM will change status from stopped to deallocating and one from Stoped (deallocated) to starting. So it is not even using one which will start faster.

In log teamcity-clouds.log we have for one of them:

[2017-08-29 14:53:08,924]   INFO [onPool-worker-0] - ouds.azure.arm.AzureCloudImage - Starting stopped virtual machine tc-agent-el8 
[2017-08-29 14:53:08,925]   INFO [8 Flush Queue 1] - .instances.StoppedInstanceTask - Instance has changed status from stopped to Scheduled to start: AzureCloudInstance{myName='tc-agent-el8'}, profile 'Azure DEV Elastic Build'{id=arm-4, projectId=_Root} 
[2017-08-29 14:53:08,925]   INFO [8 Flush Queue 1] - .server.impl.CloudEventsLogger - Cloud instance start succeeded: profile 'Azure DEV Elastic Build'{id=arm-4, projectId=_Root}, AzureCloudInstance{myName='tc-agent-el8'} 
[2017-08-29 14:53:09,695]   INFO [uled executor 1] - .server.impl.CloudEventsLogger - Cloud instance entered 'scheduled to start' state, profile 'Azure DEV Elastic Build'{id=arm-4, projectId=_Root}, AzureCloudInstance{myName='tc-agent-el8'} 
[2017-08-29 14:54:54,622]   INFO [-8111-exec-1468] - r.impl.DBCloudStateManagerImpl - Image: AzureCloudImage{myName='TC-agent-el'}, profile: profile 'Azure DEV Elastic Build'{id=arm-4, projectId=_Root} was marked to CONTAIN agent 
[2017-08-29 14:54:54,642]   INFO [-8111-exec-1468] - .server.impl.CloudEventsLogger - Detected cloud agent "tc-agent-el8" {id=506}, profile 'Azure DEV Elastic Build'{id=arm-4, projectId=_Root}, AzureCloudInstance{myName='tc-agent-el8'} 
[2017-08-29 15:03:22,825]   INFO [jectId=_Root} 2] - base.tasks.UpdateInstancesTask - Updated instance 'tc-agent-el8' status to Running based on API information 
[2017-08-29 15:03:22,825]   INFO [jectId=_Root} 2] - uds.base.AbstractCloudInstance - Changing tc-agent-el8(1462c05) status from Scheduled to start to Running  
[2017-08-29 15:03:25,384]   INFO [uled executor 4] - .server.impl.CloudEventsLogger - Cloud instance entered 'running' state, profile 'Azure DEV Elastic Build'{id=arm-4, projectId=_Root}, AzureCloudInstance{myName='tc-agent-el8'} 
[2017-08-29 15:36:12,861]   INFO [jectId=_Root} 2] - base.tasks.UpdateInstancesTask - Updated instance 'tc-agent-el8' status to Stopped based on API information 
[2017-08-29 15:36:12,861]   INFO [jectId=_Root} 2] - uds.base.AbstractCloudInstance - Changing tc-agent-el8(1462c05) status from Running to Stopped  
[2017-08-29 15:36:16,000]   INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud instance entered 'stopped' state, profile 'Azure DEV Elastic Build'{id=arm-4, projectId=_Root}, AzureCloudInstance{myName='tc-agent-el8'} 
[2017-08-29 15:36:16,000]   INFO [uled executor 5] - .server.impl.CloudEventsLogger - Cloud agent Unregistered agent "tc-agent-el8" {id=506} is no longer recognized as cloud agent because instance has gone: profileId=arm-4, imageId=TC-agent-el, instanceId=tc-agent-el8 
[2017-08-29 15:36:16,000]   INFO [uled executor 5] - ces.terminate.AgentRemoverImpl - Removing cloud agent Unregistered agent "tc-agent-el8" {id=506} because: Cloud instance (id=tc-agent-el8, imageId=TC-agent-el, profileId=arm-4) has gone (is not reported by cloud profile) 

Do you have any idea why all vm are not deallocated?
Thanks and regards,
Josef

[dtretyakov]

Thanks for reporting, it could by caused by following: initially some of your cloud agents we're started with termination timeout of 30 mins, then timeout was increased to 60 mins, but it was not propagated to stopped machines. So we need to fix this case. As a workaround you could manually remove such machines, or disable "reuse allocated virtual machines" option untill all of your machines will be recycled.

[JosefBackovsky]

Thanks for your replay. Workaround works.
Regards,
Josef