Hello,
We are having issues with the data collection for the vRO endpoint in vra 7. Data collection fails. We suspect this is related to the DEM worker, but we are unsure how to troubleshoot this. So far the only error we have is when running the data collection on the vRO endpoint:
In vRA console > infrastructure > monitoring, logs:
Endpoint Data Collection failed for endpoint Internal vRO Instance [Workflow Instance Id=31009] vRealize Orchestrator returned an error: Internal Server Error.
In C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\DEM-Core2\Logs\DEM-Core2_Errors
[UTC:2016-05-20 21:24:05 Local:2016-05-20 14:24:05] [Error]: Thread-Id="11" - context="" token="" <?xml version="1.0" encoding="utf-16"?>
<boolean>false</boolean>
Endpoint Data Collection failed for endpoint Internal vRO Instance [Workflow Instance Id=31009]
System.Net.WebException: vRealize Orchestrator returned an error: Internal Server Error.
at DynamicOps.VcoModel.Common.VcoClient.CheckResponseStatus(IRestResponse response)
at DynamicOps.VcoModel.Common.VcoClient.DecodeJsonResponse(IRestResponse response)
at DynamicOps.VcoModel.Common.VcoVersionUtils.GetPluginsInformation(VcoClient client)
at DynamicOps.VcoModel.Activities.StoreVcoPluginsInformation.Execute(CodeActivityContext context)
at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)
at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)
Also, when running data collection on a compute resources VCNS inventory, the following error occurs:
[UTC:2016-05-20 22:54:53 Local:2016-05-20 15:54:53] [Error]: Thread-Id="12" - context="" token="" <?xml version="1.0" encoding="utf-16"?>
<boolean>false</boolean>
Workflow 'vSphereVCNSInventory' failed with the following exception:
DynamicOps.VcoModel.Common.VcoException: Endpoint not found. There is no vRealize Orchestrator endpoint configured with property __VMware.VCenterOrchestrator.Plugin.NSX.Build.
at DynamicOps.VcoModel.Common.EndpointSelector..ctor(ManagementModelEntities context, String selectionFilter)
at DynamicOps.VCNSModel.Activities.VcoNsxHelper.VcoNsxConnectionInfo.GetVcoEndpoint(ManagementModelEntities managementDataContext, Guid virtualMachineId)
at DynamicOps.VCNSModel.Activities.VcoNsxHelper.VcoNsxConnectionInfo.GetConnection(ManagementModelEntities managementDataContext, VCNSEndpoint endpoint, Guid virtualMachineId)
at DynamicOps.VCNSModel.Workflows.vSphereVCNSInventory_CompiledExpressionRoot.vSphereVCNSInventory_CompiledExpressionRoot_TypedDataContext3_ForReadOnly.__Expr21Get()
at DynamicOps.VCNSModel.Workflows.vSphereVCNSInventory_CompiledExpressionRoot.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)
at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)
at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)
at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)
at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)
at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)
at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)
We are using a distributed deployment of vRA 7 and have 2 appliances with the embedded vro. Both appliances show the vco-server service as running, and I can connect to each vco instance using the orchestrator client. Also the XaaS component can load the vro workflows. I am able to browse the inventory.
Also, we noticed this problem after one of our iaas servers died on us. We had to rebuild one from scratch and shiv it in.It was the manager node, so it has the DEO, DEM and manager services running on it. Not sure if this is related, just that we noticed it after. It could have been happening earlier too...
Any suggestions on where to look to narrow this down would be greatly appreciated.
Thank you,
Darrenoid