Hi all,
Distributed (HA) vRA 6.2.3 environment just stopped working. It's been running solid for a couple years; I was logged into it just yesterday, no issues. Initial symptom was when I attempted to login today and got the awesome error message: "Login failed. Please contact your System Administrator and report error code Q2n6XUV7" (BTW -- if anyone can tell me the relevance of that error code and where I can find it in vRA logs, that would be awesome. Back on point...)
I noticed Authentication Failures in catalina.out on the Cafe VAs, so I then started looking through logs on the Identity Appliance. Lots of authentication failures there too. I shutdown the Cafe VAs and bounced the Identity Appliance. Once the Identity Appliance VAMI showed that SSO was initialized, I started the Cafe VAs again and watched the services light up in the VAMI. The eventlog-service was the first one to fail. The next time I refreshed, authentication, authorization, and identity had also failed. At the end of the startup cycle, this is what my services looked like:
Long story short, it's still not working. Below are some log entries that I believe are pertinent -- I have red-highlighted entries that I believe are important clues to this mystery. If anyone can offer guidance on what they mean or how to troubleshoot/resolve, PLEASE HELP!!
Identity Appliance: /var/log/vmware/sso/catalina.out:
This log has been suspiciously quiet for the last six months. There's no timestamp on the error highlighted below, but the next timestamp is from this afternoon (Oct 6). I'm guessing that error occurred today, but it's not clear if this is a symptom or cause of the issue.
Feb 12, 2016 11:13:24 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 21510 ms
Apr 12, 2016 1:55:31 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 3669 ms
Apr 12, 2016 1:55:52 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 20533 ms
Apr 12, 2016 3:25:38 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 4159 ms
Apr 12, 2016 3:26:02 PMorg.apache.catalina.startup.Catalina start
INFO: Server startup in 23382 ms
Exception in thread "Thread-4" java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
at java.util.HashMap$ValueIterator.next(HashMap.java:950)
at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1067)
at com.vmware.identity.session.SessionCleanupWrapper.run(SessionCleanupWrapper.java:52)
at java.lang.Thread.run(Thread.java:745)
Oct 06, 2016 2:02:12 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 3490 ms
Oct 06, 2016 2:02:34 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 21409 ms
Oct 06, 2016 3:40:29 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 2513 ms
Oct 06, 2016 3:40:53 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 2591 ms
Oct 06, 2016 3:41:13 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 19472 ms
Identity Appliance: /var/log/vmware/sso/vmware-identity-sts.log: Failed to authenticate by BST??!? WTF is BST?!?
[2016-10-06 19:19:04,141Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.ws.handlers.SOAPHeadersExtractor] Encountering node {http://www.w3.org/2000/09/xmldsig#}Signature searching for {http://www.w3.org/2000/09/xmldsig#}Signature
[2016-10-06 19:19:04,141Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.handlers.SOAPHeadersExtractor] Searching for Action in http://www.w3.org/2005/08/addressing namespace.
[2016-10-06 19:19:04,142Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.handlers.XMLSignatureValidator] Inside XMLSignatureValidator
[2016-10-06 19:19:04,142Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 INFO com.vmware.identity.sts.ws.handlers.XMLSignatureValidator] Found signature _d9f0ffd0-9d2c-4987-968b-db29da0ea7e7
[2016-10-06 19:19:04,142Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.SignatureValidator] Found KeyInfo
[2016-10-06 19:19:04,142Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.SignatureValidator] Found SecurityTokenReference
[2016-10-06 19:19:04,142Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.SignatureValidator] Found reference with URI: #_68cfd0c6-dead-4bbc-ab7e-448d12095a3d
[2016-10-06 19:19:04,142Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 INFO com.vmware.identity.sts.ws.SignatureValidator] Got signing certificate
[2016-10-06 19:19:04,144Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 INFO com.vmware.identity.sts.ws.handlers.XMLSignatureValidator] Signature _d9f0ffd0-9d2c-4987-968b-db29da0ea7e7 is valid
[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.StsServiceImpl] Start handling 'issue' request
[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.StsServiceImpl] Extracted tenantName from request: vsphere.local
[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.StsServiceImpl] Extracted saml token:[(NULL)]
[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.ws.StsServiceImpl] Found JAXB ActAs entity - null
[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.impl.STSFactoryImpl] Get STS for tenant: vsphere.local
[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.impl.STSImpl] issue() token...
[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.impl.STSImpl] Validation of request
[2016-10-06 19:19:04,146Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.impl.STSImpl] The request received is valid from 2016-10-06T19:19:03.916Z till 2016-10-06T19:29:03.916Z and now is: Thu Oct 06 14:19:04 CDT 2016
[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.perf] 'idm.getClockTolerance' took 3 ms.
[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.impl.LifetimeConvertor] Gets date object from request for datetime: 2016-10-06T19:19:03.916Z
[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.impl.LifetimeConvertor] Gets date object from request for datetime: 2016-10-06T19:29:03.916Z
[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.auth.impl.UNTAuthenticator] Authenticating by UNT...
[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.auth.impl.UNTAuthenticator] No UNT found!
[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Looking up an user by subjectDN CN=cafe-8bd63676-fab1-4e6f-b882-6abab5299982 ...
[2016-10-06 19:19:04,154Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.perf] 'idm.findSolutionUserByCertDn' took 5 ms.
[2016-10-06 19:19:04,154Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.InvalidCredentialsException] About to censor authentication failure
com.vmware.identity.sts.InvalidCredentialsException: Solution user's certificate does not match the one in BST!
at com.vmware.identity.sts.auth.impl.BSTAuthenticator.checkMatchingCertificate(BSTAuthenticator.java:147)
at com.vmware.identity.sts.auth.impl.BSTAuthenticator.doAuthenticate(BSTAuthenticator.java:104)
at com.vmware.identity.sts.auth.impl.BSTAuthenticator.authenticate(BSTAuthenticator.java:71)
at com.vmware.identity.sts.auth.impl.CompositeAuthenticator.authenticate(CompositeAuthenticator.java:44)
at com.vmware.identity.sts.auth.impl.CompositeAuthenticatorPerformanceDecorator$1.call(CompositeAuthenticatorPerformanceDecorator.java:54)
at com.vmware.identity.sts.auth.impl.CompositeAuthenticatorPerformanceDecorator$1.call(CompositeAuthenticatorPerformanceDecorator.java:51)
at com.vmware.identity.performanceSupport.PerformanceDecorator.exec(PerformanceDecorator.java:36)
at com.vmware.identity.sts.auth.impl.CompositeAuthenticatorPerformanceDecorator.authenticate(CompositeAuthenticatorPerformanceDecorator.java:51)
at com.vmware.identity.sts.impl.STSImpl.issue(STSImpl.java:127)
at com.vmware.identity.sts.impl.MultiTenantSTSImpl.issue(MultiTenantSTSImpl.java:50)
at com.vmware.identity.sts.impl.MultiTenantSTSImplPerformanceDecorator$2.call(MultiTenantSTSImplPerformanceDecorator.java:89)
at com.vmware.identity.sts.impl.MultiTenantSTSImplPerformanceDecorator$2.call(MultiTenantSTSImplPerformanceDecorator.java:86)
at com.vmware.identity.performanceSupport.PerformanceDecorator.exec(PerformanceDecorator.java:36)
at com.vmware.identity.sts.impl.MultiTenantSTSImplPerformanceDecorator.issue(MultiTenantSTSImplPerformanceDecorator.java:86)
at com.vmware.identity.sts.ws.StsServiceImpl.issue(StsServiceImpl.java:159)
at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:250)
at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:150)
at com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:261)
at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:100)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:641)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:600)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:585)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:482)
at com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:314)
at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:608)
at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:259)
at com.sun.xml.ws.transport.http.servlet.ServletAdapter.invokeAsync(ServletAdapter.java:213)
at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:159)
at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:194)
at com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:80)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
[2016-10-06 19:19:04,154Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.StsServiceImpl] com.vmware.identity.sts.InvalidCredentialsException: Invalid credentials
at com.vmware.identity.sts.InvalidCredentialsException.buildPublic(InvalidCredentialsException.java:45)
at com.vmware.identity.sts.ws.StsServiceImpl.issue(StsServiceImpl.java:163)
at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:250)
at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:150)
at com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:261)
at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:100)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:641)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:600)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:585)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:482)
at com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:314)
at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:608)
at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:259)
at com.sun.xml.ws.transport.http.servlet.ServletAdapter.invokeAsync(ServletAdapter.java:213)
at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:159)
at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:194)
at com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:80)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
[2016-10-06 19:19:04,154Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 DEBUG com.vmware.identity.sts.ws.SOAPFaultHandler] Ws Fault:
com.vmware.identity.sts.ws.WSFaultException: com.vmware.identity.sts.InvalidCredentialsException: Invalid credentials
at com.vmware.identity.sts.ws.StsServiceImpl.throwFault(StsServiceImpl.java:440)
at com.vmware.identity.sts.ws.StsServiceImpl.issue(StsServiceImpl.java:163)
at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:250)
at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:150)
at com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:261)
at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:100)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:641)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:600)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:585)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:482)
at com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:314)
at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:608)
at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:259)
at com.sun.xml.ws.transport.http.servlet.ServletAdapter.invokeAsync(ServletAdapter.java:213)
at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:159)
at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:194)
at com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:80)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.vmware.identity.sts.InvalidCredentialsException: Invalid credentials
at com.vmware.identity.sts.InvalidCredentialsException.buildPublic(InvalidCredentialsException.java:45)
... 41 more
[2016-10-06 19:19:04,155Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 INFO com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:FailedAuthentication and description: Invalid credentials
[2016-10-06 19:19:07,077Z tomcat-http--19 DEBUG com.vmware.identity.sts.ws.handlers.LogContextHandler] unable to extract correlation id from request. generated new correllation id [cfde93bc-f894-43ac-9f68-ac379c5eda3d]
[2016-10-06 19:19:07,078Z tomcat-http--19 DEBUG com.vmware.identity.sts.ws.handlers.LogContextHandler] extracted tenant [vsphere.local] from the request
[2016-10-06 19:19:07,078Z tomcat-http--19 DEBUG com.vmware.identity.sts.ws.handlers.LogContextHandler] extracted tenant name [vsphere.local] from the request
These BST authentication events are happening roughly 20 times per minute.
<identityVA>:/var/log/vmware/sso # grep 'Authenticating by BST' vmware-identity-sts.log
[2016-10-06 19:19:04,149Z tomcat-http--1 vsphere.local 845db8e1-e193-4feb-af50-3ebf8768f782 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:07,086Z tomcat-http--19 vsphere.local cfde93bc-f894-43ac-9f68-ac379c5eda3d TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:09,915Z tomcat-http--5 vsphere.local 61e4f1d8-9ce4-41c5-9ec1-aa9a0ef5b4c7 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:13,418Z tomcat-http--23 vsphere.local e934216b-31b2-470f-80bd-067a2313003a TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:16,318Z tomcat-http--16 vsphere.local 43a2b4e2-5d1b-4f0c-907b-9eb42be17a54 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:17,023Z tomcat-http--18 vsphere.local 227c2a84-1442-4519-a7dd-cbfda66b1c45 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:24,232Z tomcat-http--11 vsphere.local 21136f00-a064-48fd-9917-a03954c49b1e TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:30,000Z tomcat-http--10 vsphere.local f7c050f8-0740-45a9-a2b3-bbaba0904f6e TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:33,484Z tomcat-http--36 vsphere.local 388087db-8176-41da-b260-56a6a9d89fb5 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:36,380Z tomcat-http--49 vsphere.local ae8e2b2b-4886-405f-8731-214460d8ec93 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:37,101Z tomcat-http--38 vsphere.local 9083d604-f727-4d0e-8720-c4075f078d7b TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:37,344Z tomcat-http--14 vsphere.local 84c84833-89cc-4496-9106-0063cd9a5970 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:44,314Z tomcat-http--31 vsphere.local d59e3202-d276-42aa-b6f8-35f7afd5c485 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:50,071Z tomcat-http--12 vsphere.local 6bec09e4-9e6e-4d3a-a993-7961f0004808 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:53,549Z tomcat-http--48 vsphere.local a442d8ea-0e18-410b-92fb-ee91dc0dd058 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:56,442Z tomcat-http--33 vsphere.local aec686bd-0c73-4337-a102-81a379498fb4 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:19:57,168Z tomcat-http--4 vsphere.local 25794f2e-d6e7-4476-a26e-265b16bcdd91 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:00,067Z tomcat-http--28 vsphere.local 73192347-960e-4e8a-847d-e95c127521b4 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:04,388Z tomcat-http--45 vsphere.local 1050ec90-49da-482a-90db-5177a69d934c TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:07,833Z tomcat-http--15 vsphere.local 62d11ed2-46cf-4d76-bae7-7ab5b209fb7e TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:10,154Z tomcat-http--29 vsphere.local 32327146-ddd2-44b7-88df-4e6ba053fc3d TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:13,609Z tomcat-http--20 vsphere.local 104c631d-b6e6-4576-b1a9-792a93f39224 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:16,506Z tomcat-http--50 vsphere.local 51aeb270-0266-4e55-9319-5d04fc9be3dd TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:17,250Z tomcat-http--35 vsphere.local 710bde46-b45d-4e05-b14f-089ba3dc6944 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:24,465Z tomcat-http--44 vsphere.local 21a6668d-c9f0-49df-8c56-04f49ecd0ad0 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:30,217Z tomcat-http--47 vsphere.local 50cfdef7-a086-4471-88b0-1f5badd418e5 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:33,687Z tomcat-http--24 vsphere.local f40877eb-df35-4fff-8047-23099c4044a7 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:36,572Z tomcat-http--27 vsphere.local 65df49a9-6aae-4672-ac58-360b46323469 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:37,325Z tomcat-http--40 vsphere.local 784e8a88-4087-47b1-8776-74a11075cc6e TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:38,065Z tomcat-http--2 vsphere.local 5cca0e91-a164-4882-a4ba-e40a74fccca3 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:44,532Z tomcat-http--22 vsphere.local d71139c2-9856-4f5b-9f6c-c6420367f571 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:50,299Z tomcat-http--37 vsphere.local 0378f771-8969-4b99-ada3-e62667002f4c TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:53,746Z tomcat-http--32 vsphere.local 33e7b8c6-7466-48ac-83e1-c88986a90c5f TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:56,630Z tomcat-http--43 vsphere.local c00c811e-19d7-4388-a567-2e7e328eb432 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
[2016-10-06 19:20:57,390Z tomcat-http--7 vsphere.local aa917b8c-aa64-4ec5-a745-ddf911895981 TRACE com.vmware.identity.sts.auth.impl.BSTAuthenticator] Authenticating by BST...
On the 'primary' CAFE appliance: /var/log/vmware/vcac/catalina.out:
There are many many many authentication failure messages in catalina.out on the Cafe VA. Here are some examples:
2016-10-06 14:15:16,010 vcac: [component="cafe:catalog" priority="INFO" thread="mainTaskExecutor-3" tenant=""] com.vmware.vcac.core.service.identity.config.ServiceIdentityInitializer.initServiceName:76 - The property serviceName initialized to: catalog-service.
2016-10-06 14:15:16,010 vcac: [component="cafe:catalog" priority="INFO" thread="mainTaskExecutor-3" tenant=""] com.vmware.vcac.core.service.registry.config.ServiceRegistryInitializer.call:261 - Service catalog-service registration started ...
2016-10-06 14:15:16,067 vcac: [component="cafe:catalog" priority="ERROR" thread="mainTaskExecutor-3" tenant=""] com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage:133 - SOAP fault
javax.xml.ws.soap.SOAPFaultException: Invalid credentials
at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178)
at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:117)
at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:184)
at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:203)
at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:130)
at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:81)
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:767)
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:697)
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:458)
at com.vmware.vcac.authentication.sts.SamlTokenService$SolutionSamlTokensHolder.newValue(SamlTokenService.java:511)
at com.vmware.vcac.authentication.sts.SamlTokenService$SolutionSamlTokensHolder.newValue(SamlTokenService.java:488)
at com.vmware.vcac.platform.support.TemporalVariable.get(TemporalVariable.java:29)
at com.vmware.vcac.authentication.sts.SamlTokenService.acquireSolutionToken(SamlTokenService.java:80)
at com.vmware.vcac.core.componentregistry.rest.client.EndPointAuthenticationManager.getServiceSolutionToken(EndPointAuthenticationManager.java:293)
at com.vmware.vcac.core.componentregistry.rest.client.EndPointAuthenticationManager.access$200(EndPointAuthenticationManager.java:36)
at com.vmware.vcac.core.componentregistry.rest.client.EndPointAuthenticationManager$SolutionSsoAuthentication.getHoKToken(EndPointAuthenticationManager.java:414)
at com.vmware.vcac.authentication.sso.client.BaseSsoAuthentication.hashCode(BaseSsoAuthentication.java:96)
at com.vmware.vcac.core.componentregistry.rest.client.RestClientEndPointFactory.getComponentRegistryRestClient(RestClientEndPointFactory.java:216)
at com.vmware.vcac.core.componentregistry.rest.client.RestClientEndPointFactory.getEndPointService(RestClientEndPointFactory.java:178)
at com.vmware.vcac.core.componentregistry.rest.client.RestClientEndPointFactory.getDefaultRestClientByEndPointTypeWhenAvailable(RestClientEndPointFactory.java:265)
at com.vmware.vcac.core.componentregistry.rest.client.SolutionRestClientManager.restClientForSolutionUserByTypeWhenAvailable(SolutionRestClientManager.java:86)
at com.vmware.vcac.core.componentregistry.rest.client.SolutionRestClientManager.restClientForSolutionUserByTypeWhenAvailable(SolutionRestClientManager.java:80)
at com.vmware.vcac.core.service.registry.config.ServiceRegistryInitializer$2.call(ServiceRegistryInitializer.java:262)
at com.vmware.vcac.core.service.registry.config.ServiceRegistryInitializer$2.call(ServiceRegistryInitializer.java:258)
at com.vmware.vcac.platform.rest.client.support.RetriableOperation.call(RetriableOperation.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-10-06 14:15:16,068 vcac: [component="cafe:catalog" priority="INFO" thread="mainTaskExecutor-3" tenant=""] com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.handleFaultCondition:834 - Provided credentials are not valid.
2016-10-06 14:15:16,068 vcac: [component="cafe:catalog" priority="WARN" thread="mainTaskExecutor-3" tenant=""] com.vmware.vcac.platform.rest.client.support.RetriableOperation.call:74 - Exception handled during retry operation with message: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.
2016-10-06 14:15:16,068 vcac: [component="cafe:catalog" priority="INFO" thread="mainTaskExecutor-3" tenant=""] com.vmware.vcac.platform.rest.client.support.RetriableOperation.call:76 - Retries left: [20]. Sleeping for [20] seconds before the next retry attempt.
2016-10-06 14:15:22,351 vcac: [component="cafe:identity" priority="WARN" thread="tomcat-http--3" tenant="vsphere.local"] org.springframework.web.client.RestTemplate.handleResponseError:581 - GET request for "https://clouddev.na.com/component-registry/endpoints/types/sso" resulted in 503 (Service Temporarily Unavailable); invoking error handler
2016-10-06 14:15:22,358 vcac: [component="cafe:identity" priority="WARN" thread="tomcat-http--3" tenant="vsphere.local"] org.springframework.web.client.RestTemplate.handleResponseError:581 - GET request for "https://clouddev.na.com/component-registry/endpoints/types/sso" resulted in 503 (Service Temporarily Unavailable); invoking error handler
2016-10-06 14:15:22,359 vcac: [component="cafe:identity" priority="WARN" thread="tomcat-http--3" tenant="vsphere.local"] com.vmware.vcac.authentication.http.spring.SamlTokenAuthenticationFilter.handleException:115 - Authentication request for failed: Unable to authenticate with SamlToken
2016-10-06 14:15:22,359 vcac: [component="cafe:identity" priority="WARN" thread="tomcat-http--3" tenant="vsphere.local"] com.vmware.vcac.authentication.http.spring.SamlTokenAuthenticationFilter.handleException:116 - Failed:
org.springframework.security.authentication.BadCredentialsException: Unable to authenticate with SamlToken
at com.vmware.vcac.authentication.http.spring.SamlTokenAuthenticationFilter.attemptAuthentication(SamlTokenAuthenticationFilter.java:51)
at com.vmware.vcac.authentication.http.spring.BaseTokenAuthenticationFilter.doFilter(BaseTokenAuthenticationFilter.java:55)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilterWithTenancyContext(TenancyContextFilter.java:67)
at com.vmware.vcac.authentication.http.tenancy.TenancyContextFilter.doFilter(TenancyContextFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.vmware.vcac.authentication.http.SamlAuthenticationException: Failed to create IdP configuration for tenant vsphere.local
at com.vmware.vcac.authentication.http.SamlTokenExtractor.extractToken(SamlTokenExtractor.java:223)
at com.vmware.vcac.authentication.http.SamlTokenExtractor.extractToken(SamlTokenExtractor.java:221)
at com.vmware.vcac.authentication.http.SamlTokenExtractor.extractSamlToken(SamlTokenExtractor.java:78)
at com.vmware.vcac.authentication.http.spring.SamlTokenAuthenticationFilter.attemptAuthentication(SamlTokenAuthenticationFilter.java:34)
... 35 more
Caused by: java.lang.IllegalStateException: Failed to create IdP configuration for tenant vsphere.local
at com.vmware.vcac.authentication.http.idp.IdPMetadataSettingManager.getOrCreateIdPConfigurationIfNeeded(IdPMetadataSettingManager.java:58)
at com.vmware.vcac.authentication.http.SamlTokenExtractor.getRootCertificatesForTenant(SamlTokenExtractor.java:244)
at com.vmware.vcac.authentication.http.SamlTokenExtractor.getVerifier(SamlTokenExtractor.java:234)
at com.vmware.vcac.authentication.http.SamlTokenExtractor.extractToken(SamlTokenExtractor.java:212)
... 38 more
Caused by: org.springframework.web.client.HttpServerErrorException: 503 Service Temporarily Unavailable
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:94)
at com.vmware.vcac.platform.rest.client.error.ResponseErrorHandler.handleError(ResponseErrorHandler.java:49)
at org.springframework.web.client.RestTemplate.handleResponseError(RestTemplate.java:588)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:546)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:517)
at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:255)
at com.vmware.vcac.platform.rest.client.impl.RestClientImpl.get(RestClientImpl.java:237)
at com.vmware.vcac.platform.rest.client.services.AbstractService.get(AbstractService.java:64)
at com.vmware.vcac.core.componentregistry.rest.client.service.EndpointService.getSsoEndPoint(EndpointService.java:102)
at com.vmware.vcac.core.componentregistry.rest.client.IdpMetadataSettingRetrieverImpl.getSsoEndpoint(IdpMetadataSettingRetrieverImpl.java:32)
at com.vmware.vcac.core.componentregistry.rest.client.IdpMetadataSettingRetrieverImpl.getIdPConfiguration(IdpMetadataSettingRetrieverImpl.java:24)
at com.vmware.vcac.authentication.http.idp.IdPMetadataSettingManager.getOrCreateIdPConfigurationIfNeeded(IdPMetadataSettingManager.java:40)
... 41 more