2015-07-27 21:15:12,647 qpid-server main INFO KERNEL - MicroKernel.start() : MicroKernel.start() : MicroKernel start initiated... 2015-07-27 21:15:13,188 qpid-server main INFO KERNEL.CONFIG - KernelConfiguration.loadKernelProperties() : KernelConfiguration.loadKernelProperties() : MicroKernel properties : {debugsession.tracesize.max=5120, rbac.number.of.tries.for.email.search=1, shutdownServicesOnStop=false, default.response.format=json, record.fault=true, debugsession.timeout=120, region=dc-1, validation.targetendpoint.connection.check.enabled=true, bundle.validation.enabled=false, javax.xml.xpath.XPathFactory=http://java.sun.com/jaxp/xpath/dom=org.apache.xpath.jaxp.XPathFactoryImpl, python.verbose=error, lib.dir.name=lib/infra/services,lib/gateway/services,lib/gateway/steps,lib/analytics/services, javax.xml.transform.TransformerFactory=net.sf.saxon.TransformerFactoryImpl, pod=central, mdc.logging.enable=true, json.request.schema.validation.enabled=true, checkmarx.scan.validator.enabled=false, syslog.tcp.connect.timeout.millis=30000, validation.entity.name.enabled=false, virtual.host.restricted.ports=8080, syslog.tcp.socket.creation.elapsed.millis=30000, authenticated.user.header.name=X-Apigee-Current-User, bundle.validation.schema.enabled=true, astyanax.datastores=kms-datastore,dc-datastore,cache-datastore,counter-datastore,keyvaluemap-datastore,application-datastore,audit-datastore, syslog.tcp.max.allowed.pending.writes=15} 2015-07-27 21:15:23,354 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : MicroKernel.scanDeployments() : Some modules and services were present in the profile but not in the class path 2015-07-27 21:15:23,356 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing modules: [] 2015-07-27 21:15:23,357 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing services: [AXHadoopPersistenceService] 2015-07-27 21:15:23,357 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Not exiting this time but continuing anyway. 2015-07-27 21:15:23,561 qpid-server main INFO c.a.r.f.c.Container - Container.getConnectors() : Container.getConnectors() - Serving http on port - 8083 2015-07-27 21:15:24,007 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : WebService 2015-07-27 21:15:24,009 qpid-server WebServer INFO o.e.jetty.server.Server - Server.doStart() : jetty-8.0.4.v20111024 2015-07-27 21:15:25,450 qpid-server WebServer INFO o.e.j.w.StandardDescriptorProcessor - StandardDescriptorProcessor.visitServlet() : NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet 2015-07-27 21:15:26,051 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 21:15:26,056 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 21:15:26,234 qpid-server main INFO o.a.c.f.i.CuratorFrameworkImpl - CuratorFrameworkImpl.start() : Starting 2015-07-27 21:15:26,985 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : ZooKeeperService 2015-07-27 21:15:27,340 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:15:27,595 qpid-server main-EventThread INFO o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.postState() : State change: SUSPENDED 2015-07-27 21:15:28,582 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:15:28,777 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 21:15:29,704 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:15:29,819 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 21:15:30,821 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:15:30,925 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 21:15:30,927 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 21:15:30,939 qpid-server main ERROR c.a.f.FeatureFlagService - FeatureFlagsPersistance.init() : Error building FeatureFlagsPersistance. Is Zookeeper configured? 2015-07-27 21:15:30,947 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : FeatureFlagService 2015-07-27 21:15:31,928 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:15:32,081 qpid-server main WARN SERVICES.COMMUNICATION - MBeanManager.getRMIConnectorProperties() : RMI host is not provided. Fetching host address from network registry 2015-07-27 21:15:32,249 qpid-server WebServer INFO o.e.j.s.AbstractConnector - AbstractConnector.doStart() : Started SelectChannelConnector@0.0.0.0:8083 STARTING 2015-07-27 21:15:33,044 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:15:33,527 qpid-server main WARN RMI - RMIServer.createSocketFactories() : SSL is disabled for 0.0.0.0:1102 2015-07-27 21:15:34,053 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : CommunicationService 2015-07-27 21:15:34,159 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : OrganizationService 2015-07-27 21:15:34,182 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:15:34,832 qpid-server main INFO HTTP.SERVICE - HTTPTransport.start() : Idle threshold of concurrent channels is 1000 (1000 per selector). Idle channles will be close when this limit is reached. 2015-07-27 21:15:35,072 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPTransport 2015-07-27 21:15:35,103 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPProxy 2015-07-27 21:15:35,256 qpid-server main INFO FILE_LOGGER - BufferedWriter.start() : Starting logging thread 2015-07-27 21:15:35,282 qpid-server main INFO FILE_LOGGER - SizeBasedFileRotator.() : Creating sizeBased rotator with maxSize 104857600 2015-07-27 21:15:35,285 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : EventService 2015-07-27 21:15:35,287 qpid-server Thread-2 INFO FILE_LOGGER - BufferedWriter.run() : Signalling threadInit completion 2015-07-27 21:15:35,318 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:15:35,389 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : RepositoryService 2015-07-27 21:15:35,468 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : AlertService 2015-07-27 21:15:35,656 qpid-server Apigee-Main-9 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 21:15:35,700 qpid-server Apigee-Main-9 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 21:15:35,914 qpid-server main WARN REGISTRATION - CommunicationInfoBuilder.loadIPAddresses() : CommunicationInfoBuilder.loadIPAddresses : File /opt/apigee/ec2.properties is missing. Will use localhost as default for all the hosts 2015-07-27 21:15:36,446 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:15:36,549 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 21:15:37,555 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:15:38,661 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 21:15:39,025 192.168.227.132 qpid-server Apigee-Main-7 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 21:15:39,029 192.168.227.132 qpid-server Apigee-Main-7 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 21:15:39,080 192.168.227.132 qpid-server Apigee-Main-2 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 21:15:39,083 192.168.227.132 qpid-server Apigee-Main-2 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 21:15:39,665 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:15:40,785 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 21:15:40,788 192.168.227.132 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 21:15:40,789 192.168.227.132 qpid-server main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Got a life cycle exception while starting service [ServerRegistrationService, Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable] : com.apigee.zookeeper.ZooKeeperException{ code = zookeeper.ErrorCheckingPathExistence, message = Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, associated contexts = []} 2015-07-27 21:15:41,144 192.168.227.132 qpid-server main ERROR KERNEL - MicroKernel.deployAll() : MicroKernel.deployAll() : Error in deploying the deployment : CommunicationService com.apigee.zookeeper.ZooKeeperException: Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:335) ~[zookeeper-1.0.0.jar:na] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:319) ~[zookeeper-1.0.0.jar:na] at com.apigee.services.repository.zookeeper.ZKRepository.exists(ZKRepository.java:256) ~[repository-impl-1.0.0.jar:na] at com.apigee.services.repository.RepositoryServiceImpl.exists(RepositoryServiceImpl.java:223) ~[repository-impl-1.0.0.jar:na] at com.apigee.registration.info.StatusBuilder.build(StatusBuilder.java:26) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.buildServerInfo(ServerRegistrationServiceImpl.java:265) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.start(ServerRegistrationServiceImpl.java:115) ~[registration-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:69) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:59) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deployDependantServices(ServiceDeployer.java:255) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:62) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:147) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:130) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:124) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.main(MicroKernel.java:75) [microkernel-1.0.0.jar:na] Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:327) ~[zookeeper-1.0.0.jar:na] ... 14 common frames omitted 2015-07-27 21:57:43,614 qpid-server main INFO KERNEL - MicroKernel.start() : MicroKernel.start() : MicroKernel start initiated... 2015-07-27 21:57:43,719 qpid-server main INFO KERNEL.CONFIG - KernelConfiguration.loadKernelProperties() : KernelConfiguration.loadKernelProperties() : MicroKernel properties : {debugsession.tracesize.max=5120, rbac.number.of.tries.for.email.search=1, shutdownServicesOnStop=false, default.response.format=json, record.fault=true, debugsession.timeout=120, region=dc-1, validation.targetendpoint.connection.check.enabled=true, bundle.validation.enabled=false, javax.xml.xpath.XPathFactory=http://java.sun.com/jaxp/xpath/dom=org.apache.xpath.jaxp.XPathFactoryImpl, python.verbose=error, lib.dir.name=lib/infra/services,lib/gateway/services,lib/gateway/steps,lib/analytics/services, javax.xml.transform.TransformerFactory=net.sf.saxon.TransformerFactoryImpl, pod=central, mdc.logging.enable=true, json.request.schema.validation.enabled=true, checkmarx.scan.validator.enabled=false, syslog.tcp.connect.timeout.millis=30000, validation.entity.name.enabled=false, virtual.host.restricted.ports=8080, syslog.tcp.socket.creation.elapsed.millis=30000, authenticated.user.header.name=X-Apigee-Current-User, bundle.validation.schema.enabled=true, astyanax.datastores=kms-datastore,dc-datastore,cache-datastore,counter-datastore,keyvaluemap-datastore,application-datastore,audit-datastore, syslog.tcp.max.allowed.pending.writes=15} 2015-07-27 21:57:47,922 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : MicroKernel.scanDeployments() : Some modules and services were present in the profile but not in the class path 2015-07-27 21:57:47,923 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing modules: [] 2015-07-27 21:57:47,923 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing services: [AXHadoopPersistenceService] 2015-07-27 21:57:47,923 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Not exiting this time but continuing anyway. 2015-07-27 21:57:48,094 qpid-server main INFO c.a.r.f.c.Container - Container.getConnectors() : Container.getConnectors() - Serving http on port - 8083 2015-07-27 21:57:48,228 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : WebService 2015-07-27 21:57:48,232 qpid-server WebServer INFO o.e.jetty.server.Server - Server.doStart() : jetty-8.0.4.v20111024 2015-07-27 21:57:48,740 qpid-server WebServer INFO o.e.j.w.StandardDescriptorProcessor - StandardDescriptorProcessor.visitServlet() : NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet 2015-07-27 21:57:48,818 qpid-server main INFO o.a.c.f.i.CuratorFrameworkImpl - CuratorFrameworkImpl.start() : Starting 2015-07-27 21:57:48,923 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 21:57:48,924 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 21:57:48,987 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : ZooKeeperService 2015-07-27 21:57:49,027 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:57:50,157 qpid-server main-EventThread INFO o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.postState() : State change: SUSPENDED 2015-07-27 21:57:50,180 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 21:57:50,836 qpid-server WebServer INFO o.e.j.s.AbstractConnector - AbstractConnector.doStart() : Started SelectChannelConnector@0.0.0.0:8083 STARTING 2015-07-27 21:57:51,160 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:57:52,260 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 21:57:53,261 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:57:53,364 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 21:57:53,365 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 21:57:53,367 qpid-server main ERROR c.a.f.FeatureFlagService - FeatureFlagsPersistance.init() : Error building FeatureFlagsPersistance. Is Zookeeper configured? 2015-07-27 21:57:53,368 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : FeatureFlagService 2015-07-27 21:57:53,726 qpid-server main WARN SERVICES.COMMUNICATION - MBeanManager.getRMIConnectorProperties() : RMI host is not provided. Fetching host address from network registry 2015-07-27 21:57:53,947 qpid-server Apigee-Main-9 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 21:57:53,978 qpid-server Apigee-Main-9 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 21:57:54,032 qpid-server Apigee-Main-3 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 21:57:54,033 qpid-server Apigee-Main-3 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 21:57:54,264 qpid-server main WARN RMI - RMIServer.createSocketFactories() : SSL is disabled for 0.0.0.0:1102 2015-07-27 21:57:54,370 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:57:54,421 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : CommunicationService 2015-07-27 21:57:54,442 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : OrganizationService 2015-07-27 21:57:54,487 qpid-server main INFO HTTP.SERVICE - HTTPTransport.start() : Idle threshold of concurrent channels is 1000 (1000 per selector). Idle channles will be close when this limit is reached. 2015-07-27 21:57:54,523 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPTransport 2015-07-27 21:57:54,527 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPProxy 2015-07-27 21:57:54,548 qpid-server main INFO FILE_LOGGER - BufferedWriter.start() : Starting logging thread 2015-07-27 21:57:54,551 qpid-server main INFO FILE_LOGGER - SizeBasedFileRotator.() : Creating sizeBased rotator with maxSize 104857600 2015-07-27 21:57:54,551 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : EventService 2015-07-27 21:57:54,554 qpid-server Thread-2 INFO FILE_LOGGER - BufferedWriter.run() : Signalling threadInit completion 2015-07-27 21:57:54,568 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : RepositoryService 2015-07-27 21:57:54,571 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : AlertService 2015-07-27 21:57:54,651 qpid-server main WARN REGISTRATION - CommunicationInfoBuilder.loadIPAddresses() : CommunicationInfoBuilder.loadIPAddresses : File /opt/apigee/ec2.properties is missing. Will use localhost as default for all the hosts 2015-07-27 21:57:55,504 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:57:56,605 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 21:57:57,094 192.168.227.132 qpid-server Apigee-Main-3 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 21:57:57,096 192.168.227.132 qpid-server Apigee-Main-3 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 21:57:57,140 192.168.227.132 qpid-server Apigee-Main-5 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 21:57:57,142 192.168.227.132 qpid-server Apigee-Main-5 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 21:57:57,607 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:57:57,708 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 21:57:58,710 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 21:57:58,966 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 21:57:58,966 192.168.227.132 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 21:57:58,966 192.168.227.132 qpid-server main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Got a life cycle exception while starting service [ServerRegistrationService, Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable] : com.apigee.zookeeper.ZooKeeperException{ code = zookeeper.ErrorCheckingPathExistence, message = Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, associated contexts = []} 2015-07-27 21:57:58,971 192.168.227.132 qpid-server main ERROR KERNEL - MicroKernel.deployAll() : MicroKernel.deployAll() : Error in deploying the deployment : CommunicationService com.apigee.zookeeper.ZooKeeperException: Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:335) ~[zookeeper-1.0.0.jar:na] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:319) ~[zookeeper-1.0.0.jar:na] at com.apigee.services.repository.zookeeper.ZKRepository.exists(ZKRepository.java:256) ~[repository-impl-1.0.0.jar:na] at com.apigee.services.repository.RepositoryServiceImpl.exists(RepositoryServiceImpl.java:223) ~[repository-impl-1.0.0.jar:na] at com.apigee.registration.info.StatusBuilder.build(StatusBuilder.java:26) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.buildServerInfo(ServerRegistrationServiceImpl.java:265) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.start(ServerRegistrationServiceImpl.java:115) ~[registration-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:69) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:59) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deployDependantServices(ServiceDeployer.java:255) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:62) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:147) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:130) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:124) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.main(MicroKernel.java:75) [microkernel-1.0.0.jar:na] Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:327) ~[zookeeper-1.0.0.jar:na] ... 14 common frames omitted 2015-07-27 22:09:25,333 qpid-server main INFO KERNEL - MicroKernel.start() : MicroKernel.start() : MicroKernel start initiated... 2015-07-27 22:09:25,410 qpid-server main INFO KERNEL.CONFIG - KernelConfiguration.loadKernelProperties() : KernelConfiguration.loadKernelProperties() : MicroKernel properties : {debugsession.tracesize.max=5120, rbac.number.of.tries.for.email.search=1, shutdownServicesOnStop=false, default.response.format=json, record.fault=true, debugsession.timeout=120, region=dc-1, validation.targetendpoint.connection.check.enabled=true, bundle.validation.enabled=false, javax.xml.xpath.XPathFactory=http://java.sun.com/jaxp/xpath/dom=org.apache.xpath.jaxp.XPathFactoryImpl, python.verbose=error, lib.dir.name=lib/infra/services,lib/gateway/services,lib/gateway/steps,lib/analytics/services, javax.xml.transform.TransformerFactory=net.sf.saxon.TransformerFactoryImpl, pod=central, mdc.logging.enable=true, json.request.schema.validation.enabled=true, checkmarx.scan.validator.enabled=false, syslog.tcp.connect.timeout.millis=30000, validation.entity.name.enabled=false, virtual.host.restricted.ports=8080, syslog.tcp.socket.creation.elapsed.millis=30000, authenticated.user.header.name=X-Apigee-Current-User, bundle.validation.schema.enabled=true, astyanax.datastores=kms-datastore,dc-datastore,cache-datastore,counter-datastore,keyvaluemap-datastore,application-datastore,audit-datastore, syslog.tcp.max.allowed.pending.writes=15} 2015-07-27 22:09:29,868 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : MicroKernel.scanDeployments() : Some modules and services were present in the profile but not in the class path 2015-07-27 22:09:29,869 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing modules: [] 2015-07-27 22:09:29,869 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing services: [AXHadoopPersistenceService] 2015-07-27 22:09:29,869 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Not exiting this time but continuing anyway. 2015-07-27 22:09:30,028 qpid-server main INFO c.a.r.f.c.Container - Container.getConnectors() : Container.getConnectors() - Serving http on port - 8083 2015-07-27 22:09:30,211 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : WebService 2015-07-27 22:09:30,214 qpid-server WebServer INFO o.e.jetty.server.Server - Server.doStart() : jetty-8.0.4.v20111024 2015-07-27 22:09:30,657 qpid-server WebServer INFO o.e.j.w.StandardDescriptorProcessor - StandardDescriptorProcessor.visitServlet() : NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet 2015-07-27 22:09:30,839 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 22:09:30,840 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 22:09:30,865 qpid-server main INFO o.a.c.f.i.CuratorFrameworkImpl - CuratorFrameworkImpl.start() : Starting 2015-07-27 22:09:31,027 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : ZooKeeperService 2015-07-27 22:09:31,071 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:09:31,187 qpid-server main-EventThread INFO o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.postState() : State change: SUSPENDED 2015-07-27 22:09:32,190 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:09:32,605 qpid-server WebServer INFO o.e.j.s.AbstractConnector - AbstractConnector.doStart() : Started SelectChannelConnector@0.0.0.0:8083 STARTING 2015-07-27 22:09:33,302 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:09:34,298 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:09:35,195 qpid-server Apigee-Main-8 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 22:09:35,212 qpid-server Apigee-Main-8 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:09:35,273 qpid-server Apigee-Main-3 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 22:09:35,274 qpid-server Apigee-Main-3 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:09:35,398 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:09:36,400 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:09:37,500 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:09:37,501 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:09:37,503 qpid-server main ERROR c.a.f.FeatureFlagService - FeatureFlagsPersistance.init() : Error building FeatureFlagsPersistance. Is Zookeeper configured? 2015-07-27 22:09:37,504 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : FeatureFlagService 2015-07-27 22:09:37,675 qpid-server main WARN SERVICES.COMMUNICATION - MBeanManager.getRMIConnectorProperties() : RMI host is not provided. Fetching host address from network registry 2015-07-27 22:09:38,076 qpid-server main WARN RMI - RMIServer.createSocketFactories() : SSL is disabled for 0.0.0.0:1102 2015-07-27 22:09:38,221 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : CommunicationService 2015-07-27 22:09:38,256 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : OrganizationService 2015-07-27 22:09:38,316 qpid-server Apigee-Main-8 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 22:09:38,322 qpid-server Apigee-Main-8 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:09:38,347 qpid-server Apigee-Main-2 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 22:09:38,348 qpid-server Apigee-Main-2 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:09:38,408 qpid-server main INFO HTTP.SERVICE - HTTPTransport.start() : Idle threshold of concurrent channels is 1000 (1000 per selector). Idle channles will be close when this limit is reached. 2015-07-27 22:09:38,449 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPTransport 2015-07-27 22:09:38,453 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPProxy 2015-07-27 22:09:38,464 qpid-server main INFO FILE_LOGGER - BufferedWriter.start() : Starting logging thread 2015-07-27 22:09:38,470 qpid-server main INFO FILE_LOGGER - SizeBasedFileRotator.() : Creating sizeBased rotator with maxSize 104857600 2015-07-27 22:09:38,471 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : EventService 2015-07-27 22:09:38,474 qpid-server Thread-2 INFO FILE_LOGGER - BufferedWriter.run() : Signalling threadInit completion 2015-07-27 22:09:38,490 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : RepositoryService 2015-07-27 22:09:38,493 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : AlertService 2015-07-27 22:09:38,504 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:09:38,627 qpid-server main WARN REGISTRATION - CommunicationInfoBuilder.loadIPAddresses() : CommunicationInfoBuilder.loadIPAddresses : File /opt/apigee/ec2.properties is missing. Will use localhost as default for all the hosts 2015-07-27 22:09:39,605 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:09:40,606 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:09:40,708 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:09:41,366 192.168.227.132 qpid-server Apigee-Main-2 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 22:09:41,367 192.168.227.132 qpid-server Apigee-Main-2 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:09:41,376 192.168.227.132 qpid-server Apigee-Main-3 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 22:09:41,378 192.168.227.132 qpid-server Apigee-Main-3 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:09:41,709 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:09:42,830 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:09:42,830 192.168.227.132 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:09:42,831 192.168.227.132 qpid-server main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Got a life cycle exception while starting service [ServerRegistrationService, Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable] : com.apigee.zookeeper.ZooKeeperException{ code = zookeeper.ErrorCheckingPathExistence, message = Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, associated contexts = []} 2015-07-27 22:09:42,837 192.168.227.132 qpid-server main ERROR KERNEL - MicroKernel.deployAll() : MicroKernel.deployAll() : Error in deploying the deployment : CommunicationService com.apigee.zookeeper.ZooKeeperException: Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:335) ~[zookeeper-1.0.0.jar:na] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:319) ~[zookeeper-1.0.0.jar:na] at com.apigee.services.repository.zookeeper.ZKRepository.exists(ZKRepository.java:256) ~[repository-impl-1.0.0.jar:na] at com.apigee.services.repository.RepositoryServiceImpl.exists(RepositoryServiceImpl.java:223) ~[repository-impl-1.0.0.jar:na] at com.apigee.registration.info.StatusBuilder.build(StatusBuilder.java:26) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.buildServerInfo(ServerRegistrationServiceImpl.java:265) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.start(ServerRegistrationServiceImpl.java:115) ~[registration-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:69) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:59) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deployDependantServices(ServiceDeployer.java:255) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:62) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:147) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:130) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:124) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.main(MicroKernel.java:75) [microkernel-1.0.0.jar:na] Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:327) ~[zookeeper-1.0.0.jar:na] ... 14 common frames omitted 2015-07-27 22:19:32,989 qpid-server main INFO KERNEL - MicroKernel.start() : MicroKernel.start() : MicroKernel start initiated... 2015-07-27 22:19:33,082 qpid-server main INFO KERNEL.CONFIG - KernelConfiguration.loadKernelProperties() : KernelConfiguration.loadKernelProperties() : MicroKernel properties : {debugsession.tracesize.max=5120, rbac.number.of.tries.for.email.search=1, shutdownServicesOnStop=false, default.response.format=json, record.fault=true, debugsession.timeout=120, region=dc-1, validation.targetendpoint.connection.check.enabled=true, bundle.validation.enabled=false, javax.xml.xpath.XPathFactory=http://java.sun.com/jaxp/xpath/dom=org.apache.xpath.jaxp.XPathFactoryImpl, python.verbose=error, lib.dir.name=lib/infra/services,lib/gateway/services,lib/gateway/steps,lib/analytics/services, javax.xml.transform.TransformerFactory=net.sf.saxon.TransformerFactoryImpl, pod=central, mdc.logging.enable=true, json.request.schema.validation.enabled=true, checkmarx.scan.validator.enabled=false, syslog.tcp.connect.timeout.millis=30000, validation.entity.name.enabled=false, virtual.host.restricted.ports=8080, syslog.tcp.socket.creation.elapsed.millis=30000, authenticated.user.header.name=X-Apigee-Current-User, bundle.validation.schema.enabled=true, astyanax.datastores=kms-datastore,dc-datastore,cache-datastore,counter-datastore,keyvaluemap-datastore,application-datastore,audit-datastore, syslog.tcp.max.allowed.pending.writes=15} 2015-07-27 22:19:37,245 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : MicroKernel.scanDeployments() : Some modules and services were present in the profile but not in the class path 2015-07-27 22:19:37,251 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing modules: [] 2015-07-27 22:19:37,251 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing services: [AXHadoopPersistenceService] 2015-07-27 22:19:37,252 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Not exiting this time but continuing anyway. 2015-07-27 22:19:37,395 qpid-server main INFO c.a.r.f.c.Container - Container.getConnectors() : Container.getConnectors() - Serving http on port - 8083 2015-07-27 22:19:37,567 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : WebService 2015-07-27 22:19:37,570 qpid-server WebServer INFO o.e.jetty.server.Server - Server.doStart() : jetty-8.0.4.v20111024 2015-07-27 22:19:37,945 qpid-server WebServer INFO o.e.j.w.StandardDescriptorProcessor - StandardDescriptorProcessor.visitServlet() : NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet 2015-07-27 22:19:38,126 qpid-server main INFO o.a.c.f.i.CuratorFrameworkImpl - CuratorFrameworkImpl.start() : Starting 2015-07-27 22:19:38,143 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 22:19:38,144 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 22:19:38,332 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : ZooKeeperService 2015-07-27 22:19:38,368 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:19:38,484 qpid-server main-EventThread INFO o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.postState() : State change: SUSPENDED 2015-07-27 22:19:39,485 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:19:39,611 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:19:39,785 qpid-server WebServer INFO o.e.j.s.AbstractConnector - AbstractConnector.doStart() : Started SelectChannelConnector@0.0.0.0:8083 STARTING 2015-07-27 22:19:40,603 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:19:41,704 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:19:42,705 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:19:42,815 qpid-server Apigee-Main-9 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 22:19:42,818 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:19:42,818 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:19:42,826 qpid-server main ERROR c.a.f.FeatureFlagService - FeatureFlagsPersistance.init() : Error building FeatureFlagsPersistance. Is Zookeeper configured? 2015-07-27 22:19:42,826 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : FeatureFlagService 2015-07-27 22:19:42,855 qpid-server Apigee-Main-9 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:19:42,940 qpid-server Apigee-Main-4 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 22:19:42,946 qpid-server Apigee-Main-4 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:19:43,062 qpid-server main WARN SERVICES.COMMUNICATION - MBeanManager.getRMIConnectorProperties() : RMI host is not provided. Fetching host address from network registry 2015-07-27 22:19:43,413 qpid-server main WARN RMI - RMIServer.createSocketFactories() : SSL is disabled for 0.0.0.0:1102 2015-07-27 22:19:43,571 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : CommunicationService 2015-07-27 22:19:43,599 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : OrganizationService 2015-07-27 22:19:43,678 qpid-server main INFO HTTP.SERVICE - HTTPTransport.start() : Idle threshold of concurrent channels is 1000 (1000 per selector). Idle channles will be close when this limit is reached. 2015-07-27 22:19:43,717 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPTransport 2015-07-27 22:19:43,721 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPProxy 2015-07-27 22:19:43,734 qpid-server main INFO FILE_LOGGER - BufferedWriter.start() : Starting logging thread 2015-07-27 22:19:43,737 qpid-server main INFO FILE_LOGGER - SizeBasedFileRotator.() : Creating sizeBased rotator with maxSize 104857600 2015-07-27 22:19:43,738 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : EventService 2015-07-27 22:19:43,741 qpid-server Thread-2 INFO FILE_LOGGER - BufferedWriter.run() : Signalling threadInit completion 2015-07-27 22:19:43,760 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : RepositoryService 2015-07-27 22:19:43,764 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : AlertService 2015-07-27 22:19:43,819 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:19:43,849 qpid-server main WARN REGISTRATION - CommunicationInfoBuilder.loadIPAddresses() : CommunicationInfoBuilder.loadIPAddresses : File /opt/apigee/ec2.properties is missing. Will use localhost as default for all the hosts 2015-07-27 22:19:44,920 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:19:45,921 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:19:45,962 192.168.227.132 qpid-server Apigee-Main-3 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 22:19:45,965 192.168.227.132 qpid-server Apigee-Main-3 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:19:45,974 192.168.227.132 qpid-server Apigee-Main-4 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 22:19:45,976 192.168.227.132 qpid-server Apigee-Main-4 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:19:46,022 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:19:47,023 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:19:47,124 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:19:47,125 192.168.227.132 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:19:47,126 192.168.227.132 qpid-server main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Got a life cycle exception while starting service [ServerRegistrationService, Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable] : com.apigee.zookeeper.ZooKeeperException{ code = zookeeper.ErrorCheckingPathExistence, message = Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, associated contexts = []} 2015-07-27 22:19:47,132 192.168.227.132 qpid-server main ERROR KERNEL - MicroKernel.deployAll() : MicroKernel.deployAll() : Error in deploying the deployment : CommunicationService com.apigee.zookeeper.ZooKeeperException: Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:335) ~[zookeeper-1.0.0.jar:na] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:319) ~[zookeeper-1.0.0.jar:na] at com.apigee.services.repository.zookeeper.ZKRepository.exists(ZKRepository.java:256) ~[repository-impl-1.0.0.jar:na] at com.apigee.services.repository.RepositoryServiceImpl.exists(RepositoryServiceImpl.java:223) ~[repository-impl-1.0.0.jar:na] at com.apigee.registration.info.StatusBuilder.build(StatusBuilder.java:26) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.buildServerInfo(ServerRegistrationServiceImpl.java:265) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.start(ServerRegistrationServiceImpl.java:115) ~[registration-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:69) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:59) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deployDependantServices(ServiceDeployer.java:255) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:62) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:147) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:130) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:124) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.main(MicroKernel.java:75) [microkernel-1.0.0.jar:na] Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:327) ~[zookeeper-1.0.0.jar:na] ... 14 common frames omitted 2015-07-27 22:31:56,491 qpid-server main INFO KERNEL - MicroKernel.start() : MicroKernel.start() : MicroKernel start initiated... 2015-07-27 22:31:56,688 qpid-server main INFO KERNEL.CONFIG - KernelConfiguration.loadKernelProperties() : KernelConfiguration.loadKernelProperties() : MicroKernel properties : {debugsession.tracesize.max=5120, rbac.number.of.tries.for.email.search=1, shutdownServicesOnStop=false, default.response.format=json, record.fault=true, debugsession.timeout=120, region=dc-1, validation.targetendpoint.connection.check.enabled=true, bundle.validation.enabled=false, javax.xml.xpath.XPathFactory=http://java.sun.com/jaxp/xpath/dom=org.apache.xpath.jaxp.XPathFactoryImpl, python.verbose=error, lib.dir.name=lib/infra/services,lib/gateway/services,lib/gateway/steps,lib/analytics/services, javax.xml.transform.TransformerFactory=net.sf.saxon.TransformerFactoryImpl, pod=central, mdc.logging.enable=true, json.request.schema.validation.enabled=true, checkmarx.scan.validator.enabled=false, syslog.tcp.connect.timeout.millis=30000, validation.entity.name.enabled=false, virtual.host.restricted.ports=8080, syslog.tcp.socket.creation.elapsed.millis=30000, authenticated.user.header.name=X-Apigee-Current-User, bundle.validation.schema.enabled=true, astyanax.datastores=kms-datastore,dc-datastore,cache-datastore,counter-datastore,keyvaluemap-datastore,application-datastore,audit-datastore, syslog.tcp.max.allowed.pending.writes=15} 2015-07-27 22:32:50,573 qpid-server main INFO KERNEL - MicroKernel.start() : MicroKernel.start() : MicroKernel start initiated... 2015-07-27 22:32:50,831 qpid-server main INFO KERNEL.CONFIG - KernelConfiguration.loadKernelProperties() : KernelConfiguration.loadKernelProperties() : MicroKernel properties : {debugsession.tracesize.max=5120, rbac.number.of.tries.for.email.search=1, shutdownServicesOnStop=false, default.response.format=json, record.fault=true, debugsession.timeout=120, region=dc-1, validation.targetendpoint.connection.check.enabled=true, bundle.validation.enabled=false, javax.xml.xpath.XPathFactory=http://java.sun.com/jaxp/xpath/dom=org.apache.xpath.jaxp.XPathFactoryImpl, python.verbose=error, lib.dir.name=lib/infra/services,lib/gateway/services,lib/gateway/steps,lib/analytics/services, javax.xml.transform.TransformerFactory=net.sf.saxon.TransformerFactoryImpl, pod=central, mdc.logging.enable=true, json.request.schema.validation.enabled=true, checkmarx.scan.validator.enabled=false, syslog.tcp.connect.timeout.millis=30000, validation.entity.name.enabled=false, virtual.host.restricted.ports=8080, syslog.tcp.socket.creation.elapsed.millis=30000, authenticated.user.header.name=X-Apigee-Current-User, bundle.validation.schema.enabled=true, astyanax.datastores=kms-datastore,dc-datastore,cache-datastore,counter-datastore,keyvaluemap-datastore,application-datastore,audit-datastore, syslog.tcp.max.allowed.pending.writes=15} 2015-07-27 22:33:01,379 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : MicroKernel.scanDeployments() : Some modules and services were present in the profile but not in the class path 2015-07-27 22:33:01,380 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing modules: [] 2015-07-27 22:33:01,381 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing services: [AXHadoopPersistenceService] 2015-07-27 22:33:01,386 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Not exiting this time but continuing anyway. 2015-07-27 22:33:01,788 qpid-server main INFO c.a.r.f.c.Container - Container.getConnectors() : Container.getConnectors() - Serving http on port - 8083 2015-07-27 22:33:02,374 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : WebService 2015-07-27 22:33:02,377 qpid-server WebServer INFO o.e.jetty.server.Server - Server.doStart() : jetty-8.0.4.v20111024 2015-07-27 22:33:03,649 qpid-server WebServer INFO o.e.j.w.StandardDescriptorProcessor - StandardDescriptorProcessor.visitServlet() : NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet 2015-07-27 22:33:04,091 qpid-server main INFO o.a.c.f.i.CuratorFrameworkImpl - CuratorFrameworkImpl.start() : Starting 2015-07-27 22:33:04,173 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 22:33:04,178 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 22:33:04,624 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : ZooKeeperService 2015-07-27 22:33:04,876 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:33:05,082 qpid-server main-EventThread INFO o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.postState() : State change: SUSPENDED 2015-07-27 22:33:06,044 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:33:06,248 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:33:07,187 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:33:08,294 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:33:08,874 qpid-server WebServer INFO o.e.j.s.AbstractConnector - AbstractConnector.doStart() : Started SelectChannelConnector@0.0.0.0:8083 STARTING 2015-07-27 22:33:09,299 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:33:10,422 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:33:10,424 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:33:10,436 qpid-server main ERROR c.a.f.FeatureFlagService - FeatureFlagsPersistance.init() : Error building FeatureFlagsPersistance. Is Zookeeper configured? 2015-07-27 22:33:10,436 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : FeatureFlagService 2015-07-27 22:33:11,302 qpid-server Apigee-Main-6 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 22:33:11,339 qpid-server Apigee-Main-6 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:33:11,423 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:33:11,489 qpid-server main WARN SERVICES.COMMUNICATION - MBeanManager.getRMIConnectorProperties() : RMI host is not provided. Fetching host address from network registry 2015-07-27 22:33:11,570 qpid-server Apigee-Main-9 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 22:33:11,577 qpid-server Apigee-Main-9 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:33:12,563 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:33:12,867 qpid-server main WARN RMI - RMIServer.createSocketFactories() : SSL is disabled for 0.0.0.0:1102 2015-07-27 22:33:13,533 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : CommunicationService 2015-07-27 22:33:13,564 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : OrganizationService 2015-07-27 22:33:13,665 qpid-server main INFO HTTP.SERVICE - HTTPTransport.start() : Idle threshold of concurrent channels is 1000 (1000 per selector). Idle channles will be close when this limit is reached. 2015-07-27 22:33:13,776 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPTransport 2015-07-27 22:33:13,785 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPProxy 2015-07-27 22:33:13,839 qpid-server main INFO FILE_LOGGER - BufferedWriter.start() : Starting logging thread 2015-07-27 22:33:13,847 qpid-server main INFO FILE_LOGGER - SizeBasedFileRotator.() : Creating sizeBased rotator with maxSize 104857600 2015-07-27 22:33:13,848 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : EventService 2015-07-27 22:33:13,865 qpid-server Thread-2 INFO FILE_LOGGER - BufferedWriter.run() : Signalling threadInit completion 2015-07-27 22:33:13,906 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : RepositoryService 2015-07-27 22:33:13,913 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : AlertService 2015-07-27 22:33:14,187 qpid-server main WARN REGISTRATION - CommunicationInfoBuilder.loadIPAddresses() : CommunicationInfoBuilder.loadIPAddresses : File /opt/apigee/ec2.properties is missing. Will use localhost as default for all the hosts 2015-07-27 22:33:14,624 192.168.227.132 qpid-server Apigee-Main-2 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 22:33:14,627 192.168.227.132 qpid-server Apigee-Main-2 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:33:14,649 192.168.227.132 qpid-server Apigee-Main-4 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 22:33:14,654 192.168.227.132 qpid-server Apigee-Main-4 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:33:14,723 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:33:15,845 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:33:16,848 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:33:17,681 192.168.227.132 qpid-server Apigee-Main-5 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 22:33:17,716 192.168.227.132 qpid-server Apigee-Main-5 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:33:17,736 192.168.227.132 qpid-server Apigee-Main-8 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 22:33:17,740 192.168.227.132 qpid-server Apigee-Main-8 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:33:17,950 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:33:18,951 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:33:19,056 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:33:19,057 192.168.227.132 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:33:19,059 192.168.227.132 qpid-server main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Got a life cycle exception while starting service [ServerRegistrationService, Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable] : com.apigee.zookeeper.ZooKeeperException{ code = zookeeper.ErrorCheckingPathExistence, message = Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, associated contexts = []} 2015-07-27 22:33:19,071 192.168.227.132 qpid-server main ERROR KERNEL - MicroKernel.deployAll() : MicroKernel.deployAll() : Error in deploying the deployment : CommunicationService com.apigee.zookeeper.ZooKeeperException: Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:335) ~[zookeeper-1.0.0.jar:na] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:319) ~[zookeeper-1.0.0.jar:na] at com.apigee.services.repository.zookeeper.ZKRepository.exists(ZKRepository.java:256) ~[repository-impl-1.0.0.jar:na] at com.apigee.services.repository.RepositoryServiceImpl.exists(RepositoryServiceImpl.java:223) ~[repository-impl-1.0.0.jar:na] at com.apigee.registration.info.StatusBuilder.build(StatusBuilder.java:26) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.buildServerInfo(ServerRegistrationServiceImpl.java:265) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.start(ServerRegistrationServiceImpl.java:115) ~[registration-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:69) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:59) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deployDependantServices(ServiceDeployer.java:255) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:62) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:147) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:130) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:124) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.main(MicroKernel.java:75) [microkernel-1.0.0.jar:na] Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:327) ~[zookeeper-1.0.0.jar:na] ... 14 common frames omitted 2015-07-27 22:45:45,811 qpid-server main INFO KERNEL - MicroKernel.start() : MicroKernel.start() : MicroKernel start initiated... 2015-07-27 22:45:45,887 qpid-server main INFO KERNEL.CONFIG - KernelConfiguration.loadKernelProperties() : KernelConfiguration.loadKernelProperties() : MicroKernel properties : {debugsession.tracesize.max=5120, rbac.number.of.tries.for.email.search=1, shutdownServicesOnStop=false, default.response.format=json, record.fault=true, debugsession.timeout=120, region=dc-1, validation.targetendpoint.connection.check.enabled=true, bundle.validation.enabled=false, javax.xml.xpath.XPathFactory=http://java.sun.com/jaxp/xpath/dom=org.apache.xpath.jaxp.XPathFactoryImpl, python.verbose=error, lib.dir.name=lib/infra/services,lib/gateway/services,lib/gateway/steps,lib/analytics/services, javax.xml.transform.TransformerFactory=net.sf.saxon.TransformerFactoryImpl, pod=central, mdc.logging.enable=true, json.request.schema.validation.enabled=true, checkmarx.scan.validator.enabled=false, syslog.tcp.connect.timeout.millis=30000, validation.entity.name.enabled=false, virtual.host.restricted.ports=8080, syslog.tcp.socket.creation.elapsed.millis=30000, authenticated.user.header.name=X-Apigee-Current-User, bundle.validation.schema.enabled=true, astyanax.datastores=kms-datastore,dc-datastore,cache-datastore,counter-datastore,keyvaluemap-datastore,application-datastore,audit-datastore, syslog.tcp.max.allowed.pending.writes=15} 2015-07-27 22:45:50,093 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : MicroKernel.scanDeployments() : Some modules and services were present in the profile but not in the class path 2015-07-27 22:45:50,094 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing modules: [] 2015-07-27 22:45:50,094 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing services: [AXHadoopPersistenceService] 2015-07-27 22:45:50,094 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Not exiting this time but continuing anyway. 2015-07-27 22:45:50,249 qpid-server main INFO c.a.r.f.c.Container - Container.getConnectors() : Container.getConnectors() - Serving http on port - 8083 2015-07-27 22:45:50,512 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : WebService 2015-07-27 22:45:50,514 qpid-server WebServer INFO o.e.jetty.server.Server - Server.doStart() : jetty-8.0.4.v20111024 2015-07-27 22:45:50,988 qpid-server WebServer INFO o.e.j.w.StandardDescriptorProcessor - StandardDescriptorProcessor.visitServlet() : NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet 2015-07-27 22:45:51,155 qpid-server main INFO o.a.c.f.i.CuratorFrameworkImpl - CuratorFrameworkImpl.start() : Starting 2015-07-27 22:45:51,190 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 22:45:51,191 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 22:45:51,328 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : ZooKeeperService 2015-07-27 22:45:51,377 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:45:51,512 qpid-server main-EventThread INFO o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.postState() : State change: SUSPENDED 2015-07-27 22:45:52,515 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:45:52,990 qpid-server WebServer INFO o.e.j.s.AbstractConnector - AbstractConnector.doStart() : Started SelectChannelConnector@0.0.0.0:8083 STARTING 2015-07-27 22:45:53,629 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:45:54,624 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:45:55,557 qpid-server Apigee-Main-8 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 22:45:55,573 qpid-server Apigee-Main-8 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:45:55,625 qpid-server Apigee-Main-2 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 22:45:55,626 qpid-server Apigee-Main-2 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:45:55,724 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:45:56,725 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:45:57,825 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:45:57,827 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:45:57,829 qpid-server main ERROR c.a.f.FeatureFlagService - FeatureFlagsPersistance.init() : Error building FeatureFlagsPersistance. Is Zookeeper configured? 2015-07-27 22:45:57,830 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : FeatureFlagService 2015-07-27 22:45:57,975 qpid-server main WARN SERVICES.COMMUNICATION - MBeanManager.getRMIConnectorProperties() : RMI host is not provided. Fetching host address from network registry 2015-07-27 22:45:58,335 qpid-server main WARN RMI - RMIServer.createSocketFactories() : SSL is disabled for 0.0.0.0:1102 2015-07-27 22:45:58,504 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : CommunicationService 2015-07-27 22:45:58,539 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : OrganizationService 2015-07-27 22:45:58,572 qpid-server main INFO HTTP.SERVICE - HTTPTransport.start() : Idle threshold of concurrent channels is 1000 (1000 per selector). Idle channles will be close when this limit is reached. 2015-07-27 22:45:58,616 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPTransport 2015-07-27 22:45:58,620 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPProxy 2015-07-27 22:45:58,634 qpid-server main INFO FILE_LOGGER - BufferedWriter.start() : Starting logging thread 2015-07-27 22:45:58,639 qpid-server main INFO FILE_LOGGER - SizeBasedFileRotator.() : Creating sizeBased rotator with maxSize 104857600 2015-07-27 22:45:58,642 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : EventService 2015-07-27 22:45:58,644 qpid-server Thread-2 INFO FILE_LOGGER - BufferedWriter.run() : Signalling threadInit completion 2015-07-27 22:45:58,652 qpid-server Apigee-Main-2 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 22:45:58,653 qpid-server Apigee-Main-2 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:45:58,679 qpid-server Apigee-Main-3 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 22:45:58,683 qpid-server Apigee-Main-3 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:45:58,701 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : RepositoryService 2015-07-27 22:45:58,704 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : AlertService 2015-07-27 22:45:58,790 qpid-server main WARN REGISTRATION - CommunicationInfoBuilder.loadIPAddresses() : CommunicationInfoBuilder.loadIPAddresses : File /opt/apigee/ec2.properties is missing. Will use localhost as default for all the hosts 2015-07-27 22:45:59,927 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:46:00,928 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:46:01,702 192.168.227.132 qpid-server Apigee-Main-2 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 22:46:01,704 192.168.227.132 qpid-server Apigee-Main-2 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:46:01,713 192.168.227.132 qpid-server Apigee-Main-4 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 22:46:01,715 192.168.227.132 qpid-server Apigee-Main-4 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:46:02,029 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:46:03,030 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:46:04,131 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:46:04,132 192.168.227.132 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:46:04,132 192.168.227.132 qpid-server main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Got a life cycle exception while starting service [ServerRegistrationService, Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable] : com.apigee.zookeeper.ZooKeeperException{ code = zookeeper.ErrorCheckingPathExistence, message = Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, associated contexts = []} 2015-07-27 22:46:04,138 192.168.227.132 qpid-server main ERROR KERNEL - MicroKernel.deployAll() : MicroKernel.deployAll() : Error in deploying the deployment : CommunicationService com.apigee.zookeeper.ZooKeeperException: Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:335) ~[zookeeper-1.0.0.jar:na] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:319) ~[zookeeper-1.0.0.jar:na] at com.apigee.services.repository.zookeeper.ZKRepository.exists(ZKRepository.java:256) ~[repository-impl-1.0.0.jar:na] at com.apigee.services.repository.RepositoryServiceImpl.exists(RepositoryServiceImpl.java:223) ~[repository-impl-1.0.0.jar:na] at com.apigee.registration.info.StatusBuilder.build(StatusBuilder.java:26) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.buildServerInfo(ServerRegistrationServiceImpl.java:265) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.start(ServerRegistrationServiceImpl.java:115) ~[registration-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:69) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:59) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deployDependantServices(ServiceDeployer.java:255) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:62) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:147) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:130) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:124) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.main(MicroKernel.java:75) [microkernel-1.0.0.jar:na] Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:327) ~[zookeeper-1.0.0.jar:na] ... 14 common frames omitted 2015-07-27 22:48:07,738 qpid-server main INFO KERNEL - MicroKernel.start() : MicroKernel.start() : MicroKernel start initiated... 2015-07-27 22:48:08,134 qpid-server main INFO KERNEL.CONFIG - KernelConfiguration.loadKernelProperties() : KernelConfiguration.loadKernelProperties() : MicroKernel properties : {debugsession.tracesize.max=5120, rbac.number.of.tries.for.email.search=1, shutdownServicesOnStop=false, default.response.format=json, record.fault=true, debugsession.timeout=120, region=dc-1, validation.targetendpoint.connection.check.enabled=true, bundle.validation.enabled=false, javax.xml.xpath.XPathFactory=http://java.sun.com/jaxp/xpath/dom=org.apache.xpath.jaxp.XPathFactoryImpl, python.verbose=error, lib.dir.name=lib/infra/services,lib/gateway/services,lib/gateway/steps,lib/analytics/services, javax.xml.transform.TransformerFactory=net.sf.saxon.TransformerFactoryImpl, pod=central, mdc.logging.enable=true, json.request.schema.validation.enabled=true, checkmarx.scan.validator.enabled=false, syslog.tcp.connect.timeout.millis=30000, validation.entity.name.enabled=false, virtual.host.restricted.ports=8080, syslog.tcp.socket.creation.elapsed.millis=30000, authenticated.user.header.name=X-Apigee-Current-User, bundle.validation.schema.enabled=true, astyanax.datastores=kms-datastore,dc-datastore,cache-datastore,counter-datastore,keyvaluemap-datastore,application-datastore,audit-datastore, syslog.tcp.max.allowed.pending.writes=15} 2015-07-27 22:48:19,493 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : MicroKernel.scanDeployments() : Some modules and services were present in the profile but not in the class path 2015-07-27 22:48:19,496 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing modules: [] 2015-07-27 22:48:19,496 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Missing services: [AXHadoopPersistenceService] 2015-07-27 22:48:19,497 qpid-server main ERROR KERNEL - MicroKernel.scanDeployments() : Not exiting this time but continuing anyway. 2015-07-27 22:48:19,986 qpid-server main INFO c.a.r.f.c.Container - Container.getConnectors() : Container.getConnectors() - Serving http on port - 8083 2015-07-27 22:48:20,453 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : WebService 2015-07-27 22:48:20,456 qpid-server WebServer INFO o.e.jetty.server.Server - Server.doStart() : jetty-8.0.4.v20111024 2015-07-27 22:48:21,794 qpid-server WebServer INFO o.e.j.w.StandardDescriptorProcessor - StandardDescriptorProcessor.visitServlet() : NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet 2015-07-27 22:48:22,285 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 22:48:22,286 qpid-server WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/opt/apigeeInstall/apigee4/share/apigee-1.0.0.655.a4b746c.1501081353/webapps/api/},/opt/apigeeInstall/apigee4/share/apigee/webapps/api 2015-07-27 22:48:22,491 qpid-server main INFO o.a.c.f.i.CuratorFrameworkImpl - CuratorFrameworkImpl.start() : Starting 2015-07-27 22:48:22,979 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : ZooKeeperService 2015-07-27 22:48:23,130 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:48:23,280 qpid-server main-EventThread INFO o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.postState() : State change: SUSPENDED 2015-07-27 22:48:24,281 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:48:24,463 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:48:25,429 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:48:25,544 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:48:26,546 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:48:26,673 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:48:26,674 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-07-27 22:48:26,690 qpid-server main ERROR c.a.f.FeatureFlagService - FeatureFlagsPersistance.init() : Error building FeatureFlagsPersistance. Is Zookeeper configured? 2015-07-27 22:48:26,691 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : FeatureFlagService 2015-07-27 22:48:27,173 qpid-server WebServer INFO o.e.j.s.AbstractConnector - AbstractConnector.doStart() : Started SelectChannelConnector@0.0.0.0:8083 STARTING 2015-07-27 22:48:27,716 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:48:28,024 qpid-server main WARN SERVICES.COMMUNICATION - MBeanManager.getRMIConnectorProperties() : RMI host is not provided. Fetching host address from network registry 2015-07-27 22:48:29,552 qpid-server Apigee-Main-4 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 22:48:29,596 qpid-server Apigee-Main-4 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:48:29,821 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:48:29,866 qpid-server Apigee-Main-9 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 22:48:29,875 qpid-server Apigee-Main-9 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:48:30,307 qpid-server main WARN RMI - RMIServer.createSocketFactories() : SSL is disabled for 0.0.0.0:1102 2015-07-27 22:48:30,783 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : CommunicationService 2015-07-27 22:48:30,909 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : OrganizationService 2015-07-27 22:48:31,194 qpid-server main INFO HTTP.SERVICE - HTTPTransport.start() : Idle threshold of concurrent channels is 1000 (1000 per selector). Idle channles will be close when this limit is reached. 2015-07-27 22:48:31,351 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPTransport 2015-07-27 22:48:31,359 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : HTTPProxy 2015-07-27 22:48:31,394 qpid-server main INFO FILE_LOGGER - BufferedWriter.start() : Starting logging thread 2015-07-27 22:48:31,402 qpid-server main INFO FILE_LOGGER - SizeBasedFileRotator.() : Creating sizeBased rotator with maxSize 104857600 2015-07-27 22:48:31,403 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : EventService 2015-07-27 22:48:31,405 qpid-server Thread-2 INFO FILE_LOGGER - BufferedWriter.run() : Signalling threadInit completion 2015-07-27 22:48:31,446 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : RepositoryService 2015-07-27 22:48:31,453 qpid-server main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : AlertService 2015-07-27 22:48:31,734 qpid-server main WARN REGISTRATION - CommunicationInfoBuilder.loadIPAddresses() : CommunicationInfoBuilder.loadIPAddresses : File /opt/apigee/ec2.properties is missing. Will use localhost as default for all the hosts 2015-07-27 22:48:31,985 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:48:32,097 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:48:32,915 192.168.227.132 qpid-server Apigee-Main-9 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-07-27 22:48:32,919 192.168.227.132 qpid-server Apigee-Main-9 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:48:32,941 192.168.227.132 qpid-server Apigee-Main-3 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/up/ 2015-07-27 22:48:32,945 192.168.227.132 qpid-server Apigee-Main-3 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) 2015-07-27 22:48:33,101 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:48:33,214 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:48:34,216 192.168.227.132 qpid-server main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-07-27 22:48:35,319 192.168.227.132 qpid-server main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:48:35,320 192.168.227.132 qpid-server main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, reason: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable 2015-07-27 22:48:35,324 192.168.227.132 qpid-server main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Got a life cycle exception while starting service [ServerRegistrationService, Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable] : com.apigee.zookeeper.ZooKeeperException{ code = zookeeper.ErrorCheckingPathExistence, message = Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable, associated contexts = []} 2015-07-27 22:48:35,338 192.168.227.132 qpid-server main ERROR KERNEL - MicroKernel.deployAll() : MicroKernel.deployAll() : Error in deploying the deployment : CommunicationService com.apigee.zookeeper.ZooKeeperException: Error while checking path existence for path : /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:335) ~[zookeeper-1.0.0.jar:na] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:319) ~[zookeeper-1.0.0.jar:na] at com.apigee.services.repository.zookeeper.ZKRepository.exists(ZKRepository.java:256) ~[repository-impl-1.0.0.jar:na] at com.apigee.services.repository.RepositoryServiceImpl.exists(RepositoryServiceImpl.java:223) ~[repository-impl-1.0.0.jar:na] at com.apigee.registration.info.StatusBuilder.build(StatusBuilder.java:26) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.buildServerInfo(ServerRegistrationServiceImpl.java:265) ~[registration-1.0.0.jar:na] at com.apigee.registration.ServerRegistrationServiceImpl.start(ServerRegistrationServiceImpl.java:115) ~[registration-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:69) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:59) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deployDependantServices(ServiceDeployer.java:255) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:62) ~[microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:147) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:130) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.start(MicroKernel.java:124) [microkernel-1.0.0.jar:na] at com.apigee.kernel.MicroKernel.main(MicroKernel.java:75) [microkernel-1.0.0.jar:na] Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /regions/dc-1/pods/central/servers/5b1d3e6d-909e-4d63-a714-efdbdbab87df/reachable at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:327) ~[zookeeper-1.0.0.jar:na] ... 14 common frames omitted