Errors starting management server

Report Inappropriate Content · 12-10-2015 09:08 AM

Recently installed 4.15.07.00 version in one of our environments and receiving the following error when restarting the management server.

*******************************************

2015-12-10 10:52:07,450 main INFO KERNEL - MicroKernel.start() : MicroKernel.start() : MicroKernel start initiated... 2015-12-10 10:52:07,455 main INFO KERNEL - MicroKernel.logJavaAndOSProperties() : Java/OS Properties - {java.ext.dirs=/usr/lib/jvm/java-1.7.0-oracle-1.7.0.45.x86_64/jre/lib/ext:/usr/java/packages/lib/ext, java.runtime.name=Java(TM) SE Runtime Environment, java.runtime.version=1.7.0_45-b18, java.specification.version=1.7, java.vendor=Oracle Corporation, java.class.version=51.0, java.specification.name=Java Platform API Specification, java.vendor.url=http://java.oracle.com/, java.security.auth.login.config=/home/apigee/apigee4/conf/apigee/management-server/jaas.config, java.awt.printerjob=sun.print.PSPrinterJob, java.net.preferIPv4Stack=true, os.arch=amd64, java.vm.version=24.45-b08, os.name=Linux, java.home=/usr/lib/jvm/java-1.7.0-oracle-1.7.0.45.x86_64/jre, os.version=2.6.32-573.3.1.el6.x86_64, java.endorsed.dirs=/usr/lib/jvm/java-1.7.0-oracle-1.7.0.45.x86_64/jre/lib/endorsed, java.version=1.7.0_45, java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment, java.vm.specification.version=1.7, java.vendor.url.bug=http://bugreport.sun.com/bugreport/, java.vm.specification.name=Java Virtual Machine Specification, java.specification.vendor=Oracle Corporation, java.vm.vendor=Oracle Corporation, java.vm.info=mixed mode, java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib, java.vm.name=Java HotSpot(TM) 64-Bit Server VM, java.vm.specification.vendor=Oracle Corporation, java.io.tmpdir=/var/tmp}

2015-12-10 10:52:07,503 main INFO KERNEL.CONFIG - KernelConfiguration.loadKernelProperties() : KernelConfiguration.loadKernelProperties() : MicroKernel properties : {debugsession.tracesize.max=5120, rbac.number.of.tries.for.email.search=1, shutdownServicesOnStop=true, default.response.format=json, log.level=INFO, record.fault=true, sun.net.maxDatagramSockets=2048, qpid.session.command_limit=524288, debugsession.timeout=120, region=dc-1, validation.targetendpoint.connection.check.enabled=true, bundle.validation.enabled=false, has.monetization=false, javax.xml.xpath.XPathFactory=http://java.sun.com/jaxp/xpath/dom=org.apache.xpath.jaxp.XPathFactoryImpl, python.verbose=error, lib.dir.name=lib/infra/services,lib/gateway/services,lib/gateway/steps,lib/analytics/services, javax.xml.transform.TransformerFactory=net.sf.saxon.TransformerFactoryImpl, pod=central, jsse.enableSNIExtension=false, json.request.schema.validation.enabled=true, checkmarx.scan.validator.enabled=false, validation.entity.name.enabled=false, virtual.host.restricted.ports=8080, com.warrenstrange.googleauth.rng.algorithm=SHA1PRNG, authenticated.user.header.name=X-Apigee-Current-User, qpid.session.byte_limit=8388608, bundle.validation.schema.enabled=true, com.warrenstrange.googleauth.rng.algorithmProvider=SUN, migration.mode.status=false, astyanax.datastores=kms-datastore,dc-datastore,cache-datastore,counter-datastore,keyvaluemap-datastore,application-datastore,audit-datastore,apimodel-datastore,auth-datastore,scheduler-datastore,edgenotification-datastore, jute.maxbuffer=10485750, raise.nodeprefixed.fault=true, java.io.tmpdir=/var/tmp} 2015-12-10 10:52:08,908 main ERROR KERNEL - MicroKernel.scanDeployments() : MicroKernel.scanDeployments() : Some modules and services were present in the profile but not in the class path 2015-12-10 10:52:08,909 main ERROR KERNEL - MicroKernel.scanDeployments() : Missing modules: [] 2015-12-10 10:52:08,909 main ERROR KERNEL - MicroKernel.scanDeployments() : Missing services: [CumulativeMetricsService] 2015-12-10 10:52:08,909 main ERROR KERNEL - MicroKernel.scanDeployments() : Not exiting this time but continuing anyway. 2015-12-10 10:52:08,923 main INFO FILE_LOGGER - BufferedWriter.start() : Starting logging thread 2015-12-10 10:52:08,928 main INFO FILE_LOGGER - SizeBasedFileRotator.<init>() : Creating sizeBased rotator with maxSize 134217728 2015-12-10 10:52:08,928 Thread-1 INFO FILE_LOGGER - BufferedWriter.run() : Signalling threadInit completion 2015-12-10 10:52:08,987 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : MonitoringService 2015-12-10 10:52:08,992 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : MachineKeyService 2015-12-10 10:52:09,048 main INFO c.a.r.f.c.Container - Container.getConnectors() : Container.getConnectors() - Serving http on port - 8080 2015-12-10 10:52:09,080 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : WebService 2015-12-10 10:52:09,080 WebServer INFO o.e.jetty.server.Server - Server.doStart() : jetty-8.0.4.v20111024 2015-12-10 10:52:09,187 WebServer INFO o.e.j.w.StandardDescriptorProcessor - StandardDescriptorProcessor.visitServlet() : NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet 2015-12-10 10:52:09,240 main WARN SERVICES.COMMUNICATION - MBeanManager.getRMIConnectorProperties() : RMI host is not provided. Fetching host address from network registry 2015-12-10 10:52:09,243 WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/home/apigee/apigee4/share/apigee-1.0.0.1078.fe7934c.1509010011/webapps/api/},/home/apigee/apigee4/share/apigee/webapps/api 2015-12-10 10:52:09,244 WebServer INFO o.e.j.s.h.ContextHandler - ContextHandler.callContextInitialized() : started o.e.j.w.WebAppContext{/,file:/home/apigee/apigee4/share/apigee-1.0.0.1078.fe7934c.1509010011/webapps/api/},/home/apigee/apigee4/share/apigee/webapps/api 2015-12-10 10:52:10,356 WebServer INFO o.e.j.s.AbstractConnector - AbstractConnector.doStart() : Started SelectChannelConnector@0.0.0.0:8080 STARTING 2015-12-10 10:52:10,357 main WARN RMI - RMIServer.createSocketFactories() : SSL is disabled for 0.0.0.0:1099 2015-12-10 10:52:10,424 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : CommunicationService 2015-12-10 10:52:10,651 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : OrganizationService 2015-12-10 10:52:10,660 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : LogService 2015-12-10 10:52:10,670 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : DebugService 2015-12-10 10:52:10,734 main INFO o.a.c.f.i.CuratorFrameworkImpl - CuratorFrameworkImpl.start() : Starting 2015-12-10 10:52:10,766 main INFO ZOOKEEPER - ZooKeeperServiceImpl.start() : Use binocular pathwatcher service: false 2015-12-10 10:52:10,766 main INFO ZOOKEEPER - ZooKeeperServiceImpl.start() : Use curator based pathwatcher service: false 2015-12-10 10:52:10,766 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : ZooKeeperService 2015-12-10 10:52:10,773 main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-12-10 10:52:10,814 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : LicensingService 2015-12-10 10:52:10,878 main-EventThread INFO o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.postState() : State change: SUSPENDED 2015-12-10 10:52:10,886 main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-12-10 10:52:10,944 qtp1174217921-27 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-12-10 10:52:10,949 qtp1174217921-27 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null

org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)

2015-12-10 10:52:11,250 main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-12-10 10:52:11,351 main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-12-10 10:52:11,767 main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed 2015-12-10 10:52:11,822 Timer-0 INFO c.a.l.ScheduledLicenseValidator - ScheduledLicenseValidator.run() : License found and is valid !!! 2015-12-10 10:52:11,868 main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path:/featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-12-10 10:52:11,868 Timer-0 INFO ZOOKEEPER - ZooKeeperServiceImpl.getChildren() : Retry getting children for path: /regions, reason: KeeperErrorCode = ConnectionLoss for /regions 2015-12-10 10:52:11,868 main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /featureflag, reason: KeeperErrorCode = ConnectionLoss for /featureflag 2015-12-10 10:52:11,872 main ERROR c.a.f.FeatureFlagService - FeatureFlagsPersistance.init() : Error building FeatureFlagsPersistance. Is Zookeeper configured? 2015-12-10 10:52:11,876 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : FeatureFlagService 2015-12-10 10:52:11,880 main INFO FILE_LOGGER - BufferedWriter.start() : Starting logging thread 2015-12-10 10:52:11,880 main INFO FILE_LOGGER - SizeBasedFileRotator.<init>() : Creating sizeBased rotator with maxSize 104857600 2015-12-10 10:52:11,881 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : EventService 2015-12-10 10:52:11,881 Thread-12 INFO FILE_LOGGER - BufferedWriter.run() : Signalling threadInit completion 2015-12-10 10:52:11,908 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : RepositoryService 2015-12-10 10:52:11,972 qtp1174217921-30 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-12-10 10:52:11,974 qtp1174217921-30 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null

org.apache.cxf.jaxrs.JAXRSInvoker.checkResultObject(JAXRSInvoker.java:336) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:211) org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)

2015-12-10 10:52:12,985 qtp1174217921-28 WARN REST - APIRegistry.locateSubResource() : No match found for request path /servers/self/uuid/ 2015-12-10 10:52:12,987 qtp1174217921-28 ERROR REST - ExceptionMapper.toResponse() : Error occurred : null

************************************************

sudheendras

@Suresh Karne Sorry to hear that! Have you raised a support ticket? If not, please do it here. Support will be able to assist you with the installation immediately.

Report Inappropriate Content

Yes we have raised a support ticket.

#957328

Report Inappropriate Content

This issue is resolved. We learned from support that we need to have at least 2 of the Zookeeper nodes up before we start other components. We were at 4.14.04 before the upgrade and looks like Zookeeper was not clustered in that release, hence we were able to start fine even when we had one instance of Zookeeper up.

sudheendras

Thanks for heads-up!