Hi All,
We are having issue with router not starting whenever a change is done to VHOST configs. I checked the logs for router and could find below errors/warning whenever router was restarted/started.
2017-06-14 11:26:31,401 WebServer INFO c.a.g.f.v.EntityCRUDInterceptor - EntityCRUDInterceptor.<clinit>() : Failed to retrieve security properties with the exception {} java.lang.NullPointerException: null at java.util.Properties$LineReader.readLine(Properties.java:434) ~[na:1.8.0_121] at java.util.Properties.load0(Properties.java:353) ~[na:1.8.0_121] at java.util.Properties.load(Properties.java:341) ~[na:1.8.0_121] at com.apigee.gateway.flow.validation.EntityCRUDInterceptor.<clinit>(EntityCRUDInterceptor.java:74) ~[validation-1.0.0.jar:na] at java.lang.Class.forName0(Native Method) [na:1.8.0_121] at java.lang.Class.forName(Class.java:264) [na:1.8.0_121] at com.apigee.rest.framework.CustomJAXRSInvoker.<init>(CustomJAXRSInvoker.java:78) [rest-1.0.0.jar:na] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [na:1.8.0_121] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [na:1.8.0_121] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [na:1.8.0_121] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [na:1.8.0_121] at java.lang.Class.newInstance(Class.java:442) [na:1.8.0_121] at com.apigee.rest.framework.container.RestServlet.init(RestServlet.java:51) [rest-1.0.0.jar:na] at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:456) [jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024] at org.eclipse.jetty.servlet.ServletHolder.doStart(ServletHolder.java:276) [jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) [jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024]
and
2017-06-14 11:26:31,441 main WARN REGISTRATION - CommunicationInfoBuilder.loadIPAddresses() : CommunicationInfoBuilder.loadIPAddresses : File /opt/apigee/ec2.properties is missing. Will use localhost as default for all the hosts 2017-06-14 11:26:31,845 main INFO REGISTRATION - ServerRegistrationServiceImpl.storeServerRegistrationPath() : Registering the server uuid aa834eaf-8a34-4098-8f80-e66b406260f4 with region dc-1 and pod gateway information 2017-06-14 11:26:31,888 WebServer WARN o.e.j.u.c.AbstractLifeCycle - AbstractLifeCycle.setFailed() : FAILED SelectChannelConnector@0.0.0.0:8081 FAILED: java.net.BindException: Address already in use java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) ~[na:1.8.0_121] at sun.nio.ch.Net.bind(Net.java:433) ~[na:1.8.0_121] at sun.nio.ch.Net.bind(Net.java:425) ~[na:1.8.0_121] at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[na:1.8.0_121] at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[na:1.8.0_121] at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:172) ~[jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024] at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:300) ~[jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024] at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:249) ~[jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) [jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024] at org.eclipse.jetty.server.Server.doStart(Server.java:273) [jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) [jetty-webapp-8.0.4.v20111024.jar:8.0.4.v20111024] at com.apigee.rest.framework.container.Container.start(Container.java:78) [rest-1.0.0.jar:na] at com.apigee.web.WebServiceImpl$1.run(WebServiceImpl.java:39) [webserver-1.0.0.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
For "Address already in use" issue I can guess that it is to do with some ports already listening on the ports which router is trying to listen. I checked netstat for all listening ports and could find the old port for VHOST listening. Shouldn't the restart take care of unbinding old and binding new ports?
Could anyone please help me understand these two exception and the cause of it?
Thanks...
Solved! Go to Solution.
Thanks, @Maruti Chand, issue was with root user owning the processes related to apigee.
To fix this I had to forcefully kill the router process owned by root and restart it with the normal sudoer user under apigee user with apigee-service.
Looks like the routers stop is not happening completely.
Can you try apigee-service edge-router stop and check for ps -ef | grep router and start it when no process is running.
When the router starts properly, I think the first might go away. @Mohammed Zuber
@Maruti Chand, you are correct apigee-service edge-router stop is not stopping the router.
bash:$ sudo /opt/apigee/apigee-service/bin/apigee-service edge-router stop Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0 Server is already stopped. apigee-service: edge-router: Not running (DEAD)
But ps -ef | grep router is still showing that router is running.
bash:$ ps -ef | grep router root 17435 1 1 Jun09 ? 01:56:31 /usr/java/latest/bin/java -classpath /opt/apigee/edge-router-4.16.09-0.0.891/conf::/opt/apigee/edge-router-4.16.09-0.0.891/lib//*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//actions/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//analytics/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//analytics/libraries/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//analytics/services/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//cps/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//gateway/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//gateway/libraries/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//gateway/services/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//gateway/steps/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//infra/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//infra/libraries/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//infra/services/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//kernel/*:/opt/apigee/edge-router-4.16.09-0.0.891/lib//thirdparty/*:/opt/apigee/edge-router-4.16.09-0.0.891/webapps/api/:/opt/apigee/edge-router-4.16.09-0.0.891/utils/scripts/sampling/:/opt/apigee/edge-router-4.16.09-0.0.891/utils/scripts/utilities/ -Xms256m -Xmx512m -XX:MaxPermSize=128m -XX:MetaspaceSize=128m -Djava.security.auth.login.config=/opt/apigee/edge-router-4.16.09-0.0.891/conf/jaas.config -Dinstallation.dir=/opt/apigee/edge-router-4.16.09-0.0.891 -Dprofile=router -Dlib.dir.name=lib/infra/services,lib/gateway/services,lib/gateway/steps,lib/analytics/services -Dpython.verbose=error -Ddebugsession.timeout=120 -Ddefault.response.format=json -Dauthenticated.user.header.name=X-Apigee-Current-User -Dbundle.validation.enabled=false -Dbundle.validation.schema.enabled=true -Dcasssandra.maxConnectTimeInMillis=-1 -Drbac.number.of.tries.for.email.search=1 -Dvirtual.host.restricted.ports=8080 -Djavacallout.policy.validate=false -Dastyanax.datastores=taurus-datastore,kms-datastore,dc-datastore,cache-datastore,counter-datastore,keyvaluemap-datastore,application-datastore,audit-datastore,apimodel-datastore,auth-datastore,scheduler-datastore,edgenotification-datastore -Djson.request.schema.validation.enabled=true -Djavax.xml.transform.TransformerFactory=net.sf.saxon.TransformerFactoryImpl -Dvalidation.entity.name.enabled=true -Dcheckmarx.scan.validator.enabled=true -Djavax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom=org.apache.xpath.jaxp.XPathFactoryImpl -Dvalidation.targetendpoint.connection.check.enabled=true -DshutdownServicesOnStop=true -Dsun.net.maxDatagramSockets=2048 -Dhas.monetization=${microkernel_hasMonetization} -Dmigration.mode.status=false -Djute.maxbuffer=10485750 -Dqpid.session.command_limit=524288 -Dqpid.session.byte_limit=8388608 -Djava.io.tmpdir=/var/tmp -Dlog.level=INFO -Dregion=dc-1 -Dpod=gateway -Djsse.enableSNIExtension=false -Dcom.warrenstrange.googleauth.rng.algorithm=SHA1PRNG -Dcom.warrenstrange.googleauth.rng.algorithmProvider=SUN -Djdk.tls.allowUnsafeServerCertChange=true -Djava.security.properties=../conf/jvmsecurity.properties -Djava.security.nssprovider.enabled=false -Dbean.retry.enabled=true -DuseStringDeduplication=false -Dlicense.file.location=/opt/apigee/customer/conf/license.txt -Dconf.dir=/opt/apigee/edge-router-4.16.09-0.0.891/conf -Ddata.dir=/opt/apigee/var/log/edge-router com.apigee.kernel.MicroKernel
How can I stop the router properly?
Looks like some issues with your installation. Your router is running as root user but it should be apigee. check this http://docs.apigee.com/private-cloud/latest/installation-requirements#creatingtheapigeeuser.
Can you go through that link and check if you took care of all the pre-requisites? @Mohammed Zuber
Thanks, @Maruti Chand, issue was with root user owning the processes related to apigee.
To fix this I had to forcefully kill the router process owned by root and restart it with the normal sudoer user under apigee user with apigee-service.
@Mohammed Zuber thank you for sharing the solution.
Please make sure /opt/apigee (all files and folders) are owned by apigee:apigee
User | Count |
---|---|
2 | |
1 | |
1 | |
1 | |
1 |