16.09 private cloud installation problematic anomaly

[ccovney]

Hi everyone,

During the initial installation setup for edge private cloud version 16.09, I've run into a truly baffling error: the installation claims that the apigee-service-4.16.09 is already installed and therefore does not create any of the necessary file/folder structure!

I'm following the steps in official Apigee docs here:

http://docs.apigee.com/private-cloud/latest/install-edge-apigee-setup-utility

The installation step in question (from above docs) and its subsequently problematic output are the following:

sudo bash /tmp/bootstrap_4.16.09.sh apigeeuser=uName apigeepassword=pWord

=== Installing apigee-service:
+ yum install -y apigee-service
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Package apigee-service-4.16.09-0.0.486.noarch already installed and latest version
Nothing to do
+ rpm -q apigee-serviceapigee-service-4.16.09-0.0.486.noarch

=== Storing JAVA_HOME=/usr/lib/jvm/jre in /opt/apigee/etc/defaults.sh
bootstrap_4.16.09.sh: id=root:root

=== End work - success!

[root@BSEX1VAPG01 tmp]# /opt/apigee/apigee-service/bin/apigee-service apigee-setup install
-bash: /opt/apigee/apigee-service/bin/apigee-service: No such file or directory

You'll notice in the above output at the very end, when I try to execute the next installation step, the apigee files are not there!

Why is this happening? How can I wipe the slate clean so that I can retry the installation without any issues?

Thanks in advance for any help or insight!

Best,

Chris

[DChiesa]

Hi Chris!

Happy New Year !

Have you tried removing the package?

yum remove apigee-service

After that, re-run the bootstrap script.

If that doesn't work, you can also try re-installing it:

yum reinstall apigee-service

[ccovney]

Thanks @Dino! Happy new year to you too!

By running the yum remove, I was able to get past my initial error and am now one step further. However, now I'm seeing a similar error on the very two next steps in the installation. When I run:

/opt/apigee/apigee-service/bin/apigee-service apigee-setup install

I get the following output:

-----------------------

Running transaction

Installing : apigee-setup-4.16.09-0.0.939.noarch 1/1

/var/tmp/rpm-tmp.glks02: line 6: /opt/apigee/apigee-lib/apigee-lib.sh: No such file or directory

warning: %post(apigee-setup-4.16.09-0.0.939.noarch) scriptlet failed, exit status 1

Non-fatal POSTIN scriptlet failure in rpm package apigee-setup-4.16.09-0.0.939.noarch

Verifying : apigee-setup-4.16.09-0.0.939.noarch 1/1

Installed:

apigee-setup.noarch 0:4.16.09-0.0.939

Complete!

-----------------

The next step then fails due to the apigee-setup file/folder structure not being there:

[root@BSEX1VAPG01 tmp]# /opt/apigee/apigee-setup/bin/setup.sh -p ds -f /tmp/myConfig.txt
-bash: /opt/apigee/apigee-setup/bin/setup.sh: No such file or directory

I do see an apigee-setup-4.16.09... directory:

[root@BSEX1VAPG01 tmp]# ls /opt/apigee
apigee-service/ apigee-service-4.16.09-0.0.486/ apigee-setup-4.16.09-0.0.939/
<br>

Not sure why the apigee-lib and apigee-setup dirs are not getting created.

I also noticed when looking back at the outputs from the various steps that there seemed to be one more issue buried in there. when running:

bash /tmp/bootstrap_4.16.09.sh apigeeuser=uName apigeepasswordpWord

I see the following warnings and error in the output:

-------------------------

=== Configuring package manager:

+ rpm -qa apigee*

+ rpm -e apigee-repo

warning: file /etc/yum/vars/apigeestage: remove failed: No such file or directory

warning: file /etc/yum/vars/apigeerepohost: remove failed: No such file or directory

warning: file /etc/yum/vars/apigeerepobasepath: remove failed: No such file or directory

warning: file /etc/yum/vars/apigeereleasever: remove failed: No such file or directory

warning: file /etc/yum/vars/apigeeprotocol: remove failed: No such file or directory

warning: file /etc/yum/vars/apigeepriority: remove failed: No such file or directory

warning: file /etc/yum/vars/apigeeexclude: remove failed: No such file or directory

warning: file /etc/yum/vars/apigeecredentialswithat: remove failed: No such file or directory

+ rpm -e apigeeprio-repo

error: package apigeeprio-repo is not installed

(error can be ignored)

+ yum install -y https://intralinks:***@software.apigee.com//apigee-repo-1.0-6.x86_64.rpm

Loaded plugins: product-id, search-disabled-repos, subscription-manager

.....

....

---------------------------------

Not sure how decisive this error is. Any idea how to proceed?

Thanks a lot for your help, Dino!

Best,

Chris

[DChiesa]

The file you are missing is provided by the "apigee-lib" package.

yum install apigee-lib 

would get it.

But... there's something else wrong with your system, or your install. You shouldn't be having the basic errors you are seeing. Something else is afoot here. You can yum install the missing package here, but... what else is missing? Why? I don't know.

Maybe you should tear everything down, and start over?

yum remove apigee*
yum remove baas* 
rm -rf /opt/apigee/data/*
rm -rf /tmp/bootstrap.sh
yum clean all
yum remove apigee-repo

And then ... download bootstrap.sh again.

 curl -s https://<user>:<pass>@software.apigee.com/bootstrap.sh -o /tmp/bootstrap.sh

..and then follow along with the install instructions from the top.

[ccovney]

Thanks @Dino

The steps for tearing everything down worked! I was able to install the datastore on the first 3 nodes and the management server on node 1. (I'm doing a 5 node install btw)

However, now the Router/MP installation on node 2 is failing with the below message:

-------------

apigee-configutil: edge-router: # OK

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

Server is already stopped.

apigee-service: edge-router: Not running (DEAD)

apigee-configutil: edge-router: # OK

apigee-service: edge-router: Not running (NO_LOCKFILE)

apigee-service: edge-router: status=2, continuing

apigee-service: edge-router: OK

apigee-service: edge-router: edge-router is running

Checking for router on 127.0.0.1 port 8081 ..... OK

Checking for router uuid .............................................................................................................................................................................................................................................................................................................

Unable to get uuid for router.

Error: setup.sh: /opt/apigee/apigee-service/bin/apigee-service exited with unexpected status 1

---------------------

I was able to find the following errors in the following egde router logs:

------------------------

/opt/apigee/var/log/edge-router/edge-router.log:

java.io.IOException: Cannot run program "/opt/nginx/scripts/apigee-nginx": error=2, No such file or directory

at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)

-----------------------

/opt/apigee/var/log/edge-router/logs/system.log:

2017-01-05 13:11:09,012 main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed

....

....

2017-01-05 12:37:12,907 main INFO NGIX-ADAPTOR - NginxLoadBalancingAdaptor.clearExistingConfigs() : Attempting to clear existing conf files at /opt/nginx/conf.d

2017-01-05 12:37:12,911 main WARN LB-SYNC - FileWriter$1.postVisitDirectory() : dir found under conf.d

2017-01-05 12:37:12,912 main WARN NGIX-ADAPTOR - NginxUtil.checkConfig() : config testing with all files

2017-01-05 12:37:12,912 main INFO NGIX-ADAPTOR - NginxUtil.checkConfig() : Testing config of Nginx on this machine using command /opt/nginx/scripts/apigee-nginx configtest and file

2017-01-05 12:37:12,915 main INFO NGIX-ADAPTOR - NginxLoadBalancingAdaptor.initialize() : Nginx config test failed

2017-01-05 12:37:12,915 main ERROR LOAD-BALANCER - LoadBalancingManagementServiceImpl.start() : Could not Initialize adaptor nginx

2017-01-05 12:37:12,915 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : LoadBalancingManagementService

2017-01-05 12:37:12,918 main INFO ZOOKEEPER - ZooKeeperServiceImpl.registerPathWatcher() : attaching watcher on path /regions/dc-1/pods/gateway/bindings/servers/{uuid}/organizations/{org}/environments/{env}, async is true

2017-01-05 12:37:12,921 main INFO ZOOKEEPER - ZooKeeperServiceImpl.registerPathWatcher() : attaching watcher on path /regions/dc-1/pods/gateway/bindings/orgs/{org}, async is true

2017-01-05 12:37:12,924 main INFO ZOOKEEPER - ZooKeeperServiceImpl.registerPathWatcher() : attaching watcher on path /regions/dc-1/pods/gateway/bindings/orgenvs/{org}/environments/{env}, async is true

2017-01-05 12:37:12,928 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : MessageProcessorManagementService

2017-01-05 12:37:12,928 main ERROR Router-Service - RouterServiceImpl.start() : Router not started because, Load Balancing could not be initialized

2017-01-05 12:37:12,936 main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Got a life cycle exception while starting service [RouterService, Nginx Initialization failed] : {}

com.apigee.errors.http.server.ServiceFailureException: Nginx Initialization failed

at com.apigee.proxy.service.RouterServiceImpl.start(RouterServiceImpl.java:65) ~[message-router-proxy-1.0.0.jar:na]

----------------------------

It appears that the issue points to a root cause which is related to nginx setup. Not sure why nginx setup would fail by itself.

Is there somewhere else I can look for clues? I have the process very well understood to this point, so tearing everything down again (and again) is not a problem for me.

Thanks again for all your help so far, Dino!

Best,

Chris

[#6]

Hi @Chris Covney, you still seeing the issues?

I have seen this error earlier if the /opt/nginx/conf.d doesn't get cleared and if it has stale data.
your log says it Attempted to clear existing conf files at /opt/nginx/conf.d but not sure if it was successful. Can you delete that manually and try again?

[#7]

Thanks @Dino, I was facing the same issue in 4.18.01, tearing out and reinstall worked for me, Thank you very much..

[cmbrown]

Based on the nginx error you posted; what permissions are you running the router install under ? check that it is root versus apigee user

[ccovney]

Hi @Christin

Good question! I am running the entire install as root.

Thanks for looking into this!

Best,

Chris

[ccovney]

Hi @Maruti Chand & @Dino

I was able to get one step further by removing all contents from /opt/nginx. Now the router UUID is found; however, the check to see if the router is up fails:

---------------------------------

apigee-service: edge-router: edge-router is running

Checking for router on 127.0.0.1 port 8081 ............. OK

Checking for router uuid 649f58ba-bb61-469e-b169-ceb955961242

Checking if router is up .............................................................

router is not up.

Error: setup.sh: /opt/apigee/apigee-service/bin/apigee-service exited with unexpected status 1

-------------------------------

When I dig into the logs this time around, I see the same error in:

/opt/apigee/var/log/edge-router/edge-router.log:

java.io.IOException: Cannot run program "/opt/nginx/scripts/apigee-nginx": error=2, No such file or directory

at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)

-----------------------------

However, I see a few different errors in:

/opt/apigee/var/log/edge-router/logs/system.log:

2017-01-05 13:11:09,012 main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed

...

2017-01-06 11:18:55,249 qtp371397455-42 ERROR REST - CustomJAXRSInvoker.performInvocation() : CustomJAXRSInvoker.performInvocation : Method com.apigee.registration.ServersAPI.isServerCompletelyUp threw an exception.

...

2017-01-06 11:18:55,253 qtp371397455-42 ERROR REST - ExceptionMapper.toResponse() : Error occurred : Service not up yet.

com.apigee.registration.ServersAPI.isServerCompletelyUp(ServersAPI.java:106)

sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2017-01-06 11:18:55,255 qtp371397455-42 ERROR REST - ExceptionMapper.toResponse() : Returning error response : ErrorResponse{errorCode = null, errorMessage = Service not up yet.}

...

2017-01-06 11:18:55,576 main ERROR LOAD-BALANCER - LoadBalancingManagementServiceImpl.start() : Could not Initialize adaptor nginx

...

2017-01-06 11:18:55,603 main ERROR Router-Service - RouterServiceImpl.start() : Router not started because, Load Balancing could not be initialized

2017-01-06 11:18:55,614 main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Got a life cycle exception while starting service [RouterService, Nginx Initialization failed] : {}

com.apigee.errors.http.server.ServiceFailureException: Nginx Initialization failed

at com.apigee.proxy.service.RouterServiceImpl.start(RouterServiceImpl.java:65) ~[message-router-proxy-1.0.0.jar:na]

---------------------------

I'm quite baffled how this could be happening. I don't see any other errors in the various log files. Any ideas where to look and/or how to proceed? Thanks again for all your help so far guys!

Best,

Chris

EDIT: Still have not been able to make progress on this. I did make changes to various permission settings in the folder structure around nginx, but to no avail.

[ccovney · Accepted solution]

Directly before installing the router and message processor component, run the below commands, which are a viable workaround for this issue:

mkdir /opt/apigee/etc/edge-router.d
echo RUN_USER=root >> /opt/apigee/etc/edge-router.d/RUN_USER.sh
chown -R apigee:apigee /opt/apigee/etc/edge-router.d

Best,

Chris