Apigee Install Validation Test fails

[rmishra]

Hello,

I have a 7 node install of Apigee on a private cloud. I have two nodes running the edge-router and the message processor.

Here are some basic details about the install:

Details of our installation:

OS Version: Red Hat Enterprise Linux Server release 7.3 (Maipo)

Apigee Version: 4.17.01

1. /opt/apigee is owned by the apigee:apigee user and group
2. /opt/nginx is owned by apigee:apigee user and group
3. apigee-edge-router is running as apigee
4. nginx is running as apigee
5. SELinux is set to Permissive

Description of our problem:

1. The problem was diagnosed when we were testing our install, using the information @ http://docs.apigee.com/private-cloud/latest/test-install . All the validation was successful EXCEPT curl -H "Host: VALIDATE.apigee.com" http://10.45.8.66:59001/v1/apigee/healthCheck....FAILED!
2. Upon analysis, we realized that this happens because a VHOST is not being created on Nginx when the validation script runs. The .conf files in conf.d get marked as .conf.bad and the Nginx Error logs show

2017/05/22 16:27:26 [warn] 69826#69826: the "user" directive makes sense only if the master process runs with super  -user privileges, ignored in /opt/nginx/conf/nginx-configtest-69814.conf:
22017/05/22 16:27:26 [alert] 69844#69844: setrlimit(RLIMIT_NOFILE, 400000) failed (1: Operation not permitted)2017/05/22 16:27:26 [alert] 69845#69845: setrlimit(RLIMIT_NOFILE, 400000) failed (1: Operation not permitted)
2017/05/22 16:27:26 [alert] 63949#63949: worker process 68228 exited on signal 11 (core dumped)
2017/05/22 16:27:27 [alert] 63949#63949: worker process 68229 exited on signal 11 (core dumped)

3..The error around setrlimit goes away when i manually raise the number of limit for apigee user and the user directive error goes away when i comment out the user directive on nginx.conf. But we can't find out why the nginx worker process is exiting (moreover, do we really have to make the other changes manually?)

Summary of the problem: I can't figure out why nginx is creating a core dump and exiting.

I see the same "nginx worker exiting" error when i try to provision an org, the org gets created, the environment gets created, the management server ui shows the virtual host but nginx does not listen on that port. Furthermore, it renames the conf.d files to conf.bad

Any pointers towards resolving this problem will be appreciated

Regards,

Rahul

Note:Typing questions and comments in all caps and using unnecessary punctuation (!!1! / ???) will not increase the likelihood of your question being answered.

[#2]

Can you follow this https://community.apigee.com/articles/28987/nginx-playbook-how-to-troubleshoot-bad-config-file.html and share the output?

[rmishra]

I can't access the link, permission is denied.

[#4]

@rmishra Try the below steps

• Rename the bad config file custorg_test_default.conf.bad to its original file name custorg_test_default.conf

[root@apiproxy-t11 conf.d]# ls -l 
total 96-rw-r--r--1 apigee apigee 522Jul2008:410-default.conf.bad 
-rw-r--r--1 apigee apigee 577Jul2008:420-fallback.conf 
-rw-r--r--1 apigee apigee 1062Jul2008:180-map.conf 
-rw-r--r--1 apigee apigee 1887Jul2008:42 custorg_test_default.conf

• Ran the config test as shown below:

[root@apiproxy-t11 conf.d]# /opt/nginx/scripts/apigee-nginx configtest

[rmishra]

Getting somewhere,

/opt/nginx/scripts/apigee-nginx configtest

nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /opt/nginx/conf/nginx.conf:2

md5_dgst.c(80): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!

/opt/nginx/scripts/apigee-nginx: line 93: 25958 Aborted (core dumped) ${nginx} -t -c ${conffile} $FLAG

I am also seeing the same Nginx issue on an unrelated question

https://community.apigee.com/questions/41894/apigee-dev-portal-installation-issue-apigee-lb-doe.html

Both my kernel and the underlying SSL library are fips enabled

$cat /proc/sys/crypto/fips_enabled 1 c

$openssl version OpenSSL 0.9.8zg-fips 11 Jun 2015

I see that the in one of nginx conf's the sslCiphers say !MD5, so not sure how MD5 is kicking in..

[#6]

@rmishra, Can you upgrade the latest openssl version and check if you still see the same issue?

cc @jhunt@apigee.com

[rmishra]

I did upgrade openssl which is being used by nginx

Actually, we had two versions of open ssl - one from centrify and another from RHEL, we now aliased openssl to RHEL version and it gives

[apigee@*** ~]$ openssl version

OpenSSL 1.0.1e-fips 11 Feb 2013

But i still have the same error

[apigee@*** ~]$ /opt/nginx/scripts/apigee-nginx configtest

nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /opt/nginx/conf/nginx.conf:2

md5_dgst.c(80): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!

/opt/nginx/scripts/apigee-nginx: line 93: 94409 Aborted (core dumped) ${nginx} -t -c ${conffile} $FLAG

[rmishra]

So, we worked with @Russell Blewitt via a support ticket, we have found a temporary work around for this issue by deleting /etc/system-fips.

This disables the self checking of binaries

Obviously, disabling fips isn't a permanent option, so we are working with Russel for a final solution.

[#9]

@rmishra Good to know that you are unblocked. thanks for the update.
I have the same openssl library, system-fips and don't see the issues in my setup.BTW just wanted to check if you deleted the .bad files and restarted the routers after you updated the openssl version.

[rmishra]

@Maruti Chand Yes i did delete the .bad files , it still didn't work.We are in the process of getting a patch from Apigee