On-premises VHost port 9001 issue

Not applicable

Hi ,

I have configured virtual host port 9001 as per documentation.

below is sa-org.txt file for your reference.

[root@Rootadmin etc]# cat /tmp/sa-org.txt
IP1=xxxxxxxx
MSIP="$IP1"

ADMIN_EMAIL="xxxxxx"
APIGEE_ADMINPW=XXX

NEW_USER="y"
USER_NAME=xxxx
FIRST_NAME=Ram
LAST_NAME=Mangi
USER_PWD=XXXXX
ORG_NAME=POC #lowercase letters only, no spaces, underscores, or periods
ORG_ADMIN="$USER_NAME"
ENV_NAME=prod
VHOST_PORT=9001
VHOST_NAME=default
VHOST_ALIAS="$IP1:9001"
AXGROUP=axGroupName

But port 9001 is not listening and we are unable to telnet to port 9001.i can see port 9001 under virtual host in UI. but connectivity doesnt exist to this port.

Can you help me in fixing this.

regards

ramakrishna

0 23 1,430
23 REPLIES 23

@RamaKrishna , You need to open the port. Change firewall rules to do same.

@Anil Sagar

yes, i have done that,but no luck . please see below output

[root@Rootadmin ~]# iptables -nL
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:1521
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:23
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:9001
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:9000

@Anil Sagar

awaiting your help here

Can someone help me here

Can you run below curl and see what VH are configured properly.

Try below checks..

{{MGMTSVR}}/v1/organizations/{{ORG}}/environments/{{ENV}}/virtualhosts/default

verify classficiation tree:

curl -v localhost:8082/v1/classification/tree

netstat -anp |grep 9001

Verify any iptables blocking it..

Verify RMP logs..

You should able to resolve it if you closely check.. It is hard with minimal information with out log information..

-Vinay

mp-system-2017-09-110log.zip

@vinay

As suggetsed by you i have ran below commands.


[root@Rootadmin logs]# curl -v localhost:8082/v1/classification/tree
* About to connect() to proxy 127.0.0.1 port 3128 (#0)
* Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0)
> GET http://localhost:8082/v1/classification/tree HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:8082
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< Date: Thu, 21 Sep 2017 09:27:44 GMT
< Content-Type: application/json
< Date: Thu, 21 Sep 2017 09:27:44 GMT
< Content-Length: 366
< Proxy-Connection: keep-alive
< Connection: keep-alive
<
[ {
"condition" : "(header.host matches 10.56.156.22:9001)",
"virtualHost" : {
"env" : "prod",
"name" : "default",
"org" : "POC",
"tree" : {
"elements" : [ {
"application" : "Helloworld",
"basePath" : "/hello",
"name" : "default",
"revision" : "1"
} ],
"name" : "IdentificationTree"
}
}
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0
}

------------------------------------------------------------------------------------------------------------------
[root@Rootadmin logs]# iptables -nL
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:1521
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:23
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:9001
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:9000

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
----------------------------------------------------------------------------------------------------------------------
[root@Rootadmin logs]# netstat -anp |grep 9001
[root@Rootadmin logs]#


Netstat to port 9001not returning anything

-----------------------------------------------------

Attached router and Mp system Logs. Could you help me finding the issue.

any suggestions please?

you didn't seem to run the very first command he sent to show us the actual vhost output...could you please run that and post back ?

sorry. Missed it. here it is.

[root@Rootadmin logs]# curl -v http://10.56.156.22:8080/v1/organizations/POC/environments/prod/virtualhosts/default -u poc@xxxx.com
Enter host password for user 'poc@xxxx.com':
* About to connect() to proxy 127.0.0.1 port 3128 (#0)
* Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0)
* Server auth using Basic with user 'poc@xxx.com'
> GET http://10.56.156.22:8080/v1/organizations/POC/environments/prod/virtualhosts/default HTTP/1.1
> Authorization: Basic cG9jQHRlY2htYWhpbmRyYS5jb206Q29uZmlkZW50aWEx
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 10.56.156.22:8080
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< Date: Mon, 25 Sep 2017 10:07:14 GMT
< Content-Type: application/json
< Date: Mon, 25 Sep 2017 10:07:14 GMT
< Vary: Accept-Encoding, User-Agent
< Content-Length: 133
< Proxy-Connection: keep-alive
< Connection: keep-alive
<
{
"hostAliases" : [ "10.56.156.22:9001" ],
"interfaces" : [ ],
"listenOptions" : [ ],
"name" : "default",
"port" : "9001"
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0

Hi @ramakrishna

The outputs look reasonable.

Can you please provide a couple more answers:

1. [offtop, but] any particular reason why you decided to ignore a suggestion in the comment, lowercase letters only:

ORG_NAME=POC #lowercase letters only, no spaces, underscores, or periods

2. Can you please attache the contents of the folders?

/opt/apigee/edge-router/nginx/conf.d

/opt/apigee/edge-router/nginx/logs

/opt/apigee/var/log/edge-router

3. What is an output of

curl -v http://10.56.156.22:9001

when you run it at 10.56.156.22 server?

@ylesyuk

Thanks for your response.

below are my answers.

1)

1. [offtop, but] any particular reason why you decided to ignore a suggestion in the comment, lowercase letters only:

Ans: My Bad. its a silly reason. i think caps Lock button ON that type , didn't noticed.

2) Logs attached to this ticket.

3)

[root@Rootadmin ~]# curl -v http://10.56.156.22:9001
* About to connect() to proxy 127.0.0.1 port 3128 (#0)
* Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0)
> GET http://10.56.156.22:9001/ HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 10.56.156.22:9001
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 502 Connection refused
< Content-Type: text/html
* no chunk, no close, no size. Assume close to signal end
<
* Closing connection #0
<html><body><h1>502 Connection refused</h1><p><a href='http://cntlm.sf.net/'>Cntlm</a> proxy failed to complete the request.</p></body></html>[root@Rootadmin ~]#attachment.zip

So there are at least two problems in your nginx conf.d folder:

1. The vhost /conf.d/POC_prod_default.conf.bad

conf file was marked as .bad. nginx does it when something is wrong.

2. The logs/error.log-* files contain

invalid port in upstream "10.56.156.22:-1" in /opt/nginx/conf.d/0-upstream-pools.conf:2

Your conf.d/0-upstream-pools.conf file has an extra section:

upstream 10.56.156.22_-1 {

server 10.56.156.22:-1;

keepalive 1024;

check interval=5000 rise=1 fall=2 timeout=3000 type=http default_down=true;

check_keepalive_requests 360;

check_http_send "GET / HTTP/1.1\r\nConnection: keep-alive\r\nX-Apigee.heartbeat: true\r\n\r\n";

check_http_expect_alive http_2xx http_3xx;

}

with negative port which confuses nginx.

I'd assume someone tried to adjust this nginx configuration on Sept 15.

Can you try to:

1.

remove the section with negative port, remove .bad prefix in vhost conf file and restart nginx.

If this will not help,

2. stop router,

backup then remove contents of conf.d/* directory

start router

and see if nginx regenerates conf files correctly.

@ylesyuk

As suggested, i have removed the negative port .and removed .bad prefix in vhost conf file and restarted router.after restart it didnt generate .bad vhost config file.and port 9001 is now listening(current issue is fixed)


But i came across a new issue after restarting router.I am getting below error when i try to deploy or undeploy a proxy.Attached screenshot , router and mp system logs for your reference.error.zip

Error in deployment for environment prod.
The revision is deployed, but traffic cannot flow. com.apigee.kernel.exceptions.spi.UncheckedException{ code = messaging.runtime.UnknownEventReceived, message = Received an unknown event with description DELETE Application /organizations/POC/apiproxies/Helloworld/revisions/1/, associated contexts = []}; and

below is

0-upstream-pools.conf file after removing negative port and post restart

[root@Rootadmin conf.d]# cat 0-upstream-pools.conf
upstream 10.56.156.22_8998 {
server 10.56.156.22:8998;

keepalive 1024;
check interval=5000 rise=1 fall=2 timeout=3000 type=http default_down=true;
check_keepalive_requests 360;
check_http_send "GET / HTTP/1.1\r\nConnection: keep-alive\r\nX-Apigee.heartbeat: true\r\n\r\n";
check_http_expect_alive http_2xx http_3xx;

}
[root@Rootadmin conf.d]#

that might happen after vhost repairs.

can you please undeploy/deploy it?

if you cannot undeploy it a 'normal' way, use force undeploy.

http://docs.apigee.com/management/apis/post/organizations/%7Borg_name%7D/apis/%7Bapi_name%7D/revisio...

@ylesyuk

when i try to deploy weather api proxy from on premise apigee, i am getting below error message. Coudl you suggest solution to this. screenshot attached.capture2.png

Error Deploying Revision 1 to prod
Invalid virtual host reference secure. Context Revision:1;APIProxy:weatherapi;Organization:POC;Environment:prod

the message is clear: you do not have secure vhost.

either remove its reference or create it.

@ylesyuk

I have removed its reference. But i am getting below error message when i fire a request.Attached screenshot too.capture3.png

Error Sending Request
Could not connect to http://10.56.156.22:9001/hello. Make sure the URL is correct.

Copy the url, ie, http://10.56.156.22:9001/hello

and send it from other Chrome's tab.

@ylesyukI am getting below message when i open it in browser.

{"fault":{"faultstring":"The Service is temporarily unavailable","detail":{"errorcode":"messaging.adaptors.http.flow.ServiceUnavailable"}}}

Is it 503 returned by your backend?

If yes, then Edge is working correctly. You need to sort out your backend.

@ylesyuk

I am getting 503.But i am getting the same error for other proxy as well.

i used helloworld proxy url

http://mocktarget.apigee.net

and Twilio proxy with url

https://api.twilio.com/2010-04-01/Accounts

but getting the same error code. And those are sample proxies provided by APIGEE.

Attached Trace session for two proxies. request you to please go through it and help in fixing the issue.

trace-1509625354952.zip

> As suggested, i have removed the negative port .and removed .bad prefix

> in vhost conf file and restarted router.after restart it didnt generate .bad

> vhost config file.and port 9001 is now listening(current issue is fixed)


Good. @Alex Toombs Thank you for giving it a glance, diagnosing it correctly and providing the fix!!!