Router (4.16.09.00) seems to neglect 'message-processor' reachability-flag

Not applicable

Summary:

During startup of a message-processor we set the reachability-flag to false but as soon as we register the 'message-processor to an organization/environment, requests are sent to this message-processor even when its reachability is still 'false'. This causes 404 errors as the api-proxies are not configured yet on this message-processor. Especially as configuration of the proxies is triggered by registration of this message-processor to an organization/environment.

Question:

How to determine that a message-processor is ready to serve all requests (all proxies configured and loaded)? And how to prevent messages to be sent to this message-processor before it is completely ready?

Details:

The following installation steps are executed:

/opt/apigee/apigee-service/bin/apigee-service edge-router setup -f <configuration-file>
/opt/apigee/apigee-service/bin/apigee-service edge-message-processor setup -f <configuration-file>
/opt/apigee/apigee-service/bin/apigee-service edge-mint-message-processor setup -f <configuration-file>
/opt/apigee/apigee-service/bin/apigee-service edge-message-processor restart
/opt/apigee/apigee-service/bin/apigee-service edge-message-processor wait_for_ready
/opt/apigee/apigee-service/bin/apigee-service edge-router restart
/opt/apigee/apigee-service/bin/apigee-service edge-router wait_for_ready

After this is 'message-processor' set to unreachable (BTW: router is also set to unreachable)

curl -u "<credentials>" -X POST "http://<management-server>:8080/v1/servers/<uuid>" -d "reachable=false"

Now the 'message-processor' is registered to an organization/environment via

curl -u "<credentials>" -X POST "http://<management-server>:8080/v1/organizations/<org>/environments/<env>/servers" -d "action=add&uuid=<uuid>"

At this moment all nginx-configuration are updated and reloaded and even when the reachability-flag of this 'message-processor' is set to 'false' requests are sent to this 'message-processor' which it is not able to process yet which results in 404 (not-found) errors.

Solved Solved
1 7 808
1 ACCEPTED SOLUTION

Not applicable

I found a way to have routers honor the 'message-processor' reachability flag. In 'router.properties' there are two properties:

conf_load_balancing_load.balancing.driver.nginx.target.pool.check and

conf_load_balancing_load.balancing.driver.nginx.target.pool.check_http_send which can be used to have 'edge-router' follow the 'message-processor' reachability flag and by that it is possible to update 'message-processor' without outages. See below for the values of the properties to achieve this:

conf_load_balancing_load.balancing.driver.nginx.target.pool.check=interval=5000 rise=1 fall=2 timeout=3000 type=http port=8082 default_down=true

conf_load_balancing_load.balancing.driver.nginx.target.pool.check_http_send="GET /v1/servers/self/reachable HTTP/1.0\\\\r\\\\nConnection: keep-alive\\\\r\\\\n\\\\r\\\\n"

View solution in original post

7 REPLIES 7

sgilson
Participant V

We recentyle updated the doc on reachbility. To make the MP unreachable, you now use this command:

/opt/apigee/apigee-service/bin/apigee-service edge-message-processor stop

You can see the updated doc here.

Stephen

Hmmm how would that help with loading the proxies? I assume when the 'message-processor' is stopped it will not be able to load api-proxies so this will not solve my issue.

Asked differently: what is the recommended way to add a RMP to an Apigee-system without errors (404s) as there currently does not seem a way to have the service running to load 'api-proxies' while not being sent requests by (other) Routers.

Hi William,

Can you elaborate on what you mean by "how would that help with loading proxies"?

MPs will take API proxies deployments 1) as they happen (as result of deploy event on MS) or 2) as part of the booting process as they MP loads configuration it will deploy all applicable API proxies for the environments associated to the MP.

As Stephen mentioned above, the reachability flag on the R and MPs is applicable to 15.XX and older versions of the software. 16.XX do not provides the functionality.

To disable MP reachability on 16XX and beyond, you need to stop the MP:

/opt/apigee/apigee-service/bin/apigee-service edge-message-processor stop

If you want to wait for the MP to be able to take traffic, you can run the following command after MP start:

/opt/apigee/apigee-service/bin/apigee-service edge-router wait_for_ready

I have been discussing with Engineering the possibility of brining back the reachability flag functionality as part of a future version. But for now, it is not available on 16XX, 17.01 or 17.05.

Please let me know if you need any additional info.

Hi @Maudrit,

the issue is the following:

current healthckeck configuration on the nginx is a rest call to the API port (8998 or the SSL configured one) with a header X-Apigee.heartbeat true.

Once the MP is started/registered it begins to download deployments for the environments he's a member of. But at the same time immediately responds with 200 OK on the healthcheck even if the APIs are not yet downloaded. We've observed this in 17.09 and 18.01.

As a result there's a period from the start of the MP when routers sent traffic to the MP and getting 404 because the particular APIs are not yet downloaded and operational.

One possible mitigation is to se the MP unreachable when registering and make it reachable again either after some time period or after wait_for_ready exits with 0.

So William's proposal is valid and we're planning to test the same.

BR,

Plamen

@sgilson I thinkj they are looking for more along the lines of blue green delpoyments for the RMP; from what I am reading he wants to "pre-warm" the mp into the environment but not let it service traffic until they decide its ready too.

Not applicable

I found a way to have routers honor the 'message-processor' reachability flag. In 'router.properties' there are two properties:

conf_load_balancing_load.balancing.driver.nginx.target.pool.check and

conf_load_balancing_load.balancing.driver.nginx.target.pool.check_http_send which can be used to have 'edge-router' follow the 'message-processor' reachability flag and by that it is possible to update 'message-processor' without outages. See below for the values of the properties to achieve this:

conf_load_balancing_load.balancing.driver.nginx.target.pool.check=interval=5000 rise=1 fall=2 timeout=3000 type=http port=8082 default_down=true

conf_load_balancing_load.balancing.driver.nginx.target.pool.check_http_send="GET /v1/servers/self/reachable HTTP/1.0\\\\r\\\\nConnection: keep-alive\\\\r\\\\n\\\\r\\\\n"

Hello William,

Steps mentioned by you are working. Thanks for this valuable advice. Don't know why apigee disabled this in latest versions. Is there any link where I can found all values we can use in properties file with their descriptions.

Thanks.