Getting deployment error code "application.bootstrap.FailedToConfigure"

[adevegowda]

When we are trying to deploy one of our API Proxies, we are getting the following error:

Error in deployment for environment dev.The revision is deployed, but traffic cannot flow. com.apigee.kernel.exceptions.spi.UncheckedException{ code = application.bootstrap.FailedToConfigure, message = Configuration failed, associated contexts = []}

screen-shot-2017-12-25-at-122513-pm.png

[adevegowda · Accepted solution]

1. Checked the Message Processor logs and noticed that the following error occurred during deployment of the API Proxy:

2017-12-22 00:13:49,057  Apigee-Main-87446 INFO  MESSAGING.CONFIGURATION - MessageProcessorServiceImpl.configure() : configuring [DeployEvent{organization='myorg', application='myapi', applicationRevision='42', deploymentSpec=basepath=/;env=dev;, deploymentID=null}]

2017-12-22 00:13:49,263  Apigee-Main-87446 INFO  MESSAGING.RUNTIME - AbstractConfigurator.loadXMLConfigurations() : loading entities [] for class class com.apigee.messaging.config.beans.TargetServerBean

2017-12-22 00:13:49,318  Apigee-Main-87446 ERROR c.a.p.h.d.DNSCachedAddress - DNSCachedAddress.refresh() : Unable to resolve host : input-prd-sps.cloud.splunk.com: Name or service not known

2017-12-22 00:13:49,323  Apigee-Main-87446 ERROR MESSAGING.RUNTIME - AbstractConfigurator.handleUpdate() : Fatal error deploying proxy: {}

com.apigee.rest.framework.ValidationException: Invalid syslog config: Invalid HostName 'input-prd-sps.cloud.splunk.com' for Syslog handler
        at com.apigee.messaging.runtime.destinations.SyslogDestination.<init>(SyslogDestination.java:44) ~[message-processor-1.0.0.jar:na]
        at com.apigee.messaging.runtime.destinations.SysLoggerFactory.getInstance(SysLoggerFactory.java:39) ~[message-processor-1.0.0.jar:na]
        at com.apigee.messaging.runtime.destinations.DestinationRegistry.newDestination(DestinationRegistry.java:44) ~[message-processor-1.0.0.jar:na]
        at com.apigee.steps.messagelogging.MessageLoggingStepDefinition.populateDestinations(MessageLoggingStepDefinition.java:118) ~[message-logging-1.0.0.jar:na]
        at com.apigee.steps.messagelogging.MessageLoggingStepDefinition.handleAdd(MessageLoggingStepDefinition.java:99) ~[message-logging-1.0.0.jar:na]
...<snipped>

2. Tried to telnet to the host name from the Message Processor and got the following error:

[adevegowda@rmp1 ~]$ telnet input-prd-sps.cloud.splunk.com
telnet: input-prd-sps.cloud.splunk.com: Name or service not known
input-prd-sps.cloud.splunk.com: Unknown host
[adevegowda@rmp1 ~]$

3. Checked the API Proxy and found that the host name "input-prd-sps.cloud.splunk.com" was used in one of the MessageLogging policy.

4. Replaced the Host name "input-prd-sps.cloud.splunk.com" with the proper host name. Saved the changes, undeployed and deployed the API Proxy successfully without any errors.

We need to ensure that we are using the proper/valid host name and port # in MessageLogging, ServiceCallout or any other policies to avoid this type of errors.

[#3]

@AMAR DEVEGOWDA

Three of our running cluster started showing the same error.

Following is the screenshot of error:

We are using Edge 4.17.09 on private cloud, with CentOS 7.2

Status of services on all RMP:

Interestingly I don't see system.log for message processor in all three clusters.

[root@fqdn]# ls -ltr
total 47799084
-rw-r--r--. 1 apigee apigee           0 Feb 21 07:06 kvm-service.log
-rw-r--r--. 1 apigee apigee           0 Feb 21 07:06 kvm-client.log
-rw-r--r--. 1 apigee apigee           0 Feb 21 07:06 quota-service.log
-rw-r--r--. 1 apigee apigee           0 Feb 21 07:06 quota-client.log
-rw-r--r--. 1 apigee apigee           0 Feb 21 07:06 expiringcounter-service.log
-rw-r--r--. 1 apigee apigee           0 Feb 21 07:06 expiringcounter-client.log
-rw-r--r--. 1 apigee apigee           0 Feb 21 07:06 startupruntimeerrors.log
-rw-r--r--. 1 apigee apigee           0 Feb 21 07:06 access.log
-rw-r--r--. 1 apigee apigee           0 Feb 21 07:06 audit.log
-rw-r--r--. 1 apigee apigee           0 Feb 21 07:06 envvalidator.log
-rw-r--r--. 1 apigee apigee           0 Feb 21 07:06 advisory.log
-rw-r--r--. 1 apigee apigee           0 Feb 21 07:06 cps_delta_migration.log
-rw-r--r--. 1 apigee apigee       22345 Feb 21 08:04 transactions.log437034443676139.tmp
-rw-r--r--. 1 apigee apigee       67154 Feb 22 00:09 system-2018-02-21.0.log.gz
-rw-r--r--. 1 apigee apigee       12871 Feb 23 00:09 system-2018-02-22.0.log.gz
-rw-r--r--. 1 apigee apigee       12778 Feb 24 00:09 system-2018-02-23.0.log.gz
-rw-r--r--. 1 apigee apigee       12920 Feb 25 00:09 system-2018-02-24.0.log.gz
-rw-r--r--. 1 apigee apigee    52439473 Feb 25 08:19 system.log352377893494692.tmp
-rw-r--r--. 1 apigee apigee 48851705856 Feb 25 13:07 system.log352517493464058.tmp
-rw-r--r--. 1 apigee apigee     5316608 Feb 26 00:19 system.log409405389236704.tmp
-rw-r--r--. 1 apigee apigee    36647530 Mar  4 00:09 system.log927593931590177.tmp

<br>

I tried restarting the services on one RMP using "apigee-all restart" and the services don't come up.

Still NO system.log file created:

What are the possible causes of this error? We need to debug, as all 3 of our clusters are practically unusable.

[#4]

Status of all services, initially (these are all dead now, when I attempted to restart these)

<strong>[root@fqdn]# apigee-all status
+ apigee-service edge-message-processor status
apigee-service: edge-message-processor: OK
+ apigee-service edge-router status
apigee-service: edge-router: OK<br></strong>

System.log NOT available:

[root@fqdn]# tail -100f /opt/apigee/var/log/edge-message-processor/logs/system.log
tail: cannot open ‘/opt/apigee/var/log/edge-message-processor/logs/system.log’ for reading: No such file or directory
tail: no files remaining
[root@fqdn]#<br><strong></strong>

Checked status of one of the API-Proxy, using following GET:

http://<FQDN_OF_MS_NODE>:8080/v1/organizations/<org-name>/apis/<api-proxy>/deployments

Reponse body:

{
    "environment": [
        {
            "name": "test",
            "revision": [
                {
                    "configuration": {
                        "basePath": "/",
                        "steps": []
                    },
                    "name": "1",
                    "server": [
                        {
                            "error": "com.apigee.kernel.exceptions.spi.UncheckedException{ code = application.bootstrap.FailedToConfigure, message = Configuration failed, associated contexts = []}",
                            "status": "error",
                            "type": [
                                "message-processor"
                            ],
                            "uUID": "a0a990a3-9fa9-433f-b5e1-ef7b1f39ea08"
                        },
                        {
                            "error": "com.apigee.kernel.exceptions.spi.UncheckedException{ code = application.bootstrap.FailedToConfigure, message = Configuration failed, associated contexts = []}",
                            "status": "error",
                            "type": [
                                "message-processor"
                            ],
                            "uUID": "a0f0ed01-23a2-4086-a666-b5e257a05ac6"
                        },
                        {
                            "error": "com.apigee.kernel.exceptions.spi.UncheckedException{ code = application.bootstrap.FailedToConfigure, message = Configuration failed, associated contexts = []}",
                            "status": "error",
                            "type": [
                                "message-processor"
                            ],
                            "uUID": "40404ec0-d6db-49ee-978f-11db1127c1c1"
                        },
                        {
                            "error": "com.apigee.kernel.exceptions.spi.UncheckedException{ code = application.bootstrap.FailedToConfigure, message = Configuration failed, associated contexts = []}",
                            "status": "error",
                            "type": [
                                "message-processor"
                            ],
                            "uUID": "b5e87c96-b9de-4322-aa15-f565324bfc14"
                        },
                        {
                            "status": "deployed",
                            "type": [
                                "router"
                            ],
                            "uUID": "2b24dc59-1694-4c5f-8b9f-9b4400bdb102"
                        },
                        {
                            "status": "deployed",
                            "type": [
                                "router"
                            ],
                            "uUID": "17f14f58-960a-45ed-9112-91d9d9fdc38c"
                        },
                        {
                            "status": "deployed",
                            "type": [
                                "router"
                            ],
                            "uUID": "aa12b78c-fff6-4e8a-90c9-e26a9464ffaf"
                        },
                        {
                            "status": "deployed",
                            "type": [
                                "router"
                            ],
                            "uUID": "1baf718b-eff1-47e8-b73d-85daa82d83aa"
                        }
                    ],
                    "state": "error"
                }
            ]
        }
    ],
    "name": "legacy_api-v1-CurrentUser",
    "organization": "org-name"
}<br>