Javacallout shareflow deployment error on Apigee X

[ranismita1]

Hi @ssvaidyanathan 

With all the CICD solutions, we use Apigee deploy maven & config plugin for all the Apigee platform (SaaS, X & Hybrid) . And have great understanding for these plugins when it comes to any javascript based proxy or the sharedflow. This is the first time I am working with the sharedflow which has javacallout policy where I am struggling a bit. 

When we run our CI pipeline, it successfully uploads the bundle to the Apigee X but when it makes POST call to deploy, it keeps running the GET calls in loop and later time out. Please refer the below error on the console:

"Error on CI server console"

GET https://apigee.googleapis.com/v1/organizations/test-org/environments/test1/sharedflows/hello-test/re...

accept: [application/json]

accept-encoding: [gzip]

authorization: [Bearer [Not shown in log]

x-goog-api-format-version: 2

DEBUG] returning deployed status: false

INFO] Getting Deployment Info for Revision: 9

[INFO] Using the service account file to generate a token

[DEBUG] **Access Token**

"Error on Apigee X"

"instance "us-east1-instance" reported error steps.javacallout.JavaCalloutInstantiationFailed: "Failed to instantiate the JavaCallout Class xyz"

Same bundle I was able to upload and deploy it manually. But through the pipeline using Maven not able to get through this error. 

Have all the required dependencies added to the pom (message-flow & expressions). and have no error on the console to debug this issue specifically. Also we are using the right version of the plugins. (2.1.0) 

project structure for the reference:

 X (base dir)

• sharedflowbundle
  • manifests
  • policies
  • resources
    • java
      • jar file
    • jsc
  • sharedflows
    • default.xml
  • test.xml
• pom.xml

• shared-pom.xml

Any suggestion what could I be missing here? 

[dchiesa1]

I don't have any good ideas here.

You are saying that when you deploy with the maven plugin, you see an error, and when you deploy via the UI, you do not see an error. The deployment via maven plugin happens the same way a deployment would happen via UI, or via API.  There's no difference. To my mind, the choice of tool/interface that is used to deploy, is not the root cause of the difference you are seeing. 

It may be due to a coincidental cause - for example when you deploy from maven, maybe the jar is not the same jar as when you deploy manually.  Or, when you deploy from maven, maybe you are pointing to a different Apigee organization or environment.  Possibly there are some other issues that could be co-incident with the use of the maven plugin and the CI/CD tool. 

In my experience, the most common root causes of the "Failed to instantiate the JavaCallout Class xyz" error are:

• there is a dependency that is missing at runtime
• the callout class or one of its dependencies attempts to perform a restricted action. Such as, open a local file, or read/write system properties. 

To debug THAT, I would suggest simplifying the Java callout to remove dependencies and remove actions. Make it drop-dead simple.  If you can get that working, and it deploys successfully, then you know that one of the things you had removed ( a dependency or an action) is what has caused the "Failed to instantiate" error. 

[ssvaidyanathan]

@ranismita1 - Can you check if you have enabled "overwrite" in your pom.xml for the copy-resources goal. If you have, remove it or set it to false and see. 

<plugin>
                <artifactId>maven-resources-plugin</artifactId>
                <version>2.6</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-resources</goal>
                        </goals>
                        <configuration>
                            <overwrite>false</overwrite>
                            <encoding>UTF-8</encoding>
                             <echo message="basedir parent : ${basedir}"/>
                            <outputDirectory>${basedir}/target/sharedflowbundle</outputDirectory>
                            <resources>
                                <resource>
                                    <directory>sharedflowbundle</directory>
                                </resource>
                            </resources>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

 It could be possible that the overwrite is changing the contents of the jar file too. 

[ranismita1]

Yes it was set to true and later I changed it to false as per your suggestion. Attaching the pom.xml for the reference. 

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <artifactId>hello-Encryptor</artifactId>
    <packaging>jar</packaging>
    <name>hello-Encryptor</name>

    <parent>
        <groupId>com.demo.apigeex</groupId>
        <artifactId>common-parent</artifactId>
        <version>1.0.0-SNAPSHOT</version>
        <relativePath>../pom.xml</relativePath>
    </parent>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.7</java.version>
        <testng.version>6.8.7</testng.version> <!-- current: 6.9.4 -->
        <jmockit.version>1.7</jmockit.version>
        <slf4jVersion>1.6.1</slf4jVersion>
    </properties>
    <dependencies>
        <dependency>
            <groupId>com.apigee.edge</groupId>
            <artifactId>message-flow</artifactId>
            <version>1.0.0</version>
        </dependency>
        <dependency>
            <groupId>com.apigee.edge</groupId>
            <artifactId>expressions</artifactId>
            <version>1.0.0</version>
        </dependency>

        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-core</artifactId>
            <version>2.8.47</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>RELEASE</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>javax</groupId>
            <artifactId>javaee-api</artifactId>
            <version>6.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>1.7.5</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.6.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.5</version>
            <scope>test</scope>
        </dependency>

    </dependencies>

    <build>
    <finalName>hello-Encryptor</finalName>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <executions>
                <execution>
                    <id>copy-dependencies</id>
                    <phase>prepare-package</phase>
                    <goals>
                        <goal>copy-dependencies</goal>
                    </goals>
                    <configuration>
                        <outputDirectory>${project.build.directory}/lib</outputDirectory>
                        <overWriteReleases>false</overWriteReleases>
                        <overWriteSnapshots>false</overWriteSnapshots>
                        <overWriteIfNewer>true</overWriteIfNewer>
                    </configuration>
                </execution>
            </executions>
        </plugin>

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>2.3.2</version>
            <configuration>
                <source>${java.version}</source>
                <target>${java.version}</target>
            </configuration>
        </plugin>

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <version>2.3.2</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>jar</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-resources-plugin</artifactId>
            <version>2.6</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>copy-resources</goal>
                    </goals>
                    <configuration>
                        <overwrite>false</overwrite>
                        <encoding>UTF-8</encoding>
<!--                        <echo message="basedir parent : ${target.root.dir}"/>-->
                        <outputDirectory>${target.root.dir}/target/sharedflowbundle</outputDirectory>
                        <resources>
                            <resource>
                                <directory>sharedflowbundle</directory>
                            </resource>
                        </resources>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-antrun-plugin</artifactId>
            <executions>
                <execution>
                    <phase>package</phase>
                    <configuration>
                        <tasks>
                            <copy todir="${target.root.dir}/sharedflowbundle/resources/java">
                                <fileset dir="sharedflowbundle/resources/java">
                                    <include name="hello-Encryptor.jar"/>
                                    <include name="sharedflowbundle/**" />
                                </fileset>
                                <mapper type="flatten"/>
                            </copy>
                        </tasks>
                    </configuration>
                    <goals>
                        <goal>run</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
    </build>
</project>

[ranismita1]

When I changed apigee.option to async from override it worked. I referred to this problem and the solution provided: https://www.googlecloudcommunity.com/gc/Apigee/Hybrid-deployment-hangs-when-override-option-is-used-...

But it looks like it got fixed with the latest version as per thee post but why I am seeing this behavior now again any idea?

Because I need to get back to "override" option for the seem less deployment. 

[ssvaidyanathan]

The option has nothing to do with the issue you are seeing. Try override with the latest version. I still dont know why you want to "maven-antrun-plugin". Not sure what you are trying to do? Are you trying to use the same pom to generate the jar file and put it in the resources folder before the proxy bundle is created and deployed? 

I do something like this - https://github.com/ssvaidyanathan/apigee-java-callout-pemdecode/blob/main/callout/pom.xml

It generates the jar file and then copies it over to the proxy's resources/java directory. 

Then the pom.xml within that will package the bundle and deploy it. See if you can do something like that. In your pipeline, you will call the pom under the callout directory to generate the jar and then the pipeline calls the pom within the proxy that will generate the zip (including the jar) and then deploy it to Apigee

[ranismita1]

Yes you are right there is nothing to do with the maven plugin version. I did not validate the deployment on Apigee before as it was showing deployed in the pipeline but originally it didn't get deployed. 

I have simplified the pom.xml (removed unwanted code) and also updated the plugin version with right apigee.option and I am back to the same old error.

[ERROR] Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.3.1:deploy (default-cli) on project Data-Encryptor: : MojoExecutionException: Deployment error: instance "us-east1-instance" reported error steps.javacallout.JavaCalloutInstantiationFailed: "Failed to instantiate the JavaCallout Class com.yodlee.CustomEncryption" -> [Help 1]

[dchiesa1]

check my suggestions in the above comment: check dependencies, check permissions.

[ranismita1]

Finally, I was able to solve this to change the maven-resources-plugin filtering option to false. Earlier it was set to true and somehow it was messing up with the jar file. 

<filtering>false</filtering>

We can close this issue as resolved.

Thank you @dchiesa1  & @ssvaidyanathan  for your quick responses. 

[ssvaidyanathan]

Glad you found it. In your sample code, you had not mentioned about filtering. Had I seen it, would have pointed it out right away.