We are trying to export from a 17.05 OPDK all our proxies with apigee-migrate-tool but we are seeing this error.
Fatal error: Cannot read property 'length' of undefined
The error happens after some proxies are already exported. We have around 200 proxies and the size varies between 12MB to a few KB.
Looking at the management-server logs (system.log) there are quite a few PoolTimeoutException like this one
2018-03-29 16:16:22,346 org:org-example pool-3-thread-2490 ERROR DATASTORE.CASSANDRA - AstyanaxCassandraClient.fetchDynamicCompositeColumns() : Error while querying columnfamily : [api_proxy_revisions_r21, org-example@proxy-example] for rowkey:{} com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=None(0.0.0.0):0, latency=2017(2017), attempts=2]Timed out waiting for connection at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:231) ~[astyanax-core-1.56.43-apigee.jar:na] at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:198) ~[astyanax-core-1.56.43-apigee.jar:na] at com.netflix.astyanax.connectionpool.impl.LeastOutstandingExecuteWithFailover.borrowConnection(LeastOutstandingExecuteWithFailover.java:74) ~[astyanax-core-1.56.43-apigee.jar:na]
The above error starts with the proxy that failed to be exported.
This post seems somewhat related though it was while doing a load test for the KVM. I was looking to change that same property (maxactive_cassandra_connections) though which of all these files I should change and which could be a good number to run the script? (Assuming that the change has to be done in the management-server, please correct me if am wrong)
/opt/apigee/edge-management-server/conf/cps-management.properties:kvm.maxactive_cassandra_connections=8 /opt/apigee/edge-management-server/conf/cps-management.properties:kms.maxactive_cassandra_connections=8 /opt/apigee/edge-management-server/conf/cps-management.properties:quota.maxactive_cassandra_connections=8 /opt/apigee/edge-management-server/conf/repository.properties:cassandra.defaults.maxactive_cassandra_connections=10 /opt/apigee/edge-management-server/conf/auth-datastore.properties:maxactive_cassandra_connections=10 /opt/apigee/edge-management-server/conf/cache.properties:distributed.maxactive_cassandra_connections=10 /opt/apigee/edge-management-server/conf/keyvaluemap-datastore.properties:maxactive_cassandra_connections=5 /opt/apigee/edge-management-server/conf/keymanagement-datastore-config.properties:maxactive_cassandra_connections=10 /opt/apigee/edge-management-server/conf/cps-runtime.properties:kvm.maxactive_cassandra_connections=8 /opt/apigee/edge-management-server/conf/cps-runtime.properties:kms.maxactive_cassandra_connections=8 /opt/apigee/edge-management-server/conf/cps-runtime.properties:l2cache.maxactive_cassandra_connections=8 /opt/apigee/edge-management-server/conf/cps-runtime.properties:timeseries.maxactive_cassandra_connections=8 /opt/apigee/edge-management-server/conf/cps-runtime.properties:quota.maxactive_cassandra_connections=8 /opt/apigee/edge-management-server/conf/audit.properties:maxactive_cassandra_connections=10 /opt/apigee/edge-management-server/conf/developer-datastore.properties:maxactive_cassandra_connections=10 /opt/apigee/edge-management-server/conf/counter.properties:maxactive_cassandra_connections=10
We had ran the export all proxies in a 15.07 without problems in the past, was any of those configuration changed with the recent versions?
@Maruti Chand @Baba Krishnankutty
Any help would be greatly appreciated.
Hi @Matias,
management servers are not intended for the load; you may want to wait for few sec-min (sleep) before each proxy download.
IIRC you should modify the app repo keyspace related property file to increase the maxactive_cassandra_connections if you want to see raising that fixes your issue for now.
Thanks for the reply @Maruti Chand.
> management servers are not intended for the load
Good to know. It's unfortunate that the script from that repo doesn't follow those guidelines.
I ran a test with more connections in the repository.properties though it didn't work.
Does Apigee has any other public repo to migrate proxies from one cluster to another one?