Load Balancing across Dynamic Target Servers

kalyaninturi
Participant II

Hello,

I have a use case where I have to build Target URL based on the configuration details I pull from KVM.

Until now things went well, I was reading configuration from KVM, building target URL and assigning

Target URL to 'target.url' flow variable inside Target Req Flow.

Now I got a requirement where I should set two different Target URLs inside JavaScript

and should be able to load balance across them. I don't find any documentation to implement this.

I appreciate any kind of help.

Thanks

Kalyan

Solved Solved
1 6 761
1 ACCEPTED SOLUTION

Sure.

This is a pretty straightforward thing.

Basically you want to read from the KVM (or some other resource) and then set the target.url to one or the other server endpoint. But let us not confine ourselves to the special case in which there are exactly 2 elements in the list. Suppose the list of servers has N endpoints. The goal of the logic should be to select 1 of those elements randomly, and set target.url to the value of the selected endpoint.

This is pretty easy in JavaScript. You only need to use a random selection on an array.

This function in JavaScript selects a random value from any array 'a' :

  function selectRandomValue (a) {
    var L1 = a.length,
        n = Math.floor(Math.random() * L1);
    return a[n];
  }

Suppose my list is ["foo", "bar", "bam"]. That function will randomly return one of those items from the list, each time it is called.

This is the building block you need to produce your simple load balancer. The other complementary pieces are: getting the list from the KVM, and setting the target.url. You already know how to do this, so I won't include any of that here. I guess you know how to assemble them together.


But now that we have solved the simple case, I suppose you will want more. For example you may want weighted random selection. Each item in the list might have a "weight" which is relative to the others. For example you may want the first item to be selected 50% of the time, the second element to be selected 40% of the time, and the third item 10% of the time. (This is handy when connecting Apigee to upstreams which are undergoing blue/green deployments. Maybe you want Apigee to direct only 2% of its load to the new system, while you verify its behavior.)

To do that you need a weighted random selector. I have one of those for you. [here]

To use it, you would do something like this:

var serverlist = JSON.parse(context.getVariable('servers_retrieved_from_kvm'));
// this list ought to look like this:
//      serverip      weight
// [
//    ["1.2.3.4",      50],
//    ["1.2.3.5",      10],
//    ["1.2.3.7",      10],
//    ["10.20.30.3",    1],
//    ["10.20.30.4",    1],
//    ["10.20.30.5",    1]
//  ]

// Note: the weights are not percentages; they don't need to 
// total to 100.  The percentage for each item is calculated as
// weight(item)/sum-all(weight(item))

var wrs = new WeightedRandomSelector(serverlist);
var selected = wrs.select();
var selectedServerIP = selected[0];
context.setVariable('target.url', selectedServerIP);

The next thing you may want to do is "health monitoring". You have a list of N servers, possibly weighted, and you want to load balance across all the nodes in the list, except those nodes that have been marked unhealthy.

That's something that Apigee does natively with TargetServers. You should not try to implement health monitoring in your API proxies at that layer. It's better handled in the runtime layer.

I hope this helps.

View solution in original post

6 REPLIES 6

Sure.

This is a pretty straightforward thing.

Basically you want to read from the KVM (or some other resource) and then set the target.url to one or the other server endpoint. But let us not confine ourselves to the special case in which there are exactly 2 elements in the list. Suppose the list of servers has N endpoints. The goal of the logic should be to select 1 of those elements randomly, and set target.url to the value of the selected endpoint.

This is pretty easy in JavaScript. You only need to use a random selection on an array.

This function in JavaScript selects a random value from any array 'a' :

  function selectRandomValue (a) {
    var L1 = a.length,
        n = Math.floor(Math.random() * L1);
    return a[n];
  }

Suppose my list is ["foo", "bar", "bam"]. That function will randomly return one of those items from the list, each time it is called.

This is the building block you need to produce your simple load balancer. The other complementary pieces are: getting the list from the KVM, and setting the target.url. You already know how to do this, so I won't include any of that here. I guess you know how to assemble them together.


But now that we have solved the simple case, I suppose you will want more. For example you may want weighted random selection. Each item in the list might have a "weight" which is relative to the others. For example you may want the first item to be selected 50% of the time, the second element to be selected 40% of the time, and the third item 10% of the time. (This is handy when connecting Apigee to upstreams which are undergoing blue/green deployments. Maybe you want Apigee to direct only 2% of its load to the new system, while you verify its behavior.)

To do that you need a weighted random selector. I have one of those for you. [here]

To use it, you would do something like this:

var serverlist = JSON.parse(context.getVariable('servers_retrieved_from_kvm'));
// this list ought to look like this:
//      serverip      weight
// [
//    ["1.2.3.4",      50],
//    ["1.2.3.5",      10],
//    ["1.2.3.7",      10],
//    ["10.20.30.3",    1],
//    ["10.20.30.4",    1],
//    ["10.20.30.5",    1]
//  ]

// Note: the weights are not percentages; they don't need to 
// total to 100.  The percentage for each item is calculated as
// weight(item)/sum-all(weight(item))

var wrs = new WeightedRandomSelector(serverlist);
var selected = wrs.select();
var selectedServerIP = selected[0];
context.setVariable('target.url', selectedServerIP);

The next thing you may want to do is "health monitoring". You have a list of N servers, possibly weighted, and you want to load balance across all the nodes in the list, except those nodes that have been marked unhealthy.

That's something that Apigee does natively with TargetServers. You should not try to implement health monitoring in your API proxies at that layer. It's better handled in the runtime layer.

I hope this helps.

@Dino,

Thank you for the quick response.

I am looking for the things you mentioned in the last two paragraphs. I knew how to select a node from a list of nodes using a Weighted Random Selector. But I am looking at how to load balance across all nodes in the list except those nodes that have been marked unhealthy. Looks like I cannot implement that in apigee.

Thank you

Kalyan Inturi

yes, Apigee already does that for you with the TargetServer.

You *could* try doing that in the API PRoxy layer but it would probably be the wrong thing to do.

@kalyan inturi

Did you get a solution for how to load balance across all nodes in the list except those nodes that have been marked unhealth?

@Dino,

Since we are setting up target url from script then how apigee target servers comes into the picture??

@Dino-at-Google

Could you please elaborate the weighted formula ? I tried different use cases but result are varying

Formula -  
// weight(item)/sum-all(weight(item))

Example-1 - Expected result, each letter 1 time and repeat

   
  var letterList = [
  // //  name      weight
     ["A",   1],
     ["B",   1],
     ["C",   1],
     ["D",   1],
     ["E",   1],
     ["F",   1],
     ["G",   1]
   ];
  

Example-2 Expected result , region-1, 9 times and region-2 would be called 1 time in total 10 requests

 var regionList = [
   //  name      weight
     ["region-1",    90],
     ["region-2",    10]
     ];
   

Not applicable

Example 1: all are having weight 1. So, in every 7 requests, each will get one request. then it will repeat.

formula : for A 1/(1+1+1+1+1+1+1) = 1/7 means in every 7 request 1 will go to A.

similar for B, C ..

Example 2: In every 100 requests 90 will go to region-1 and 10 will go to region-2

formula: region-1 90/(90+10) = 90/100 so in every 100 requests 90 will go to region 1

region-2 10/(90+10) = 10/100 so in every 100 requests 10 will go to region 2