TA AgentList with Amazon Web Service Autoscaling

Amazon Web Service (AWS) Autoscaling monitors your applications and automatically adjusts the capacity to maintain steady, predictable performance at the lowest possible cost. Using AWS Autoscaling, you can setup application scaling for multiple resources across multiple services in less time.

When a new agent machine is launched in the Amazon Web Service (AWS) with Autoscaling group enabled, the agent is created and added to agent lists in TA. Similarly, when the existing AWS agent machine is removed or terminated from the AWS Autoscaling group, the agent gets deleted and removed from the agent lists.

This process is done automatically using these AWS components:

  • Autoscaling Group

  • AWS Simple Notification Service

  • AWS Lambda function

This block diagram explains the process involved in registering and unregistering of the agent to or from the Agent List/Master:

Creating an Autoscaling Group

Autoscaling helps to ensure that the correct number of instances are available to handle the load for the application. The collections of instances are created and called the Autoscaling groups. The maximum or minimum number of instances are specified in each group. The Autoscaling ensures that the group never goes above or below the values set.

Note: For more information on creating the Autoscaling Group, see Create an Auto Scaling group using a launch template

Getting AWS Simple Notification Service Messages

A topic is a communication channel to send messages and subscribe to notifications. A topic name is used to create a permanent unique identifier called an Amazon Resource Name (ARN).

To create a new topic:

  1. Click Create topic on the Amazon SNS console. The Create topic dialog appears.

  2. Enter the topic name in the Topic name field.

  1. Enter the topic display name in the Display name field. This display name is required for topics with SMS subscriptions.

  1. Click Create topic. The new topic appears.

To receive messages published to a topic, you have to subscribe an endpoint to that topic. An endpoint is a mobile app, web server, email address, or an Amazon SQS queue that can receive notification messages from Amazon SNS. Once you subscribe an endpoint to a topic and the subscription is confirmed, the endpoint will receive all messages published to that topic. You configure the subscription to send the topic messages to your email account.

To create subscription:

  1. Click Create subscription in the Amazon SNS console. The Create Subscription dialog appears.

  2. Enter the topic ARN.

  1. Choose AWS Lambda from the Protocol drop-down list.

  1. Choose an email address you can use to receive the notification from the Endpoint drop-down list.

  1. Click Create subscription.

Go to your email application and open the message from AWS Notifications, and click the link to confirm your subscription. Your web browser displays a confirmation response from Amazon SNS.

For more information on creating the topic and subscribe topic, see Creating an Amazon SNS topic and What is Amazon SNS.

Creating a Lambda Function

A Lambda function is created to handle the launching of an instance or termination of an instance. For more information on creating the Lambda function, see Getting started with Lambda.

Lambda Handler

When the instance is up or down, the Lambda Handler receives the AWS SNS messages during launch or termination of the instance.

The Lambda handler reads the SNS notification message in the Lambda code, fetches the instance details, and calls the TA API call based on the instance notification type.

If the notification type is a “New Instance”, it checks whether the agent exists.

  • If the prefix of the agent name matches with the existing agent name and the agent is disabled, the latest data is updated to that existing disabled agent while creating the new instance in AWS.

  • If the IP of the agent matches with the existing agent and the agent is disabled, the latest data is updated to that existing disabled agent while creating the new instance in AWS.

If the notification type is “Terminate Instance”, it checks whether the agent references other objects like a job. If there is a reference with other objects, the agent is disabled instead of deleting the agent while removing the instance from the AWS account.

Setting up of AWS Environment

Prerequisites:

Before starting to use the AWS Autoscaling, complete these tasks.

  1. Create a new Amazon Machine Image (AMI) with Unix/Window TA agent.

    For more information on creating AMI, see Create an Amazon EBS-backed Linux AMI. If you already have the AMI with the TA agent, move to the next step.

  2. Create a Lambda function to handle the launching or termination of instance and upload the AgentListAutomation.jar file.

    • Choose this Lambda handler in the Handler text box: “com.tidal.automation.AgentListLambdaFunctionHandler”.

      Note: Memory must be more than 512 MB.

    • Choose the Timeout value based on Lambda running time, suggested value is 5 minutes.

    • Choose proper Lambda Role.

  1. Create AWS SNS notifications and subscribe with the Lambda function created in Step 2.

  1. Create Autoscaling Launch Configuration. For choosing the required configuration with the corresponding AMI, choose the IP Address Type as Assign a public IP address for every instance.

  1. Create the Autoscaling group and map it with the created SNS. Note that the Virtual Private Cloud (VPC) ID must be the same as the CM VPC.

Client Manager Properties

The table provides the list of Client Manager (CM) properties in the AgentListAutomation.jar file.

Property Name

Description

cm_servername

IP address of the Client Manager. This a mandatory field.

cm_dsppluginname

(Optional). tes-6.5 is the default value. If you need to change this value, use this property.

cm_serverport

(Optional). 8080 is the default value. If you need to change this value, use this property.

cm_username

User name of the Client Manager. If you need to change this value, use this property.

This a mandatory field.

cm_password

Password of the Client Manager. If you need to change this value, use this property. This a mandatory field.

agentlistid

ID of the TA agent list. Multiple IDs can be given with comma separated. If you need to change this value, use this property. This a mandatory field.

ostype

TA OS type. If you need to change this value, use this property. This a mandatory field.

runtimeuserid

(Optional). TA runtime user ID. Zero is the default value. If you need to change this value, use this property.

maxjoblimit

(Optional) Ten is the default value. If you need to change this value, use this property.

port

(Optional) 5912 is the default value. If you need to change this value, use this property.

active

(Optional) Y is the default value. If you need to change this value, use this property.

nameprefix

(Optional) “AWS_AGENT” is the default value. If you need to change this value, use this property.

description

(Optional) “Agent created by AWS lambda with AWS Autoscaling group” is the default value. If you need to change this value, use this property.

iptype

IP address type. The default value is public. If the value is ‘public’, the agent machine IP is the public IP address.

If the value is ‘private’, the agent machine IP is the private IP address (that is, within the subnet IP range).

AWS Properties

The table provides the list of AWS properties.

Property Name

Description

Uselambdavariable

(Optional) Y is the default value. If you need to change this value, use this property.

If the value is Y, all the property is loaded from automation.props file and also any property can be overridden through the Lambda environment variable. Note that the CM password must be encrypted by Key Management Service (KMS) key, and other property should not be encrypted.

If the value is N, all the property is loaded from automation.props file. The AWS Lambda environment variable option is not available for this condition

accesskey

AWS access key.

If you need to change this value, use this property. This a mandatory field

secretkey

AWS secret key.

If you need to change this value, use this property. This a mandatory field

Note: For security reasons, use read only user access key and secret key.

To create the user access key in the AWS management console:

  1. Click Users on the navigation pane of the Identity Access Management (IAM) console. The list of users having read only access permission is displayed. If you want to add a new user, choose Add user.

  2. Click the particular user.

  1. Click the Security Credentials tab and click Create access key. The user access key is created.

    For more information on access key creation, see Managing access keys for IAM users.

    For security reasons, add KMS key decrypt inline policy for read only user.

    The sample policy JSON file:

    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Sid": "VisualEditor0", "Effect": "Allow",
    "Action": "kms:Decrypt",
    "Resource": "arn:aws:kms:<region>:<AWS AccountId>:key/<KMS key Id>"
    }
    ]
    }