3.5. How to update SEER 1.6 Agents to MAGI

Introduction

This document describes how to convert an older SEER agent to the new MAGI framework.

The table below shows the differences that exist between the old and new framework. Each of the entries need to be addressed when converting the agent. This may seem like a lot, but many of the changes will be similar, thus once the gist is got, agents can be easily converted. Each entry is described in much more detail in the “Agents” section of this document, but for now this gives you an idea of the updates. Each entry will be briefly described after the table. This will be followed by an example, demonstrating how the updates are actually done.

Differences at a glance

Difference	Old	New
How described	In a single Python file	In a directory with multiple files
End user interface	GUI infered from well-known list variable, VARIABLES	Separate interface description language (IDL) file, well, describes the interface
Memory model	Single daemon process	Agents can be separate processes or a thread within the daemon process
Language	Python/Java	Supports any language via YAML RPC and separate IDL file
Messaging system	???	Dynamic, runtime, group-oriented messaging system
Specifying dependencies	DEPENDS and SOFTWARE variables within the script	System API call to requireSoftware()
Specifying functionality	COMMANDS variable within the script	Automatic and dynamic dispatch via YAML “RPC”
Logging interface	???	???
Testing	None	Built-in unit tests supported

GTL - briefly describe differences here, possibly in list or outline format.

GTL - state the defailt structure of the new agents: in directory, [name].py, __init__.py, [name].idl and anything else.

Now we will go through the generic steps for converting the agent. This will be followed by a concrete example. Note that in the examples, anything in square brackets ([]), is to be filled in with the appropriate but unspecified content. i.e. [agentname] would be replaced with MyCoolAgent or whatever your agent is called.

The first thing to do is to create a new directory for your agent. In the old system, agents were single files, in the new system a single agent is a collection of files in a directory. If your agent is written in python you might as well make an empty __init__.py file as well.

mkdir [montage repo dir]/backend/magi/modules/[agentname]
touch [montage repo dir]/backend/magi/modules/__init__.py

Now create, if needed, the IDL file for your agent. In MAGI, the interface is external to the agent’s code and is kept in an IDL file. For details of the IDL format, see the appropriate section in this documentation. At a minimum, add the following: name, display, description, execute, mainfile, variables, and software (if relevant).

cat > [montage repo dir]/backend/magi/modules/[agentname].idl << EOF
name: [agentname]
display: [agent's name displayed in GUI]
description: [pithy description of agent]
execute: [thread|pipe|socket]
mainfile: [name of agent's main file, probably [agentname].py]
variables:
    - name: [var_one]
      help: [pithy and useful description of var_one]
    - name: [var_two]
      help: [pithy and useful description of var_two]
software:
    - [software that agent depends on]
EOF

Now create the main agent file. This example uses python, other languages are currently out of scope as I’ve not actaully tried anything other than python. The general format of the file is: import magi specific functionality, define the function that is called by the framework and returns an instance of your agent, then the actual definition of the agent as a class. A generic example could look like this:

cat > [montage repo dir]/backend/magi/modules/[agentname].py << EOF
import logging
from magi.util.software import requireSoftware

log = logging.getLogger(__name__)

def getAgent():
    return MyCoolAgent()

class MyCoolAgent([base class]):
    """ [class description] """
    def __init__(self):
        [base class initialization]
        # initalize IDL variables
        self.var_one="hello"
        self.var_twp="world"
        requireSoftware([agent's software dependcies])

    [class specific methods, possibly overridden from base class]

Example

This section will walk you through the updates done to convert a specific agent to the MAGI framework. The agent in the example is the HTTP agent, which generates HTTP traffic by controling a set of wget clients that make HTTP requests to a set of servers running apache. It makes use of, and is dependent on, the wget and Apache packages.

The old agent, single source file:

# Copyright (C) 2009 SPARTA, Inc.
# This software is licensed under the GPLv3 license, included in
# ./GPLv3-LICENSE.txt in the source distribution

from util.platform import spawn
from backend.agent import Agent
from backend.variables import *
from backend.addon import services

class HTTPAgent(Agent):
	"""
	The HTTP generator controls a set of wget clients that make HTTP requests to a set of servers running apache
	"""
	
	DEPENDS = ['ApacheService']
	SOFTWARE = ['wget']

	AGENTGROUP = 'Traffic'
	AGENTTYPE = 'HTTP'
	NICENAME = 'Web'
	COMMANDS = ['START','STOP']
	VARIABLES = [
		Title('Web Settings'),
		NodeListVar('clients', None, 'Clients', 'Select the nodes that will become HTTP agents'),
		NodeListVar('servers', None, 'Servers', 'Select the nodes that will become HTTP servers'),
		DistVar('think', 1, 'Thinking Time', 'Function to determine time between requests'),
		DistVar('sizes', 1, 'File Sizes', 'Function to determine the size of the page requested'),
		BoolVar('usessl', False, 'Use HTTPS', 'Try to use HTTPS, requires that the server node was able to build apache with SSL')
		]

	def __init__(self): Agent.__init__(self)
	handleSTART = Agent.TGStart
	handleSTOP = Agent.TGStop
	def serverExec(self): services.ApacheService.start()
	def serverStop(self): services.ApacheService.stop()

	def clientExec(self, src, dst, size):
		cmd = "wget -nv -t 1 -O /dev/null "
		if src is not None:
			cmd += "--bind-address %s " % (src)
		if self.usessl:
			cmd += "--no-check-certificate "
		cmd += "http%s://%s/getsize.py?length=%d " % (self.usessl and 's' or '', dst, size)

		# We print this on stdout so it goes with redirected stdout
		print cmd
		subpid = spawn(cmd, None)

The new agent, script file:

# Copyright (C) 2011 SPARTA, Inc.
# This software is licensed under the GPLv3 license, included in
# ./GPLv3-LICENSE.txt in the source distribution

from magi.util.agent import TrafficClientAgent, agentmethod
from magi.util.distributions import *
from magi.util.software import requireSoftware

def getAgent():
	return WgetAgent()

class WgetAgent(TrafficClientAgent):
	"""
		The wget http generator controls a set of wget clients that make HTTP requests to a set HTTP servers.
		Most of the logic is based in TrafficClientAgent
	"""
	def __init__(self):
		TrafficClientAgent.__init__(self)
		requireSoftware('wget')
		self.url = "http://%s/getsize.py?length=%d"
		self.sizes = '1'

	def getCmd(self, dst):
		return "wget -nv -t 1 -O /dev/null " + self.url % (dst, eval(self.sizes))

The new agent interface description file:

name: wgethttpclient
display: Wget HTTP Client
description: a simple HTTP client agent using wget to perform requests
execute: thread
mainfile: http_wget.py
inherits:
   - trafficclient
variables:
   - name: sizes
     help: the sizes of data to request via the URL, default is random less than 1M
   - name: url
     help: URL template to use for requsting data, expects a %s for host and then %d for size
software:
   - wget

Example Screenshots

Screenshot of the HTTP agent in MAGI 1.67

Note: this is not the actual screenshot of the new agent. This is a bunny. Actual screenshot TBD.