This section cover the basics of creating MAGI enabled experiments. The primitives and abstractions discussed in this section are demonstrated with examples in Case Studies
MAGI is a workflow management system that provides deterministic control over the various components in an experiment. It is used to express and automate the procedure of an experiment.
The procedure is expressed is a YAML-based Agent Activation Language. The MAGI orchestrator tool parses the procedure and maintains experiment wide state to execute the procedure. The orchestrator sends and receives events from agents on the experiment nodes, and enforces synchronization points called as triggers to deterministically execute the procedure.
The experiment procedure is expressed as streams of events. An event is signal to invoke a particular behavior in an agent. An agent is a functional behavior that is mapped onto one or more experiment nodes, collectively called a group. The agent implementation, referenced with the directive code, is the piece of code that is activated when events are sent to the agent.
The data management layer helps is collecting and querying data sensed by the agents.
Agent Activation Language (AAL) is a YAML-based descriptive language that to describe an experiment’s workflow. An AAL specification has mainly three parts:
The group directive enables a coupling between the experiment procedure and the experiment nodes. MAGI groups are used to define sets of one or more experiment nodes that typically have the same behavior. The groups are referred to within the agent descriptions as discussed below.
An example of a group declaration is as follows:
groups:
clients: [ node1, node2, node7 ]
defender: [ router3 ]
The MAGI agent directive maps a functional behavior on a set of experiment nodes. An agent provides a set of behaviors that can be invoked with events as discussed below. The agent directive requires the following keys:
- agent: name for the set of nodes that represent the behavior
- group: a set of experiment nodes the will functional as the agent
- path: the path to the agent implementation code
- code: directory name of the agent implementation. The code directive is used in the absence of a path directive. MAGI requires the agent implementation to be part of the python distribution if the path directive is not specified.
- execargs: zero or more arguments that will be passed to the agent during initialization.
An example of an agent declaration is as follows:
agents: smallwebclient: group: smallclients path: /share/magi/modules/http_client/http_client.tar.gz execargs: { servers: [ servernode ], interval: '2', sizes: 'minmax(300,500)' }
An event stream is a list of events and triggers that are parsed and executed by the orchestrator. The procedure typically contains multiple event streams. Different event streams execute concurrently and are synchronized with each other using triggers.
The set of event streams listed using the streamstarts directive are invoked at the start of procedure. However, note that the the orchestrator will perform several setup actions, such as create groups, load agents, get status, before the event streams start.
An event stream consists of events and triggers.
An event signals an agent method. It invokes a procedure implemented in an agent. An event is sent to a group of experiment nodes. An event directive requires the following keys:
- agent: the agent to send the event
- method: the method to be invoked
- args: zero or more arguments required by the method
Additionally, it can also contain the trigger key to flag the return status from the method. The return status can be either True or False.
An example of the event declaration is as follows:
- type: event
agent: server_agent
method: startServer
trigger: serverStartedSuccessfully
args: {}
Triggers are used as a synchronization mechanism, guard points, or rendezvous points in an experiment procedure. Triggers are of two types that can be combined is several different ways:
Event-based triggers are those that are received from agents after a method. These triggers are used to notify the completion of the called method and to receive the return value. The MAGI orchestrator keeps track of outstanding triggers to follow the experiment execution state space. If a trigger is received with an excepted return value, it is considered to be successfully complete, and the execution moves ahead. The default expected return value is True. Also, by default, on completion of a trigger, the procedure execution moves to the next line in the event stream. However, triggers can also be used to jump to the start of an event stream. This combined with a non-singleton set of expected return values can be used for conditional branching.
The serverStartedSuccessfully is an example of an event-based trigger. When the server_agent returns True after the method startServer the orchestrator tags it as a received trigger, and the execution moves to the next line in the event stream.
- type: trigger triggers: [ { event: serverStartedSuccessfully } ]An example of using the event-based triggers for conditional branching, from the Case Study: Feedback, is as follows:
- type: trigger triggers: [ { event: intfSensed, result: -1, target: 'increaseTraffic' }, { event: intfSensed, result: 1, target: 'reduceTraffic' }, { event: intfSensed, result: 0, target: 'controlLoop'}]Note
Triggers, as of now, support only an exact match of the return value.
Time-based triggers wait for a specified amount time to elapse at the orchestrator before proceeding.
An example of using the time-based triggers is as follows:
- type: trigger triggers: { [ timeout: 60000 ] } # wait for 60 seconds
The Case Studies discuss several specific examples of declaring groups, agents, events, and triggers.
The MAGI orchestrator is a tool that parses an AAL file and orchestrates an experiment based on the specified procedures.
The orchestrator iterates over the list of groups and for each group sends a request to all the mapped nodes to join the group. A corresponding reply adds the replier to the group. Messages addressed to a group are sent to all the nodes that are part of the group.
The orchestrator iterates over the list of agents and for each agent sends an agent load request to the mapped groups. An agent load message specifies to start an instance of the agent implementation and to put it in a listening mode, where the instance waits for further messages.
Next, the orchestrator concurrently executes all the event streams listed as part of streamstarts.
The MAGI orchestrator has a predefined event stream called initialization. It is this stream that creates the groups and loads the agents. Once the initialization stream is complete, the event streams from the AAL are executed in parallel. The MAGI orchestrator has another predefined event stream called exit. The purpose of this event stream is to unload all the agents and disjoin groups. All workflows should end up executing this stream for a clean exit.
The orchestrator requires two things to run, an instatated experiment and an AAL file. When you run the orchestrator, you pass it the project and experiment name, and the AAL file containing events to orchestrate. It connects to the experiment’s MAGI messaging system and orchestrates the experiment. It sends events and receives triggers via the MAGI messaging system.
The magi_orchestrator.py tool is used to enact the specified AAL. The command below runs the orchestrator as described, writing the output log to orch.log.
> magi_orchestrator.py --project myProject --experiment myExperiment --events myEvents.aal --logfile orch.log
Note
The experiment nodes on the DETER testbed are accessible only from the DETER Ops machine (users.isi.deterlab.net). Hence the orchestrator needs to be run from the DETER Ops machine.
> ssh users.isi.deterlab.net > users $ magi_orchestrator.py --project myProject --experiment myExperiment --events myEvents.aal --logfile orch.log
The orchestrator output generally has a single line for each event or trigger in a stream. The line shows the event stream name, event status, then event specific information. In the case of an event, this is the method call, method arguments, and on whom the method is invoked. For triggers, this is the trigger status. In the event of an error, like a method returning False, or an exception thrown by an agent, the orchestrator will display the error, unload agents, break down communication groups it has built, then exit. The output line will give information about the error.
For example, following is the orchestrator output for the Case Study: Client-Server.
The various command line options for magi_orchestrator.py are as follows
Usage: magi_orchestrator.py [options]
Options:
-h, --help show this help message and exit
-p PROJECT, --project=PROJECT
Project name
-e EXPERIMENT, --experiment=EXPERIMENT
Experiment name
-b BRIDGE, --bridge=BRIDGE
Address of the bridge node to join the messaging
overlay (ex: control.exp.proj)
-c BRIDGE, --control=BRIDGE
Address of the bridge node to join the messaging
overlay (ex: control.exp.proj). This option exists for
backward compatibility.
-r PORT, --port=PORT The port to connect to on the bridge node.
-f EVENTS, --events=EVENTS
The procedure.aal file(s) to use. Can be specified
multiple times for multiple AAL files
-o LOGFILE, --logfile=LOGFILE
If given, log to the file instead of the console
(stdout).
-l LOGLEVEL, --loglevel=LOGLEVEL
The level at which to log. Must be one of none, debug,
info, warning, error, or critical. Default is info.
-n NAME, --name=NAME Name using which to connect to the messaging plane.
Default: pyorch
-x EXITONFAILURE, --exitOnFailure=EXITONFAILURE
If any method call fails (returns False), then exit
all streams, unload all agents, and exit the
orchestrator. Default value is True
-g GROUPBUILDTIMEOUT, --groupBuildTimeout=GROUPBUILDTIMEOUT
When building the initial groups for agents in the
given AAL, use the timeout given (in milliseconds)
when waiting for group formation to complete.
--nocolor If given, do not use color in output.
-v, --verbose Tell orchestrator to print info about what its doing
-t, --tunnel Tell orchestrator to tunnel data through Deter Ops
(users.deterlab.net). Must specify the bridge node.
-u USERNAME, --username=USERNAME
Username for creating tunnel. Required only if
different from current shell username.
-d, --display Display the procedure execution graphically
-j, --justparse Parse and display the procedure file specified with -f
MAGI’s data management layer helps agents collect data. Users can query for the collected data by connecting to the data management layer.
The Data Management section has more information about MAGI’s data management layer.