.. _magilearn: MAGI Tutorial =============== This section cover the **basics** of creating MAGI enabled experiments. The primitives and abstractions discussed in this section are demonstrated with examples in :ref:`magicases` Introduction ^^^^^^^^^^^^^^^ MAGI is a workflow management system that provides deterministic control over the various components in an experiment. It is used to express and automate the procedure of an experiment. The procedure is expressed is a YAML-based Agent Activation Language. The MAGI **orchestrator** tool parses the procedure and maintains experiment wide state to execute the procedure. The **orchestrator** sends and receives **events** from **agents** on the experiment nodes, and enforces synchronization points called as **triggers** to deterministically execute the procedure. The experiment **procedure** is expressed as streams of events. An **event** is signal to invoke a particular behavior in an agent. An **agent** is a functional behavior that is mapped onto one or more experiment nodes, collectively called a **group**. The agent implementation, referenced with the directive **code** or **module**, is the piece of code that is activated when events are sent to the agent. Agent Activation Language ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Agent Activation Language (AAL) is a YAML-based descriptive language that to describe an experiment’s workflow. An AAL specification has mainly three parts: 1. Groups The **group** directive enables a coupling between the experiment procedure and the experiment nodes. MAGI groups are used to define sets of one or more experiment nodes that typically have the same behavior. The groups are referred to within the agent descriptions as discussed below. An example of a group declaration is as follows: .. code-block:: yaml groups: clients: [ node1, node2, node7 ] defender: [ router3 ] 2. Agents The MAGI agent directive maps a functional behavior on a set of experiment nodes. An agent provides a set of behaviors that can be invoked with events as discussed below. The agent directive requires the following keys: - agent: name for the set of nodes that represent the behavior - group: a set of experiment nodes the will functional as the agent - path: the path to the agent implementation code - code: directory name of the agent implementation. The code directive is used in the absence of a path directive. MAGI requires the agent implementation to be part of the python distribution if the path directive is not specified. - execargs: zero or more arguments that will be passed to the agent during initialization. An example of an agent declaration is as follows: .. code-block:: yaml agents: smallwebclient: group: smallclients path: /share/magi/modules/http_client/http_client.tar.gz execargs: { servers: [ servernode ], interval: '2', sizes: 'minmax(300,500)'} 3. Event Streams An event stream is a list of **events** and **triggers** that are parsed and executed by the orchestrator. The procedure typically contains multiple event streams. Different event streams execute concurrently and are synchronized with each other using **triggers**. The set of event streams listed using the **streamstarts** directive are invoked at the start of procedure. However, note that the the orchestrator will perform several setup actions, such as create groups, load agents, get status, before the event streams start. An event stream consists of events and triggers. 3.1 Events An event signals an agent method. It invokes a procedure implemented in an agent. An event is sent to a group of experiment nodes. An event directive requires the following keys: - agent: the agent to send the event - method: the method to be invoked - args: zero or more arguments required by the method Additionally, it can also contain the trigger key to flag the return status from the method. The return status can be either True or False. An example of the event declaration is as follows: .. code-block:: yaml - type: event agent: server_agent method: startServer trigger: serverStartedSuccessfully args: {} 3.2 Triggers triggers are used as a synchronization mechanism, guard points, or rendezvous points in an experiment procedure. Triggers are of two types that can be combined is several different ways: - Event-based triggers that are received from agents after a method. The **serverStartedSuccessfully** is an example of an event-based trigger. The MAGI orchestrator keeps track of outstanding triggers to follow the experiment execution state space. When the server_agent returns True after the method *startServer* the orchestrator tags it as a received trigger. An example of using the event-based triggers is as follows: .. code-block:: yaml - type: trigger triggers: [ {event: ClientStopped} ] - Time-based triggers wait for a specified amount time to elapse at the orchestrator before proceeding. An example of using the time-based triggers is as follows: .. code-block:: yaml - type: trigger triggers: { [ timeout: 60000 ] } # wait for 60 seconds The :ref:`magicases` discussed several specific examples of declaring groups, agents, events, and triggers. MAGI Orchestrator ^^^^^^^^^^^^^^^^^^^ The MAGI orchestrator is a tool that parses an AAL file and orchestrates an experiment based on the specified procedures. 1. Join Groups The orchestrator iterates over the list of groups and for each group sends a request to all the mapped nodes to join the group. A corresponding reply adds the replier to the group. Messages addressed to a group are sent to all the nodes that are part of the group. 2. Load Agents The orchestrator iterates over the list of agents and for each agent sends an agent load request to the mapped groups. An agent load message specifies to start an instance of the agent implementation and to put it in a listening mode, where the instance waits for further messages. 3. Execute Event Streams Next, the orchestrator concurrently executes all the event streams listed as part of streamstarts. The MAGI orchestrator has a predefined event stream called exit. The purpose of this event stream is to unload all the agents and disjoin groups. All workflows should end up executing this stream for a clean exit. The magi_orchestrator.py tools is used to enact the specified AAL. The various command line options are as follows .. code-block:: bash Usage: magi_orchestrator.py [options] Options: -h, --help show this help message and exit -c CONTROL, --control=CONTROL The control node to connect to (i.e. control.exp.proj) -f EVENTS, --events=EVENTS The events.aal file(s) to use. Can be specified multiple times for multiple AAL files -l LOGLEVEL, --loglevel=LOGLEVEL The level at which to log. Must be one of none, debug, info, warning, error, or critical. Default is info. -o LOGFILE, --logfile=LOGFILE If given, log to the file instead of the console (stdout). -e EXITONFAILURE, --exitOnFailure=EXITONFAILURE If any method call fails (returns False), then exit all streams, unload all agents, and exit the orchestrator. Default value is True -g GROUPBUILDTIMEOUT, --groupBuildTimeout=GROUPBUILDTIMEOUT When building the initial groups for agents in the given AAL, use the timeout given (in milliseconds) when waiting for group formation to complete. --nocolor If given, do not use color in output. -v, --verbose Tell orchestrator to print info about what its doing -n, --tunnel Tell orchestrator to tunnel data through Deter Ops (users.deterlab.net).