MARL Plots#

Multi-Agent Reinforcement Learning (MARL) Plotting Module#

This module provides visualization tools for analyzing the performance of multi-agent reinforcement learning (MARL) algorithms. It includes functions for plotting policies, rewards, and positions of agents over time.

Dependencies:#

  • InflGame.utils

  • InflGame.MARL

Usage:#

The policy_histogram function visualizes the Q-table as a policy heatmap, while the reward_plot and pos_plot functions plot the rewards and positions of agents over time, respectively. The policy_deterministically_to_actions function simulates deterministic actions for agents based on their policies.

Examples#

import numpy as np
import torch
from InflGame.MARL.async_game import influencer_env_async
from InflGame.MARL.MARL_plots import policy_histogram, reward_plot, pos_plot, policy_deterministically_to_actions

# Define environment configuration
env_config = {
    "num_agents": 3,
    "initial_position": [0.2, 0.5, 0.8],
    "bin_points": np.linspace(0, 1, 100),
    "resource_distribution": np.random.rand(100),
    "step_size": 0.01,
    "domain_type": "1d",
    "domain_bounds": [0, 1],
    "infl_configs": {"infl_type": "gaussian"},
    "parameters": [0.1, 0.1, 0.1],
    "fixed_pa": 0,
    "NUM_ITERS": 100
}

# Initialize the environment
env = influencer_env_async(config=env_config)

# Simulate deterministic actions
q_tensor = torch.rand((3, 100, 3))  # Example Q-tensor
pos_matrix, reward_matrix = policy_deterministically_to_actions(env=env, q_tensor=q_tensor, num_step=50)

# Plot policy heatmap for player 0
policy_fig = policy_histogram(q_tensor=q_tensor, player_id=0)
policy_fig.show()

# Plot rewards over time
reward_fig = reward_plot(reward_matrix=reward_matrix, possible_agents=env.possible_agents)
reward_fig.show()

# Plot positions over time
pos_fig = pos_plot(pos_matrix=pos_matrix, possible_agents=env.possible_agents, domain_bounds=env_config["domain_bounds"])
pos_fig.show()

Functions

InflGame.MARL.MARL_plots.agent_position_trajectory(pos_data, window_size=10000, num_agents=None, title_ads=[], name_ads=[], short_title=False, save=False, save_types=['.png', '.svg'], font={'cbar_size': 12, 'default_size': 12, 'font_family': 'sans-serif', 'legend_size': 12, 'title_size': 14}, paper_figure={'figure_id': 'agent_trajectory', 'paper': False, 'section': 'A'}, figsize=(10, 6), num_ticks=6, axis_return=False)#

Plots agent positions over time with scientific notation x-axis labels.

This function visualizes the trajectory of agent positions throughout training, with x-axis labels showing the actual episode count (accounting for window averaging). The x-axis uses scientific notation for readability with large episode counts.

Parameters:
pos_datanp.ndarray

Position data with shape (time_steps, num_agents). Assumes data is already window-averaged (each row represents window_size episodes).

window_sizeint, optional

The number of episodes averaged per data point (for x-axis labeling). Default is 10000.

num_agentsint, optional

Number of agents. If None, inferred from pos_data.shape[1].

title_adsList[str], optional

List of additional title components to append. Default is [].

name_adsList[str], optional

List of additional name components for file saving. Default is [].

short_titlebool, optional

If True, use a shorter figure title. Default is False.

savebool, optional

If True, saves the figure to file. Default is False.

save_typesList[str], optional

List of file extensions for saving. Default is ['.png', '.svg'].

fontdict, optional

Font configuration dictionary with keys:

  • 'default_size': Default font size (default: 12)

  • 'title_size': Title font size (default: 14)

  • 'legend_size': Legend font size (default: 12)

  • 'font_family': Font family (default: ‘sans-serif’)

paper_figuredict, optional

Paper figure configuration with keys:

  • 'paper': bool, whether this is a paper figure

  • 'section': str, section identifier

  • 'figure_id': str, figure identifier

figsizeTuple[float, float], optional

Figure size as (width, height). Default is (10, 6).

num_ticksint, optional

Approximate number of x-axis ticks to display. Default is 6.

axis_returnbool, optional

If True, returns the axes object instead of figure. Default is False.

Returns:
matplotlib.figure.Figure or matplotlib.axes.Axes

Figure object (or Axes if axis_return=True) showing agent position trajectories.

Examples

Basic usage with pre-averaged position data:

>>> import numpy as np
>>> pos_averaged = np.random.rand(400, 3)  # 400 windows, 3 agents
>>> fig = agent_position_trajectory(pos_averaged, window_size=10000, 
...                                  title_ads=['Experiment 1'])

Save figure for publication:

>>> fig = agent_position_trajectory(pos_averaged, window_size=10000,
...                                  save=True, 
...                                  paper_figure={'paper': True, 'section': 'results', 'figure_id': 'trajectory'})
InflGame.MARL.MARL_plots.policy_deterministically_to_actions(env, q_table=None, q_tensor=None, initial_position=array([0, 1]), num_step=10, temperature=1)#

Simulates deterministic actions for agents based on their policies. By doing the following

1. The Q-table is converted to a policy using a softmax function. i.e.

\[P(a|s) = \frac{e^{Q(s,a)/T}}{\sum_{a'} e^{Q(s,a')/T}}\]
where:

  • \(a\) is the action

  • \(s\) is the current state

  • \(a'\) is the next state

  • \(T\) is the temperature parameter

  • \(P(a|s)\) is the probability of taking action \(a\) in state \(s\)

  • \(Q(s,a)\) is the Q-value for action \(a\) in state \(s\)

  1. The maximum action is selected for each state.

  2. The environment is stepped through the selected actions for a specified number of steps.

  3. The positions and rewards are recorded at each step.

Parameters:
envinfluencer_env_async

The environment object.

q_tabledict, optional

Q-table in dictionary format. Defaults to None.

q_tensortorch.Tensor, optional

Q-table as a torch.Tensor. Defaults to None.

initial_positionnp.ndarray

Initial position of players. Defaults to np.array([0, 1]).

num_stepint

Number of steps to simulate. Defaults to 10.

temperaturefloat

A smoothness factor for the softmax function. Defaults to 1.

Returns:
tuple[torch.Tensor, torch.Tensor]

Position matrix and reward matrix as torch.Tensors.

InflGame.MARL.MARL_plots.policy_histogram(num_agents=2, q_table=None, q_tensor=None, agent_id=0, temperature=1, title_ads=[], name_ads=[], save=False, save_types=['.png', '.svg'], font={'default_size': 12, 'font_family': 'sans-serif', 'legend_size': 12, 'title_size': 14}, paper_figure={'figure_id': 'policy_histogram', 'paper': False, 'section': 'A'}, figsize=(6, 6), axis_return=False)#

Visualizes the Q-table as a policy using a softmax function and plots it as a heatmap.

\[P(a|s) = \frac{e^{Q(s,a)/T}}{\sum_{a'} e^{Q(s,a')/T}}\]
where:

  • \(a\) is the action

  • \(s\) is the current state

  • \(a'\) is an alternate action

  • \(T\) is the temperature parameter

  • \(P(a|s)\) is the probability of taking action \(a\) in state \(s\)

  • \(Q(s,a)\) is the Q-value for action \(a\) in state \(s\)

Parameters:
num_agentsint

Number of agents (used for saving metadata / titles). Defaults to 2.

q_tabledict, optional

Q-table in dictionary format. Defaults to None.

q_tensortorch.Tensor, optional

Q-table as a torch.Tensor. Defaults to None.

agent_idint

Agent’s ID number. Defaults to 0.

temperaturefloat

A smoothness factor for the softmax function. Defaults to 1.

title_adsList[str], optional

List of additional title components to append. Default is [].

name_adsList[str], optional

List of additional name components for file saving. Default is [].

savebool, optional

If True, saves the figure to file. Default is False.

save_typesList[str], optional

List of file extensions for saving. Default is ['.png', '.svg'].

fontdict, optional

Font configuration dictionary with keys 'default_size', 'title_size', 'legend_size', and 'font_family'.

paper_figuredict, optional

Paper figure configuration with keys 'paper', 'section', and 'figure_id'.

figsizeTuple[float, float], optional

Figure size as (width, height). Default is (6, 6).

axis_returnbool, optional

If True, returns the axes object instead of figure. Default is False.

Returns:
matplotlib.figure.Figure or matplotlib.axes.Axes

Figure representing the policy as a heatmap (or Axes if axis_return=True).

InflGame.MARL.MARL_plots.pos_plot(pos_matrix, possible_agents, domain_bounds, title_ads=[], name_ads=[], save=False, save_types=['.png', '.svg'], font={'default_size': 12, 'font_family': 'sans-serif', 'legend_size': 12, 'title_size': 14}, paper_figure={'figure_id': 'pos_plot', 'paper': False, 'section': 'A'}, figsize=(6, 6), axis_return=False)#

Plots the positions of all players over time.

Parameters:
pos_matrixtorch.Tensor

Matrix containing positions for each player at each step.

possible_agentsdict

Dictionary of possible agents in the environment.

domain_boundslist

List containing the lower and upper bounds of the domain.

title_adsList[str], optional

List of additional title components to append. Default is [].

name_adsList[str], optional

List of additional name components for file saving. Default is [].

savebool, optional

If True, saves the figure to file. Default is False.

save_typesList[str], optional

List of file extensions for saving. Default is ['.png', '.svg'].

fontdict, optional

Font configuration dictionary with keys 'default_size', 'title_size', 'legend_size', and 'font_family'.

paper_figuredict, optional

Paper figure configuration with keys 'paper', 'section', and 'figure_id'.

figsizeTuple[float, float], optional

Figure size as (width, height). Default is (6, 6).

axis_returnbool, optional

If True, returns the axes object instead of figure. Default is False.

Returns:
matplotlib.figure.Figure or matplotlib.axes.Axes

A figure of the agent positions through time using the optimal policy (or Axes if axis_return=True).

InflGame.MARL.MARL_plots.reward_plot(reward_matrix, possible_agents, title_ads=[], name_ads=[], save=False, save_types=['.png', '.svg'], font={'default_size': 12, 'font_family': 'sans-serif', 'legend_size': 12, 'title_size': 14}, paper_figure={'figure_id': 'reward_plot', 'paper': False, 'section': 'A'}, figsize=(6, 6), axis_return=False)#

Plots the rewards for all players over time.

Parameters:
reward_matrixtorch.Tensor

Matrix containing rewards for each player at each step.

possible_agentsdict

Dictionary of possible agents in the environment.

title_adsList[str], optional

List of additional title components to append. Default is [].

name_adsList[str], optional

List of additional name components for file saving. Default is [].

savebool, optional

If True, saves the figure to file. Default is False.

save_typesList[str], optional

List of file extensions for saving. Default is ['.png', '.svg'].

fontdict, optional

Font configuration dictionary with keys 'default_size', 'title_size', 'legend_size', and 'font_family'.

paper_figuredict, optional

Paper figure configuration with keys 'paper', 'section', and 'figure_id'.

figsizeTuple[float, float], optional

Figure size as (width, height). Default is (6, 6).

axis_returnbool, optional

If True, returns the axes object instead of figure. Default is False.

Returns:
matplotlib.figure.Figure or matplotlib.axes.Axes

A figure of the reward through time using the optimal policy (or Axes if axis_return=True).