General RL Utilities#

Multi-Agent Reinforcement Learning Utilities#

This module provides utility functions for multi-agent reinforcement learning (MARL) in influencer games. It includes functions to compute influence matrices, probability matrices, and influence kernels for agents interacting in a shared environment. The module supports various influence kernels, including Gaussian, Jones, Dirichlet, and Multi-variate Gaussian kernels.

Mathematical Definitions:#

Probability Matrix: The probability matrix \(G\) is defined as:

\[G_{i,k} = \frac{f_i(x_i, b_k)}{\sum_{j=1}^{N} f_j(x_j, b_k)}\]
where:
- \(f_i(x_i, b_k)\) is the influence of agent \(i\) on resource point \(b_k\)
- \(N\) is the total number of agents
- \(b_k\) is the `k`th resource point
Influence Matrix: The influence matrix \(I\) is defined as having the components:

\[\iota_{i,k} = f_i(x_i, b_k)\]
where:
- \(f_i(x_i, b_k)\) is the influence of agent \(i\) on resource point \(b_k\)
Influence Kernels: Various influence kernels are supported, including Gaussian, Jones, Dirichlet, and Multi-variate Gaussian kernels.

Dependencies:#

InflGame.utils
InflGame.kernels

Usage:#

The prob_matrix function computes the probability matrix for agents influencing resource points, while the influence_matrix function calculates the influence matrix for all agents. The influence function computes the influence of a specific agent’s kernel over resource points.

Functions

InflGame.MARL.utils.MARL_utils.influence_matrix_optimized(num_agents, agents_pos, bin_points, infl_configs, parameters, fixed_pa, infl_cshift=False, infl_fshift=False, cshift=None, Q=0.0)#

Compute the influence matrix of all agents over the bin points.

Returns a matrix whose rows are per-agent influence kernels

\[f_{i}(x_i,b)\]

for each agent \(i\) and resource/bin points \(b \in \mathbb{B}\).

There are several types of preset influence kernels, including:

Gaussian influence kernel

\[f_i(x_i,b,\sigma) = e^{-\frac {(x_i-b)^2}{2\sigma^2}}\]
Jones influence kernel

\[f_i(x_i,b,p) = |x-b|^p\]
Dirichlet influence kernel

\[f_i(\mathbb{\alpha},b)=\frac{1}{\beta(\alpha)}\prod_{l=1}^{L} b_l^{(\alpha_l-1)}\]

where \(L\) is the number of dimensions and \(b_l\) is the \(l\) th component of the bin point \(b\).

Here \(\mathbf{\alpha}\) is the parameter vector for the Dirichlet influence kernel, but \(\alpha_\phi\) is the fixed parameter such that

\[\alpha_l=\frac{\alpha_\phi}{x_{(i,\phi)}}*x_{(i,l)}\]

where \(x_{(i,\phi)}\) is the \(\phi\) th component of the position of the \(i\) th agent and \(x_{(i,l)}\) is the the \(l\) th component of the position of the \(i\) th agent.

Multi-variate Gaussian influence kernel

\[f_i(\mathbf{x}_i,\mathbf{b},\Sigma) = e^{-\frac{(\mathbf{x}_i-\mathbf{b})^T \Sigma^{-1} (\mathbf{x}_i-\mathbf{b})}{2}}\]

where \(\Sigma\) is the covariance matrix of the multi-variate Gaussian influence kernel.

Custom influence kernel (user-defined)

This influence kernel is defined by the user and can be any function that takes in the agent’s position, bin points, and parameters. Examples of custom influence kernels are provided in the demos.

Parameters:

num_agentsint: Number of agents.
agents_postorch.Tensor | numpy.ndarray: Positions of the agents.
bin_pointstorch.Tensor | numpy.ndarray: Positions of the resource points.
infl_configsdict: Configuration for the influence type.
parameterslist | numpy.ndarray | torch.Tensor: Parameters for the influence function.
fixed_paint: Fixed parameter index for Dirichlet-type kernels.
infl_cshiftbool: Whether to apply a constant influence shift.
infl_fshiftbool: Whether to apply a functional influence shift.
cshifttorch.Tensor, optional: Constant shift tensor (used when infl_cshift is True).
Qfloat: Scaling factor for functional shift (used when infl_fshift is True).

Returns:

torch.Tensor: Influence matrix of shape (num_agents, num_bins).

InflGame.MARL.utils.MARL_utils.observation_to_position(observations, possible_positions)#

Convert observations to positions in the domain.

Parameters:

observationsdict[str, int]: Current observations of all agents.
possible_positionslist[list[int]]: Lookup table mapping observation indices to positions.

Returns:

list[list[int]]: List of positions corresponding to the observations.

InflGame.MARL.utils.MARL_utils.positions_list(num_observations, num_agents=2)#

Generates a list of all possible positions for players based on the number of observations.

Parameters:

num_observationsint: The number of observations in the environment.
num_agentsint: The number of players in the environment.

Returns:

list: A list of all possible positions for the players.

InflGame.MARL.utils.MARL_utils.possible_observations(possible_agents, num_observations, num_agents)#

Generates all possible observations for the agents.

Parameters:

possible_agentslist[str]: A list of agent identifiers.
num_observationsint: The number of observations in the environment.
num_agentsint: The number of players in the environment.

Returns:

list[dict[str, int]]: A list of dictionaries representing all possible observations for the agents.

InflGame.MARL.utils.MARL_utils.prob_matrix(*args, **kwargs)#: Backward compatibility wrapper.

InflGame.MARL.utils.MARL_utils.prob_matrix_optimized(num_agents, agents_pos, bin_points, infl_configs, parameters, fixed_pa, infl_cshift=False, infl_fshift=False, cshift=None, Q=0.0)#: Optimized probability matrix computation using vectorization and caching.

InflGame.MARL.utils.MARL_utils.remove_all_tuples(input_list, times=2)#

Recursively removes tuples from a list a specified number of times.

Parameters:

input_listlist: A list that may contain tuples and integers.
timesint: The number of times to recursively remove tuples.

Returns:

list: A new list with tuples flattened into integers.

InflGame.MARL.utils.MARL_utils.remove_tuples(input_list)#

Removes tuples from a list, returning a new list with only integers.

Parameters:

input_listlist: A list that may contain tuples and integers.

Returns:

list: A new list with tuples flattened into integers.

InflGame.MARL.utils.MARL_utils.reward_dict(*args, **kwargs)#: Backward compatibility wrapper.

InflGame.MARL.utils.MARL_utils.reward_dict_optimized(possible_agents, possible_positions, num_observations, num_agents, bin_points, infl_configs, parameters, fixed_pa, infl_fshift, infl_cshift, cshift, Q, resource_distribution, normalize=True)#: Highly optimized reward dictionary computation using batch processing.

InflGame.MARL.utils.MARL_utils.reward_obs(*args, **kwargs)#: Backward compatibility wrapper.

InflGame.MARL.utils.MARL_utils.reward_obs_optimized(observations, possible_agents, possible_positions, num_agents, bin_points, infl_configs, parameters, fixed_pa, infl_fshift, infl_cshift, cshift, Q, resource_distribution)#: Optimized reward computation using vectorized operations.