IQL Utilities#

IQL Utilities Module#

This module provides utility functions for Independent Q-Learning (IQL) in multi-agent reinforcement learning (MARL). The functions include methods for converting Q-tables to tensors, generating policies from Q-tensors, and adjusting parameters such as epsilon, temperature, and the number of episodes based on various scheduling strategies.

The module supports: - Conversion of Q-tables to tensor representations for efficient computation. - Policy generation using softmax over Q-values. - Dynamic adjustment of epsilon, temperature, and episodes for exploration-exploitation trade-offs. - Scheduling strategies such as cosine annealing and reverse cosine annealing.

Dependencies:#

  • InflGame.utils

Usage:#

The functions in this module can be used to preprocess Q-tables, generate policies, and dynamically adjust parameters for MARL algorithms.

Examples#

from InflGame.MARL.utils.IQL_utils import Q_table_to_tensor, Q_tensor_to_policy, adjusted_epsilon

# Convert a Q-table to a tensor
Q_table = {
    0: {0: {0: 1.0, 1: 2.0}, 1: {0: 3.0, 1: 4.0}},
    1: {0: {0: 5.0, 1: 6.0}, 1: {0: 7.0, 1: 8.0}}
}
Q_tensor = Q_table_to_tensor(Q_table)

# Generate a policy from the Q-tensor
policy = Q_tensor_to_policy(Q_tensor, temperature=0.5, agent_id=0)

# Adjust epsilon using cosine annealing
configs = {
    'TYPE': 'cosine_annealing',
    'epsilon_min': 0.1,
    'epsilon_max': 1.0
}
epsilon = adjusted_epsilon(configs, num_agents=2, episode=10, episodes=100)
print(f"Adjusted epsilon: {epsilon}")

Functions

InflGame.MARL.utils.IQL_utils.Q_table_to_tensor(Q_table)#

Converts a Q-table (nested dictionary) into a tensor representation.

\[Q_{torch}[i,j,k] = Q_{dict}[p_i,s_i, a_j]\]
where:

  • \(Q(s_i, a_j, a_k)\) is the Q-value for state \(s_i\), actions \(a_j\) , and \(p_i\) the \(i\)-th player.

Parameters:
Q_tabledict

Nested dictionary representing the Q-table. Expected structure: Q_table[agent][state][action] = value. Agents may be strings (e.g. ‘player0’) or integers.

Returns:
torch.Tensor

Tensor representation of the Q-table.

InflGame.MARL.utils.IQL_utils.Q_tensor_to_policy(q_tensor, temperature=0.5, agent_id=0)#

Converts a Q-tensor into a policy tensor using a softmax function.

\[\pi(a|s) = \]

rac{exp(Q(s, a) / T)}{sum_{a’} exp(Q(s, a’) / T)}

where:

  • \(a\) is the action.

  • \(a'\) is the set of all possible actions.

  • \(s\) is the state.

  • \(Q(s, a)\) is the Q-value for state \(s\) and action \(a\).

  • \(T\) is the temperature parameter.

  • \(\pi(a|s)\) is the policy for action \(a\) given state \(s\).

Parameters:
q_tensortorch.Tensor

Q-tensor for all players.

temperaturefloat

Temperature parameter for softmax.

agent_idint

ID of the player for which to compute the policy.

Returns:
torch.Tensor

Policy tensor for the specified player.

InflGame.MARL.utils.IQL_utils.adjusted_episodes(configs, epoch, epochs)#

Adjusts the number of episodes based on the specified configuration and epoch progress.

Parameters:
configsdict

Configuration dictionary containing episode adjustment parameters TYPE, episode_max, episode_min. TYPE (str): schedule type episodes_min (int): minimum number of episodes episodes_max (optional,int):maximum number of episodes

epochint

Current epoch number.

epochsint

Total number of epochs.

Returns:
int

Adjusted number of episodes.

InflGame.MARL.utils.IQL_utils.adjusted_epsilon(configs, num_agents, episode, episodes)#

Adjusts the epsilon value based on the specified configuration and episode progress.

Epsilon Adjustment Strategies#

Schedule Type - Epsilon Formula

Fixed - \(\epsilon = \epsilon_{ ext{constant}}\)

Cosine Annealing - :math:`epsilon = epsilon_{ ext{min}} +

rac{1}{2} (epsilon_{ ext{max}} - epsilon_{ ext{min}}) (1 + cos( rac{pi cdot ext{episode}}{ ext{episodes}}))`

Parameters:
configsdict

Configuration dictionary containing epsilon adjustment parameters.

num_agentsint

Number of players in the game.

episodeint

Current episode number.

episodesint

Total number of episodes.

Returns:
float

Adjusted epsilon value.

InflGame.MARL.utils.IQL_utils.adjusted_temperature(configs, observation, observation_space_size)#

Adjusts the temperature value based on the specified configuration and observation.

Parameters:
configsdict

Configuration dictionary containing temperature adjustment parameters.

observationint

Current observation value.

observation_space_sizeint

Size of the observation space.

Returns:
float

Adjusted temperature value.

InflGame.MARL.utils.IQL_utils.cosine_annealing_distance_dependent(value_max, value_min, time, time_crit, time_max, max_distance=None)#

Computes a value using cosine annealing based on the distance from a critical time.

\[v(t) = v_{\text{min}} + \frac{1}{2} (v_{\text{max}} - v_{\text{min}}) \left(1 + \cos\left(\frac{\pi \cdot |t - t_{\text{crit}}|}{d_{\text{max}}}\right)\right)\]
where:

  • \(v(t)\) is the computed value at time \(t\).

  • \(v_{\text{max}}\) is the maximum value.

  • \(v_{\text{min}}\) is the minimum value.

  • \(t_{\text{crit}}\) is the critical time step.

  • \(d_{\text{max}}\) is the maximum distance from the critical time, defaulting to half of \(t_{\text{max}}\).

This function adjusts the value smoothly using a cosine function, depending on the distance from a critical time step.

Parameters:
value_maxfloat

Maximum value.

value_minfloat

Minimum value.

timeint

Current time step.

time_critint

Critical time step.

time_maxint

Maximum time step.

max_distanceint

Maximum distance from the critical time. Defaults to half of time_max.

Returns:
float

Computed value based on cosine annealing.

InflGame.MARL.utils.IQL_utils.q_tables_to_q_tensors(num_runs, q_tables)#

Converts multiple Q-tables into a stacked tensor representation.

Parameters:
num_runsint

Number of runs (Q-tables).

q_tablesdict

Dictionary containing Q-tables for each run.

Returns:
torch.Tensor

Stacked tensor representation of all Q-tables.

InflGame.MARL.utils.IQL_utils.reverse_cosine_annealing(value_max, value_min, time, time_max)#

Reverse cosine annealing adjusts a value smoothly over time, starting from the minimum value and increasing toward the maximum value, following a cosine curve.

\[v(t) = v_{ ext{min}} + v_{ ext{max}} - \left\lfloor v_{ ext{min}} + \]
rac{1}{2} (v_{ ext{max}} - v_{ ext{min}})

left(1 + cosleft(

rac{pi cdot t}{t_{ ext{max}}} ight) ight) ight floor

where:

  • \(v(t)\) is the computed value at time \(t\).

  • \(v_{ ext{max}}\) is the maximum value.

  • \(v_{ ext{min}}\) is the minimum value.

  • \(t_{ ext{max}}\) is the maximum time step.

Parameters:
value_maxfloat

Maximum value.

value_minfloat

Minimum value.

timeint

Current time step.

time_maxint

Maximum time step.

Returns:
float

Computed value based on reverse cosine annealing.