Run a session of the SCML world (2020)

The SCML world (Supply Chain Management League) runs on top of NegMAS.

In this tutorial, you will test a run of this world.

Firstly, let’s import everything from the SCML app

from scml.scml2020 import *

There are several ways to create an SCML world. One of the simplest is to use the generate method of the SCML2020World class. This will create a dict that can be passed to the SCML2020World constructor as keyword arguments.

Here we explicitly set construct_graphs to True which slows the simulation yet allows us to see graph representations of what is happening in the world.

# agent_types = [DecentralizingAgent, BuyCheapSellExpensiveAgent, ReactiveAgent, IndDecentralizingAgent, MovingRangeAgent]
agent_types = [DecentralizingAgent, MovingRangeAgent]
world = SCML2020World(
    **SCML2020World.generate(
        agent_types=agent_types,
        n_steps=30
    ),
    construct_graphs=True,
)

Let’s draw a graph to see what is in this world

_, _ = world.draw()
../_images/01.run_scml2020_6_0.png

If you want to just test your installation (and do not care whether you get an accurate indicator of agent performance), you can set the number of steps to a small value (e.g. n_steps=10).

Now you can run this world simulation by just calling run.

world.run() # may take few minutes

Let’s see what happened in this run. Firstly, how many negotiations were conducted over time

plt.plot(world.stats['n_negotiations'])
plt.xlabel('Simulation Step')
plt.ylabel('N. Negotiations')
plt.show()
../_images/01.run_scml2020_10_0.png

It is clear that many negotiations happened at the beginning of the simulation with smaller number later. That is expected as the agents at the first and last production layer receive more exogenous contracts in the beginning.

Several other market statistics are available:

pprint(list(_ for _ in world.stats.keys() if "@" not in _  ))
['n_registered_negotiations_before',
 'n_contracts_dropped',
 'n_contracts_nullified_now',
 'n_bankrupt',
 'trading_price_0',
 'sold_quantity_0',
 'unit_price_0',
 'trading_price_1',
 'sold_quantity_1',
 'unit_price_1',
 'trading_price_2',
 'sold_quantity_2',
 'unit_price_2',
 'trading_price_3',
 'sold_quantity_3',
 'unit_price_3',
 'trading_price_4',
 'sold_quantity_4',
 'unit_price_4',
 'productivity',
 'market_size',
 'production_failures',
 'bankruptcy',
 'n_contracts_executed',
 'n_contracts_erred',
 'n_contracts_nullified',
 'n_contracts_cancelled',
 'n_breaches',
 'breach_level',
 'n_contracts_signed',
 'n_contracts_concluded',
 'n_negotiations',
 'n_negotiation_rounds_successful',
 'n_negotiation_rounds_failed',
 'n_negotiation_successful',
 'n_negotiation_failed',
 'n_registered_negotiations_after',
 'activity_level',
 'step_time',
 'total_time']

Let’s start by seeing how long did each step take (note that stats access the stats as a Dict[str, List] but stats_df access the same data as a pandas dataframe.

plt.bar(range(world.n_steps), world.stats_df['step_time'])
plt.xlabel('Simulation Step')
plt.ylabel('Time (s)')
plt.show()
../_images/01.run_scml2020_14_0.png

There are statistics specific for each agent that all have “_{agent_name}”. Lets check what is available for the winner agent:

winner = world.winners[0]
pprint(list(_ for _ in world.stats.keys() if winner.name in _ ))
['spot_market_quantity_02Dec@0',
 'spot_market_loss_02Dec@0',
 'balance_02Dec@0',
 'inventory_02Dec@0_input',
 'inventory_02Dec@0_output',
 'productivity_02Dec@0',
 'assets_02Dec@0',
 'bankrupt_02Dec@0',
 'score_02Dec@0']

The convension is that agent names has the form {ind}{Type}{process} where ind is a unique index, Type is a shortened version of the agent’s type name, and process is the process the agnet can run. Note that the agent’s input product has the same number as its process and its output product has the next number (i.e. an agent that runs process 1, has input product 1 and output product 2).

We can see that 8 pieces of information are available (for each time-step of the simulation):

  • bankrupt If true, the agent is bankrupt.

  • balance The money the agent has in its wallet (account).

  • inventory (input) The number of units of the agent’s input product available in its inventory (by the end of the simulation step).

  • inventory (output) The number of units of the agent’s output product available in its inventory (by the end of the simulation step).

  • assets The value of the agent’s assets (input and output products in inventory) evaluated at the trading price

  • spot market quantity The quantity bought by this agent from the spot market (of its output product on this step). This can only happen as a result of a product-breach.

  • spot market loss The spot market price for the agent. This value will go up the more the agent buys from the spot market and will be used to calculate the price for this agent at future steps. This way agents that depend on the spot market instead of negotiation get punished.

  • productivity The fraction of the agent’s production lines that were active at a given time-step.

  • score The score of the agent according to the evaluation rule of ANAC SCML 2020

Let’s see how did our agent do

#show the first and last value of each of the agent statistics
pprint({k:(v[0], v[-1]) for k, v in world.stats.items() if winner.name in k })
{'assets_02Dec@0': (162.0, 1248.2477727434095),
 'balance_02Dec@0': (7196, 10214),
 'bankrupt_02Dec@0': (False, False),
 'inventory_02Dec@0_input': (0, 0),
 'inventory_02Dec@0_output': (9, 51),
 'productivity_02Dec@0': (0.9, 0.0),
 'score_02Dec@0': (-0.0012352456766401318, 0.48752729705897674),
 'spot_market_loss_02Dec@0': (0.3, 0.5565),
 'spot_market_quantity_02Dec@0': (0, 0)}
stats = pd.DataFrame(data=world.stats)
fig, axs = plt.subplots(2, 3)
for ax, key in zip(axs.flatten().tolist(), ["score", "balance", "assets", "productivity",
                         "spot_market_quantity", "spot_market_loss"]):
    ax.plot(stats[f"{key}_{winner}"])
    ax.set(ylabel=key)
fig.show()
../_images/01.run_scml2020_19_0.png

We can for example check the activity level of this world (defined as the total amount of money transferred due to trade)

plt.plot(world.stats['activity_level'])
plt.xlabel('Simulation Step')
plt.ylabel('Activitiy Level ($)\nTotal Money Transferred')
plt.show()
../_images/01.run_scml2020_21_0.png

We can see a picture of contracting in this world as follows:

plt.plot(world.stats['n_contracts_concluded'], label='Concluded Contracts')
plt.plot(world.stats['n_contracts_cancelled'], label='Cancelled Contracts')
plt.plot(world.stats['n_contracts_signed'], label='Signed Contracts')
plt.plot(world.stats['n_contracts_executed'], label='Executed Contracts')
plt.legend()
plt.xlabel('Simulation Step')
plt.ylabel('N. Contracts')
plt.show()
../_images/01.run_scml2020_23_0.png

We can also check the breaches that happened

plt.plot(world.stats['breach_level'])
plt.xlabel('Simulation Step')
plt.ylabel('Total Breach Level')
plt.show()
../_images/01.run_scml2020_25_0.png

Notice that there can be multiple winners

winner_profits = [100 * world.scores()[_.id] for _ in world.winners]
winner_types = [_.short_type_name for _ in world.winners]
print(f"{world.winners} of type {winner_types} won at {winner_profits}%")
[02Dec@0] of type ['decentralizing'] won at [48.75272970589767]%

Let’s check how did the first winner’s inventory changes over time:

# find the keys in stats for the input and output inventory
in_key = [_ for _ in world.stats.keys() if _.startswith(f'inventory_{winner}_input')][0]
out_key = [_ for _ in world.stats.keys() if _.startswith(f'inventory_{winner}_output')][0]

# find input and output product indices
input_product, output_product = winner.awi.my_input_product, winner.awi.my_output_product
# draw
fig, (quantity, value) = plt.subplots(1, 2)
quantity.plot(world.stats[in_key], label=f"Input Product")
quantity.plot(world.stats[out_key], label=f"Output Product")
quantity.set(xlabel='Simulation Step', ylabel='Winner\'s Total Storage (item)')
quantity.legend()
value.plot(np.array(world.stats[in_key]) * np.array(world.stats[f"trading_price_{input_product}"])
              , label=f"Input Product")
value.plot(np.array(world.stats[out_key]) * np.array(world.stats[f"trading_price_{output_product}"])
              , label=f"Output Product")
value.set(xlabel='Simulation Step', ylabel='Winner\'s Inventory Value ($)')
value.legend()
fig.show()
../_images/01.run_scml2020_29_0.png

We can actually check what happens to ALL competitors:

from scml.scml2020.world import is_system_agent
fig, (profit, score) = plt.subplots(1, 2)
snames = sorted(world.non_system_agent_names)
for name in snames:
    profit.plot(100.0 * (np.asarray(world.stats[f'balance_{name}'])/world.stats[f'balance_{name}'][0] - 1.0), label=name)
    score.plot(100 * np.asarray(world.stats[f'score_{name}']), label=name)
profit.set(xlabel='Simulation Step', ylabel='Player Profit Ignoring Inventory (%)')
profit.legend(loc='lower left')
score.set(xlabel='Simulation Step', ylabel='Player Score (%)')
fig.show()
../_images/01.run_scml2020_31_0.png
from scml.scml2020.world import is_system_agent
fig, (profit, score) = plt.subplots(1, 2)
snames = sorted(world.non_system_agent_names)
for name in snames:
    profit.plot((np.asarray(world.stats[f'balance_{name}'])), label=name)
    score.plot(np.asarray(world.stats[f'score_{name}'])*(world.stats[f'balance_{name}'][0]), label=name)
profit.set(xlabel='Simulation Step', ylabel='Player Balance ($)')
profit.legend(loc='lower left')
score.set(xlabel='Simulation Step', ylabel='Player Score Unnormalized ($)')
fig.show()
../_images/01.run_scml2020_32_0.png

or just look at the end of the game

fig, (score, profit) = plt.subplots(1, 2)
final_scores = [100 * world.stats[f"score_{_}"][-1]
                for _ in world.non_system_agent_names]
final_profits = [100 * world.stats[f"balance_{_}"][-1] / world.stats[f"balance_{_}"][0] - 100
                 for _ in world.non_system_agent_names]
plt.setp(score.xaxis.get_majorticklabels(), rotation=45)
plt.setp(profit.xaxis.get_majorticklabels(), rotation=45)
score.bar(world.non_system_agent_names, final_scores)
profit.bar(world.non_system_agent_names, final_profits)
score.set(ylabel="Final Score (%)")
profit.set(ylabel="Final Profit (%)")

fig.show()
../_images/01.run_scml2020_34_0.png
fig, (score, profit) = plt.subplots(1, 2)
final_scores = [world.stats[f"score_{_}"][-1] * (world.stats[f"balance_{_}"][0])
                for _ in world.non_system_agent_names]
final_profits = [world.stats[f"balance_{_}"][-1]
                 for _ in world.non_system_agent_names]
plt.setp(score.xaxis.get_majorticklabels(), rotation=45)
plt.setp(profit.xaxis.get_majorticklabels(), rotation=45)
score.bar(world.non_system_agent_names, final_scores)
profit.bar(world.non_system_agent_names, final_profits)
score.set(ylabel="Final Unnormalized Score ($)")
profit.set(ylabel="Final Balance  ($)")

fig.show()
../_images/01.run_scml2020_35_0.png

You can inspect what happened in the simulation by plotting different output statistics. For example, we can see how did the trading price of different products change over the simulation time.

fig, axs = plt.subplots(2, 2)
for ax, key in zip(axs.flatten().tolist(), ["trading_price", "sold_quantity", "unit_price"]):
    for p in range(world.n_products):
        ax.plot(world.stats[f"{key}_{p}"], marker="x", label=f"Product {p}")
        ax.set_ylabel(key.replace("_", " ").title())
        ax.legend().set_visible(False)
axs[-1, 0].legend(bbox_to_anchor=(1, -.5), ncol=3)
fig.show()
../_images/01.run_scml2020_37_0.png
fig, axs = plt.subplots(1, 2)
for ax, key in zip(axs.flatten().tolist(), ["spot_market_quantity", "spot_market_loss"]):
    for a in world.non_system_agent_names:
        ax.plot(world.stats[f"{key}_{a}"], marker="x", label=f"{a}")
        ax.set_ylabel(key.replace("_", " ").title())
        ax.legend().set_visible(False)
axs[0].legend(bbox_to_anchor=(1, -.2), ncol=4)
fig.show()
../_images/01.run_scml2020_38_0.png

You can dig futher to understand what happened during this siumulation. For example, let’s see some of the contracts that were signed:

# create a view with only signed contracts
contracts = world.contracts_df
signed = contracts.loc[contracts.signed_at>=0, :]
fields = ["seller_name", "buyer_name", "delivery_time", "quantity", "unit_price",
          "signed_at", "executed", "breached", "nullified", "erred"]
signed[fields].sort_values(["quantity", "unit_price"], ascending=False).head(10)
seller_name buyer_name delivery_time quantity unit_price signed_at executed breached nullified erred
738 03Dec@1 06Dec@2 18 38 41 12 False True False False
353 03Dec@1 07Dec@2 12 35 43 2 False True False False
359 03Dec@1 06Dec@2 12 35 43 2 False True False False
485 07Dec@2 10Dec@3 11 32 64 5 False True False False
432 07Dec@2 11Dec@3 12 32 59 4 False True False False
387 03Dec@1 07Dec@2 8 29 40 3 True False False False
530 03Dec@1 07Dec@2 11 25 28 7 True False False False
542 03Dec@1 06Dec@2 11 25 28 7 False True False False
494 00Dec@0 03Dec@1 14 18 28 6 True False False False
549 00Dec@0 03Dec@1 14 18 28 7 False True False False

Let’s check some of the contracts that were fully executed

signed.loc[signed.executed, fields].sort_values(["quantity", "unit_price"], ascending=False).head(10)
seller_name buyer_name delivery_time quantity unit_price signed_at executed breached nullified erred
387 03Dec@1 07Dec@2 8 29 40 3 True False False False
530 03Dec@1 07Dec@2 11 25 28 7 True False False False
494 00Dec@0 03Dec@1 14 18 28 6 True False False False
839 06Dec@2 11Dec@3 21 17 60 15 True False False False
652 00Dec@0 03Dec@1 18 17 28 10 True False False False
436 02Dec@0 03Dec@1 11 17 26 4 True False False False
428 00Dec@0 03Dec@1 12 16 28 4 True False False False
570 02Dec@0 03Dec@1 16 16 28 8 True False False False
254 00Dec@0 03Dec@1 6 16 26 1 True False False False
290 02Dec@0 03Dec@1 7 16 26 1 True False False False
signed.loc[signed.breached, fields[:-4] + ["breaches"]].sort_values(["quantity", "unit_price"], ascending=False).head(10)
seller_name buyer_name delivery_time quantity unit_price signed_at breaches
738 03Dec@1 06Dec@2 18 38 41 12 03Dec@1:product(0.2894736842105263)
353 03Dec@1 07Dec@2 12 35 43 2 03Dec@1:product(0.7142857142857143)
359 03Dec@1 06Dec@2 12 35 43 2 03Dec@1:product(1.0)
485 07Dec@2 10Dec@3 11 32 64 5 07Dec@2:product(0.4375)
432 07Dec@2 11Dec@3 12 32 59 4 07Dec@2:product(0.6875)
542 03Dec@1 06Dec@2 11 25 28 7 03Dec@1:product(0.72)
549 00Dec@0 03Dec@1 14 18 28 7 00Dec@0:product(0.6666666666666666)
703 00Dec@0 03Dec@1 16 17 26 11 00Dec@0:product(0.11764705882352941)
346 00Dec@0 03Dec@1 10 16 28 2 00Dec@0:product(0.125)
914 03Dec@1 06Dec@2 21 16 28 17 03Dec@1:product(0.625)

We can now see how does the singning day affect delivery day, product and quantity

fig, ax = plt.subplots(1, 3)
for i, x in enumerate(["delivery_time", "quantity", "product_index"]):
    ax[i].scatter(signed.signed_at, signed[x])
    ax[i].set(ylabel=x.replace("_", " ").title(), xlabel="Signing Day")
fig.show()
../_images/01.run_scml2020_46_0.png
fig, ax = plt.subplots(1, 3)
for i, x in enumerate(["delivery_time", "unit_price", "product_index"]):
    ax[i].scatter(signed.quantity, signed[x])
    ax[i].set(ylabel=x.replace("_", " ").title(), xlabel="Quantity")
fig.show()
../_images/01.run_scml2020_47_0.png

Did any agents go bankrupt and when?

bankruptcy = {a: np.nonzero(stats[f"bankrupt_{a}"])[0]
        for a in world.non_system_agent_names}
pprint({k: "No" if len(v)<1 else f"at: {v[0]}" for k, v in bankruptcy.items()})
{'00Dec@0': 'No',
 '01Mov@0': 'No',
 '02Dec@0': 'No',
 '03Dec@1': 'No',
 '04Mov@1': 'No',
 '05Mov@1': 'No',
 '06Dec@2': 'No',
 '07Dec@2': 'No',
 '08Mov@2': 'No',
 '09Mov@3': 'No',
 '10Dec@3': 'No',
 '11Dec@3': 'No'}

You can see what happened during this simulation by drawing graphs at different steps. The meaning of different edge colors can be drawn as follows:

from negmas import show_edge_colors
show_edge_colors()
../_images/01.run_scml2020_51_0.png

You can see what happened in this world in a series of graphs using the draw method

world.draw(steps=(0, world.n_steps), together=False, ncols=2, figsize=(20, 20))
plt.show()
../_images/01.run_scml2020_53_0.png

You can also run a simple animation to see what happens at every step (you need to download the jupyter notebook and execute it to see the animation) :

# prepare animation
from IPython.display import clear_output
from time import perf_counter
from negmas.helpers import humanize_time
# run the world and animate it
draw_every = 1 # draw every 5 steps (drawing is REALLY slow)
single_graph = False # show a graph for every event type
what = ["contracts-signed", "contracts-breached", "contracts-executed",]
for s in range(world.n_steps):
    if s % draw_every != 0:
        continue
    world.draw(what=what, steps=(s-draw_every, s), together=single_graph, ncols=3, figsize=(20, 5))
    plt.show()
    clear_output(wait=True)
world.draw(what=what, steps=(s-draw_every, s), together=single_graph, ncols=3, figsize=(20,5))
plt.show()
../_images/01.run_scml2020_55_0.png

Note that this graph shows only what happened in the last draw_every steps.

Let’s see some statistics about the simulation.

Running a tournament

Now that you can run simple world simulations, let’s try to run a complete tournament and see its results. Let’s start by running a standard tournament (in which each agent is represented by a single factory). Running a collusion tournament will be exactly the same with the only difference that anac2020_std will be replaced with anac2020_collusion.

Note that in the real competition we use thousands of configurations and longer simulation steps (e.g. 50 \(\le\) n_steps \(\le\) 500).

from scml.scml2020.utils import anac2020_std
tournament_types = agent_types + [RandomAgent]
# may take a long time
results = anac2020_std(
    competitors=tournament_types,
    n_configs=12, # number of different configurations to generate
    n_runs_per_world=1, # number of times to repeat every simulation (with agent assignment)
    n_steps = 10, # number of days (simulation steps) per simulation
    print_exceptions=True,
)

Who was the winner?

results.winners
['decentralizing_agent']

How many simulations were actually run?

len(results.scores.run_id.unique())
36

The total number of simulations \(n_{s}\) will be \(n_t \times n_c \times n_r\) where \(n_t\) is the number of competitor agent types, \(n_c\) is the number of configurations, and \(n_r\) is the number of runs per configuration

We can also see the scores that every agent type got

results.score_stats
agent_type count mean std min 25% 50% 75% max
0 decentralizing_agent 36.0 0.641585 0.382862 0.013917 0.437722 0.624334 0.850786 1.811421
1 moving_range_agent 36.0 0.017210 0.098589 -0.164224 -0.029696 0.005919 0.055612 0.268376
2 random_agent 36.0 -0.171960 0.116389 -0.456237 -0.241420 -0.167361 -0.121497 0.000000

You can also do statistical significance testing using ttest or kstest (with multi-comparison correction)

results.kstest
a b t p n_a n_b n_effective
0 random_agent moving_range_agent 0.694444 1.365966e-08 36 36 36
1 random_agent decentralizing_agent 1.000000 4.519646e-21 36 36 36
2 moving_range_agent decentralizing_agent 0.805556 6.657934e-12 36 36 36

see the total score

results.total_scores
agent_type score
0 decentralizing_agent 0.624334
1 moving_range_agent 0.005919
2 random_agent -0.167361

or the aggregated statistics of the world. For example, let’s draw the activity level for different simulations.

plt.errorbar(range(len(results.agg_stats)),
             results.agg_stats.activity_level_mean,
             np.sqrt(results.agg_stats.activity_level_var)
             )
plt.xlabel("Simulation Number")
plt.ylabel("Activity Level")
plt.show()
../_images/01.run_scml2020_71_0.png

We can even get the scores of every agent belonging to every agent type at every simulation

results.scores.loc[:, ["agent_name", "agent_type", "score"]].head()
agent_name agent_type score
0 00Ran@0 random_agent -0.274163
1 01Mov@0 moving_range_agent -0.028083
2 02Dec@1 decentralizing_agent 0.013917
3 02Mov@1 moving_range_agent 0.000000
4 03Dec@1 decentralizing_agent 1.275761

or inspect any statistic we like

ax = sns.violinplot(data=results.stats, x="step", y="activity_level")
ax.set(ylabel="Activity Level / Business Size ($)", xlabel="Step Number", yscale="log")

fig.show()
../_images/01.run_scml2020_75_0.png

Let’s see how did the location at the production graph affect the score of each type.

results.scores["level"] = results.scores.agent_name.str.split("@", expand=True).loc[:, 1]
sns.lineplot(data=results.scores[["agent_type", "level", "score"]],
             x="level", y="score", hue="agent_type")
plt.plot([0.0] * len(results.scores["level"].unique()), "b--")
plt.show()
../_images/01.run_scml2020_77_0.png

Now that you can run simulations and complete tournament, let’s see how are we going to develop a new agent for the SCML2020 league \(\rightarrow\)

Download Notebook.