pydialogue.baseline package

Submodules

pydialogue.baseline.evaluation module

This module introduces the functions needed for evaluating an agent’s self.

pydialogue.baseline.evaluation.accuracy_dia_type(main_player, dias, eval_results)

Shows the accuracy for each dialogue type. The accuracy is computed by dividing the number of successful dialogues by the total number of dialogues.

Parameters:

• main_player (Entity) – The agent that is trained.

• dias (list) – The list of dialogues that were evaluated.

• eval_results (list) – The results from the dialogues’ evaluation. if eval_results[idx] is 1 means that the dialogue dias[idx] was successful.

Returns:

accuracies – A list of tuples in the form: (dia_type, accuracy)

Return type:

list

pydialogue.baseline.evaluation.eval_dialogue(agent_policy, dialogue)

Evaluates a dialogue by replacing the default rule-based self with the agent_policy. But first, it checks whether the dialogue goal can be fulfilled with the rule-based self.

pydialogue.baseline.evaluation.eval_dialogues(agent_policy, dialogues, save_state=True)

Evaluates multiple dialogues using the eval_dialogue function.

Parameters:

• agent_policy (Policy) – This self replaces the rule-based self in the dialogue.

• dialogues (list) – A list of Dialogue instances to be evaluated.

• save_state (bool, optional) – If True, it saves the dialogues’ state and recovers it after the execution of the dialogues.

Returns:

results – The list of evaluated dialogues.

Return type:

list

pydialogue.baseline.evaluation.generate_and_eval(dia_generator, min_num_dialogues, min_agent_dialogues, agent_policy, flush_after=None, agent_prob=None, forgetful=False, return_dias=False, notebook_run=False, generate_dialogue_func=<function generate_dialogue>, warm_up=0)

Generates a number of dialogues and evaluates how many of the generated dialogues are a success when using the agent_policy.

Parameters:

• dia_generator (DialogueGenerator) – The object used for generating dialogues.

• min_num_dialogues (int) – The minimum number of dialogues to be generated.

• min_agent_dialogues (int) – The minimum number of dialogues to be generated where the agent participates in the dialogue.

• agent_policy (Policy) – The agent self generates utterances in the dialogues.

• flush_after (int) – It calls the dia_generator.flush() after the context reaches a certain length. It is done because the context size might grow large and not fit in RAM.

• agent_prob (float) – The probability that the player will be a participant and an agent in the next generated dialogue. The player is the owner of the agent_policy. The probability value ranges from 0 to 1.

• forgetful (bool) – If True, it saves the dialogue state before the dialogue execution and later recovers the state. For example, a dialogue changes the context state when a player utters. Later, this change is reverted.

• return_dias (bool) – If False, the generated dialogues are not returned. This might save RAM, otherwise all dialogues are appended to a list that may fill the RAM after some time.

• notebook_run (bool) – This parameter indicates whether the function is run from a jupyter notebook (True) or not (False). This parameter ensures the progress bars are shown correctly in a jupyter notebook.

• generate_dialogue_func (func) – The function used for generating a dialogue before it’s evaluated.

• warm_up (int) – The number of dialogues generated before the evaluation starts. The evaluated agent can learn from them on how to respond in the dialogues that follow.

Returns:

• dias (list) – The list of generated dialogues. The list contains all dialogues (including the ones where the agent is not a participant). If return_dias is False, an empty list is returned.

• individual_accuracies (dict) – The accuracy the agent has achieved per dialogue type.

• total_accuracy (int) – The number of successful dialogues where the agent participates divided by the total number of dialogues where the agent participates.

• total_num_dias (int) – The total number of dialogues generated where the agent participates.

• time_points (list) – A list of tuples (dia_type, eval_result, context_size). It helps evaluate how the agent performs over time.

• succ_total (dict) – A mapping from the dialogue type to the tuple (#num_succ, #num_total), where #num_succ is the total number of successful dialogues and #num_total is the total number of dialogues from that specific type

• accuracy_per_step (dict) – A mapping from the maximum number of steps needed for a dialogue to be successful to the accuracy of all dialogues that have that particular maximum number of steps.

• succ_total_per_step (dict) – A mapping from the maximum number of steps needed for a dialogue to be successful to the tuple (#num_succ, #num_total), where #num_succ is the number of successful dialogues that have that particular maximum number of steps and #num_total is the total number of dialogues that have that particular maximum number of steps.

pydialogue.baseline.evaluation.generate_dialogue(dia_generator, agent, users, agent_prob=None)

Generates a dialogue where the user issues a request to the agent.

Parameters:

• dia_generator (DialogueGenerator) – The dialogue generator is used for generating the dialogue.

• agent (Entity) – The agent that receives a command from a user.

• users (list) – A list of users that can issue commands.

• agent_prob (float, optional) – The probability that the agent will participate in the generated dialogue. If the agent is not selected, then some random user will get a command from another user.

Returns:

The generated dialogue.

Return type:

Dialogue

pydialogue.baseline.evaluation.load_and_merge_results(filenames)

Load multiple pickle files containing all the saved dictionaries from the results and merge them. The dictionaries stored are: test_points, succ_total, succ_total_per_step

Later, the dictionaries individual_accuracies, total_accuracy, total_num_dias are recomputed which are necessary for printing the results. They are returned together with the merged dictionaries.

pydialogue.baseline.evaluation.plot_acc(outcomes, label, epochs=None, smooth_over=10, show_plot=True)

Plot the accuracy over time.

Parameters:

• outcomes (list) – A list of 1s and 0s where 1 is a success.

• label (str) – The title of the plot.

• epochs (list) – A list of the x-axis values

• smooth_over (int) – The number of outcomes taken and averaged over in a sliding window fashion.

• show_plot (bool) – A value indicating whether the plot should be shown.

pydialogue.baseline.evaluation.plot_acc_individual(test_points, smooth_over=50, show_plot=False)

Plot accuracy over time graphs for each dialogue type and merge them in one.

pydialogue.baseline.evaluation.plot_context_sizes(time_points, instructions_size, token_factor=1.3)

Plot the context size over time.

pydialogue.baseline.evaluation.pretty_print_acc_per_step(sol_name, acc_per_step, total_per_step)

Prints the accuracy of the dialogues relative to the number of steps needed for the dialogue to be successful.

pydialogue.baseline.evaluation.pretty_print_eval(sol_name, individual_accuracies, total_accuracy, succ_total, num_agent_dias)

Prints the metrics for the leaderboard.

Parameters:

• sol_name (str) – The unique solution’s name.

• individual_accuracies (dict) – A mapping task_name : task_accuracy.

• total_accuracy (int) – The accuracy across all tasks.

• num_agent_dias (int) – The number of dialogues used for computing the accuracies where the agent participates.

Return type:

None

pydialogue.baseline.evaluation.save_results(filename, test_points, succ_total, succ_total_per_step)

Save variables to a pickle file.

pydialogue.baseline.evaluation.update_acc_table(dia_type_success, dia_type_total, dia_type, eval_result, num_steps, success_per_step, total_per_step)

Updates the number of successful dialogues per dialogue type and the total number of dialogues per type.

Parameters:

• dia_type_success (dict) – A mapping from a dialogue type to the number of successful dialogues per dialogue type.

• dia_type_total (dict) – A mapping from a dialogue type to the number of total dialogues per dialogue type.

• dia_type (Dialogue) – The dialogue is used to fetch the user self in order to get the dialogue type. The user request dictates the dialogue type. For example, the user request can be: Hannah get the apple or Hannah, Is the apple red?

• eval_result (int) – It indicates whether the dialogue was successful. A value of 1 indicates success.

• num_steps (int) – The maximum number of steps that the dialogue needs to finish successfully.

• success_per_step (dict) – A mapping from the maximum number of steps to the number of successful dialogues that require that number.

• total_per_step (dict) – A mapping from the maximum number of steps to the total number of dialogues that require that number.

Return type:

None

pydialogue.baseline.llm_base_policies module

This module introduces classes and functions needed to implement the baseline model.

class pydialogue.baseline.llm_base_policies.ChatTrimmerBase(max_input_tokens, chars_per_token=4, per_message_overhead=4)

Bases: object

count_tokens(text)

trim_messages(messages, preserve_system=True)

class pydialogue.baseline.llm_base_policies.LLMPolicyBase(player, database, main_model_kwargs, ser_model_kwargs, main_instr_filename, ser_instr_filename, trimmer, interrupt_dialogue=True, dialogue=None)

Bases: Policy, ABC

deserialize(serialized_output)

execute(include_goal=False, **params)

Uses the model to utter a sentence. If there is an error in the self, an empty sentence is returned.

generate_main_output()

abstract generate_output(messages, model_kwargs)

generate_ser_output(main_output_str)

get_goal(**params)

Returns the goal of the self.

get_steps(**params)

Returns the list of valid responses.

recover_state(state)

Recovers the members that change with time when the dialogue is run.

save_msgs_to_file(filename_main, filename_ser)

save_state()

Saves the members that change with time when the dialogue is run.

class pydialogue.baseline.llm_base_policies.OAIPolicy(player, database, main_model_kwargs, ser_model_kwargs, main_instr_filename, ser_instr_filename, trimmer, client, interrupt_dialogue=True)

Bases: LLMPolicyBase

generate_output(messages, model_kwargs)

pydialogue.baseline.llm_base_policies.extract_from_tags(tag, text)

Extracts the text inside the tags.

pydialogue.baseline.llm_policies module

class pydialogue.baseline.llm_policies.ChatTrimmer(tokenizer, max_input_tokens, encode_kwargs=None, per_message_overhead=6)

Bases: object

count_tokens(text)

decode_text(tokens)

encode_text(text)

trim_messages(messages, preserve_system=True)

class pydialogue.baseline.llm_policies.LLMPolicy(player, database, main_model_kwargs, ser_model_kwargs, main_instr_filename, ser_instr_filename, trimmer, model, tokenizer, device, gc_after=True, autoregressive=True, interrupt_dialogue=True, dialogue=None)

Bases: LLMPolicyBase

execute(include_goal=False, **params)

Uses the model to utter a sentence. If there is an error in the self, an empty sentence is returned.

generate_output(messages, model_kwargs)

class pydialogue.baseline.llm_policies.LLMPolicyAlt(player, database, model, main_model_kwargs, ser_model_kwargs, tokenizer, device, main_instr_filename, ser_instr_filename, max_context_len, gc_after=True, autoregressive=True, dialogue=None)

Bases: Policy

execute(include_goal=False, **params)

Uses the model to utter a sentence. If there is an error in the self, an empty sentence is returned.

get_goal(**params)

Returns the goal of the self.

get_steps(**params)

Returns the list of valid responses.

recover_state(state)

Recovers the members that change with time when the dialogue is run.

save_state()

Saves the members that change with time when the dialogue is run.

pydialogue.baseline.rag_policies module

pydialogue.baseline.serializers module

This module offers functions for serializing a Sentence to a list of strings and deserializing a list of strings to a Sentence.

pydialogue.baseline.serializers.attach_to_element(elem, elem_to_attach, world)

Attaches a child element (elem_to_attach) to a parent element (elem).

pydialogue.baseline.serializers.deserialize(cmds, world, stack=None)

Parses a list of tokens (strings) and converts them into an instance of type Sentence.

The parsing is done in the following manner: Each time a beginning token is encountered, the corresponding object instance is created and added to a stack. The beginning keyword indicates that the next few tokens belong to the object instance, and those tokens are attached to the object instance using the attach_to_element function. When the end keyword is encountered, the object instance is attached to a parent object if it exists.

Please check the serialize function for an example of serializing a Sentence object. The deserialization just reverses this process; it converts the list into a Sentence object.

Parameters:

• cmds (list) – A list of tokens/strings to be deserialized into an object.

• world (World) – The simulated world.

• stack (list, optional) – The stack is used for parsing the tokens.

Raises:

Exception – An exception is raised if there is an error during the parsing.

Returns:

last_sent – The parsed sentence.

Return type:

Sentence

pydialogue.baseline.serializers.serialize(elem, world)

Serializes an object into a list of tokens.

The serialization is done in such a way that only the meaning of the sentence is represented and the syntactic structure is ignored.

For each object type, the serialization is done as follows:

• Sentence: serializes the describers that represent the semantics.

• Describer: serializes the arguments in the following manner:

  b<ArgKey> <serialized argument value> e<ArgKey>

• set or list: serializes each of the elements recursively and uses bset/eset or blist/elist to indicate the beginning or end of the serialization.

• Entity: serializes the Entity’s description.

• all other: adds the element in the list of serialized objects.

For the following sentence: “Hans has the red apple and the chicken.” the sentence object can be created using sentences.have(world.player, “has”, None, [world.small_apple, world.chicken]) The sentence object will be serialized as a list of tokens:

['<sent>',
  '<desc>',
    '<Arg-PAG>', '<entity>', 'small', 'person', '</entity>', '</Arg-PAG>',
    '<Rel>', 'has', '</Rel>',
    '<Arg-PPT>', '<set>', '<entity>', 'red', 'apple', '</entity>', '<entity>', 'chicken', '</entity>', '</set>',
    '</Arg-PPT>',
  '</desc>',
 '</sent>']

The indents are there just to better visualize the serialization.

Parameters:

• elem (any) – The object to be serialized.

• world (World) – The simulated world.

Returns:

cmds – The serialized object.

Return type:

list of strings

Module contents