A collection of dialogue templates inspired by early language development.
Templates Description
This pack evaluates language understanding and reasoning in a simulated
textual environment. The tasks involve following simple instructions,
understanding entity descriptions, and reasoning about the roles and
relationships described in the dialogue.
The developmental framing is inspired by receptive-language abilities
associated with children between 36 and 48 months and is used as a design
reference rather than as a measure of children's actual language abilities.
The pack consists of five templates that generate task-oriented dialogues:
GoDirection, GoLocation, LookItem, DropItem, and GetItem.
The templates are integrated into our library and the source code can be
found here.
Why this is challenging
Although the individual tasks are simple, solving them can require several
capabilities at the same time. In particular, the dialogues may require
in-context continual learning, in-context social learning, and multi-hop
reasoning.
The environment evolves as the dialogue progresses, so information that was
correct earlier may no longer describe the current state. Models must
therefore identify which information remains relevant, use information from
other agents, and reason over the resulting context.
Our experiments suggest that combining these requirements can be challenging
for current language models. The benchmark is intended to help investigate
these challenges in a controlled, reproducible setting.
Example Dialogue
System:
I will provide you with a context, a log of previous responses from players in an imaginary textual world. Each sentence is an action taken by a player or feedback from the environment. Your task is to determine the appropriate next response for the correct player based on the latest prompt in the context.
...
Output your final response exactly once inside <next_response> tags.
Do not output anything before, after, or outside these tags. Stop generating
more text after </next_response>.
...
User:
Hannah says: Hannah, go east.
Hannah tries going east.
Hannah goes east from the bedroom.
Hannah looks in the kitchen.
Hannah sees the wooden container and the red table.
Andy's type is cat.
Heidi's size is medium.
Andy says: big person, go south.
the big person goes south from the bedroom.
the big person looks in the living room.
the big person sees the rug, the medium blue table, and the small carrot.
Hannah says: Heidi, go west
Heidi tries going west.
Heidi can not go west. the white medium place has no direction west.
Heidi says: Heidi can not go west. the white medium place has no direction west.
Heidi's type is person.
Andy says: the medium person, go north
the medium person tries going north
the medium person goes north from the white medium place.
the medium person looks in the bedroom.
the medium person sees the medium carrot, the small blue table, and the toy.
Andy says: medium person, go to the medium table.
Try it yourself
What should the model reply based on the dialogue above?
Correct response:
<next_response>The medium person tries going south.</next_response>