Label taxonomy¶

A task in state/tasks/<benchmark>.yml carries two sets of labels:

Display tags (display_tags:, vocabulary state/display_tags.yml), in two groups: Capability, what the agent has to be able to do, and Task Domain, what kind of task it is. They are what the site shows. Every task page and task list shows a task's Capability tags first, then its Task Domain tags, each group in its own colour, and the task list filters by both. A tagged task carries at least one tag from each group.
Detailed labels (labels:, vocabulary state/taxonomy.yml): eight facets, each a question you ask about a task, listed below the tags. They are kept as they are, for analysis and scripts/suggest_labels.py, and are not shown on task pages.

9 display tags in 2 groups; 45 detailed labels across 8 facets.

Edit the display tags in place below (with make edit), or in state/display_tags.yml; edit the detailed labels in state/taxonomy.yml.

Capability¶

What the agent has to be able to do to solve the task. Take every one the goal requires.

Perception & Understanding¶

perception-understanding

Perception, object/state recognition, semantic grounding, spatial understanding.

Planning & Reasoning¶

planning-reasoning

Task planning, sequencing, tool/affordance reasoning, geometric reasoning.

Rolls up BEHAVIOR skills: ignite, chop

Tagged on 435 task(s): Acrylic · Applesoranges, Acrylic · Bedroom, Acrylic · Bluehorses, Acrylic · Greatwave, Acrylic · Kajikazawa, Acrylic · Plumpark, Acrylic · Redfuji, Acrylic · Suddenshower, Acrylic · Sunflowers, Acrylic · Vision, Align Blocks, Align Chopsticks, Aloha Hand Over, Aloha Single Peg Insertion, Animals In Bin, Apple And Yogurt In Bowl, Assembling Gift Baskets, Attach a Camera to a Tripod, Bagels On Plate, Balance Board, Ball Into Bottle, Ball Into Jar, Ball Onto Tower, Banana Then Rubiks Cube, Bananas In Bin One More, Bananas In Bin Three Total, Bananas In Crate, Bananas Out Of Bin, Bbq Sauce In Bin, Beat block hammer, Black Items In Bin, Block Stacking Order Agnostic, Block Stacking Specified Order, Blocks In Bin, Blocks ranking rgb, Blocks ranking size, Blue brush · sweep forward, Blue brush · sweep right, Boxing Books Up for Storage, Boxing Food After Dinner, Bringing Paper To Recycling, Bringing Water, Bringing in Wood, Can Meat, Canning Food, Carrying Out Garden Furniture, Carrying in Groceries, Chop an Onion, Chopping Wood, Claw hammer · swing down, Claw hammer · swing side, Clean A Keyboard, Clean Boxing Gloves, Clean Up Broken Glass, Clean Up Toys, Clean Up Your Desk, Clean Your Rusty Garden Tools, Clean a Patio, Clean a Trumpet, Cleaning Up Branches And Twigs, Cleaning Up Plates and Food, Clear Organic Objects, Clearing Food from Table into Fridge, Clutter Plastic, Clutter Pumpkin, Collect Screws, Collecting Aluminum Cans, Collecting Children's Toys, Common sense · select_billiards_common_sense, Common sense · select_chemistry_tube_common_sense, Common sense · select_fruit_common_sense, Composite · AddLemonToFish, Composite · AssembleCookingArray, Composite · BeverageSorting, Composite · BreadAndCheese, Composite · CandleCleanup, Composite · CerealAndBowl, Composite · ChooseMeasuringCup, Composite · ClearFoodWaste, Composite · ClearSink, Composite · CookieDoughPrep, Composite · DeliverBrewedCoffee, Composite · DessertUpgrade, Composite · DisplayMeatVariety, Composite · DivideBasins, Composite · DrainVeggies, Composite · DrinkwareConsolidation, Composite · MakeIceLemonade, Composite · MakeLoadedPotato, Composite · MealPrepStaging, Composite · MicrowaveDefrostMeat, Composite · OrganizeCoffeeCondiments, Composite · OvenBroilFish, Composite · PlateSteakMeal, Composite · PortionInTupperware, Composite · PreRinseStation, Composite · PrepForSanitizing, Composite · PrepMarinatingMeat, Composite · ReheatMeal, Composite · RetrieveMeat, Composite · SanitizePrepCuttingBoard, Composite · ScrubBowl, Composite · SearingMeat, Composite · SeasoningSpiceSetup, Composite · SetUpCuttingStation, Composite · StockingBreakfastFoods, Composite · SweetenCoffee, Composite · ThawInSink, Composite · TongBuffetSetup, Composting Waste, Condiments In Bin, Cook A Brisket, Cook A Frozen Pie, Cook Bacon, Cook Broccolini, Cook Brussels Sprouts, Cook Cabbage, Cook Hot Dogs, Cooking Clear Plate, Cooking Pick Pasta Tool, Cover With Lid, Cubes And Blocks In Bin, Cylinder Through Hole, Differentiate Cubes, Dishes In Bin, Dispose Of Batteries, Dispose Of Glass, Dominos, Dump bin big bin, Dynamic3D · Balance beam, Dynamic3D · Constrained cupboard, Dynamic3D · Dynamo, Dynamic3D · Scoop pour, Dynamic3D · Shelf, Dynamic3D · Sort blocks, Dynamic3D · Sweep into drawer, Dynamic3D · Sweep simple, Dynamic3D · Tossing, Electronics In Bin, Flat eraser · wipe c, Flat eraser · wipe smile, Flat spatula · flip over, Flat spatula · serve plate, Food Packing By Color, Food Packing2 Boxes, Food Packing2 Cans, Food Packing3 Boxes, Food Packing3 Cans, Freeze Fruit, Freeze Pies, Fruits Green Limes On Plate, Fruits Moving, Fruits Moving Orange Or Lime, Fruits On Plate, Fruits On Plate3, Fruits Onion To Plate, Fruits Oranges On Plate, GR1 · PnPBottleToCabinetClose, GR1 · PnPCanToDrawerClose, GR1 · PnPCupToDrawerClose, GR1 · PnPMilkToMicrowaveClose, GR1 · PnPPotatoToMicrowaveClose, GR1 · PnPWineToCabinetClose, Gap Retrieve, Getting Organized For Work, Green Spoons In Pot, Halve An Egg, Hammers In Left Bin, Handle eraser · wipe c, Handle eraser · wipe smile, Handover block, Handover microphone, Hanging mug, Hiding Easter Eggs, Hold Cup, Installing A Fax Machine, Installing A Modem, Installing A Scanner, Installing Smoke Detectors, Jugs On Shelf, Kaishu · Ai, Kaishu · Chun, Kaishu · Da, Kaishu · Dao, Kaishu · De, Kaishu · Feng, Kaishu · Fu, Kaishu · Guo, Kaishu · He, Kaishu · Houdezaiwu, Kaishu · Hua, Kaishu · Jia, Kaishu · Jiang, Kaishu · Jing, Kaishu · Kou, Kaishu · Le, Kaishu · Lin, Kaishu · Long, Kaishu · Ma, Kaishu · Ming, Kaishu · Ningjingzhiyuan, Kaishu · Qing, Kaishu · Ran, Kaishu · Ren, Kaishu · Shan, Kaishu · Shu, Kaishu · Shui, Kaishu · Tian, Kaishu · Tiandaochouqin, Kaishu · Xin, Kaishu · Yong, Kaishu · Yongzibafa, Kaishu · Yue, Kaishu · Zhong, Kaishu · Zi, Kinematic3D · Obstruction 3D, Kinematic3D · Packing 3D, Laying Tile Floors, Lettering · Donum, Lettering · Inunum, Lettering · Minim, Lettering · Minimum, Lettering · Minium, Lettering · Moon, Lettering · Nimium, Lettering · Nun, Lettering · Omnium, Lettering · Unpin, Line · Butterfly, Line · City, Line · Cottage, Line · Fish, Line · Flower, Line · Landscape, Line · Rainbow, Line · Rocket, Line · Sailboat, Loading The Car, Locomotion · Maze, Long screwdriver · spin horizontal, Long screwdriver · spin vertical, Make Cabinet Doors, Make Gift Bags For Baby Showers, Make Microwave Popcorn, Make Pizza, Make Rose Centerpieces, Mallet hammer · swing down, Mallet hammer · swing side, Manipulation · Bookshelf simple, Manipulation · Cabinet, Manipulation · Door, Manipulation · Kitchen, Manipulation · Package, Manipulation · Room, Manipulation · Spoon, Manipulation · Truck, Manipulation · Window, Manipulation · book_rearrange, Manipulation · find_unseen_object, Manipulation · hammer_nail_and_hang_picture, Manipulation · heat_food, Manipulation · play_math_game, Manipulation · select_nth_largest_poker, Manipulation · select_painting, Manipulation · select_unique_type_mahjong, Manipulation · take_out_cool_drink, Manipulation · texas_holdem, Meta-World · Button press topdown wall, Meta-World · Button press wall, Meta-World · Hammer, Meta-World · Pick place wall, Meta-World · Push wall, Meta-World · Reach wall, Meta-World · Stick pull, Meta-World · Stick push, Move Banana To Bagel Plate, Move Cube, Moving Boxes To Storage, Multi-stage · ArrangeVegetables, Multi-stage · MicrowaveThawing, Multi-stage · PreSoakPan, Multi-stage · PrepareCoffee, Multi-stage · RestockPantry, Non Hammer Tools In Right Bin, Oil · Bathers, Oil · Cafeterrace, Oil · Cypresses, Oil · Irises, Oil · Milkmaid, Oil · Monalisa, Oil · Pearlearring, Oil · Saintevictoire, Oil · Starrynight, Oil · Temeraire, Oil · Wanderer, One Bottle In Square Pail, One Bottle On Shelf, Open microwave, Organizing Art Supplies, Organizing School Stuff, Outfit A Basic Toolbox, Packing Meal For Delivery, Phone Or Remote In Bin, Physical QA · friction_qa, Physical QA · magnetism_qa, Physical QA · reflection_qa, Picking Up Toys, Picking Up Trash, Pinch Card, Place Book, Place Tall Box, Place bread basket, Place burger fries, Place can basket, Place cans plastic box, Place dual shoes, Place object basket, Plastic Bottles In Square Pail, Polishing Shoes, Preparing Lunch Box, Put Bowl On Shelf Top, Put Mugs On Shelf, Put Together A Basic Pruning Kit, Put Two Mugs On Shelf, Put bottles dustbin, Put object cabinet, Putting Away Halloween Decorations, Putting Away Toys, Putting Dirty Dishes In Sink, Putting Dishes Away After Cleaning, Putting Shoes On Rack, Putting Up Christmas Decorations Inside, Raise Platform, Re Shelving Library Books, Rearrange Your Room, Rearranging Kitchen Furniture, Recycle Carton, Recycle Cartons On Box, Recycle Cartons Vertical Crate, Red Dishes In Bin, Red Items In Bin, Red brush · sweep forward, Red brush · sweep right, Reorient All Mugs, Reorient White Mugs, Retrieve Cube, Retrieve Roll, Roll Up Ball, Round Dough Sheet, Rubiks Cube And Banana, Rubiks Cube Or Banana, Rubiks Cube Then Banana, Rubiks Cubes In Bin, Scan object, Scrubbing Bathroom Floor, Seal Colander, Semantic · select_chemistry_tube_semantic, Semantic · select_mahjong_semantic, Semantic · select_poker_semantic, Separate Marbles And Sand, Set Up A Coffee Station In Your Kitchen, Setting Mousetraps, Setting The Fire, Setting The Table, Setup A Bar For A Cocktail Party, Sharpie marker · draw smile, Sharpie marker · write c, Short screwdriver · spin horizontal, Short screwdriver · spin vertical, Slicing Vegetables, Sorting Books On Shelf, Sorting Bottles Cans And Paper, Sorting Household Items, Sorting Vegetables, Spoon spatula · flip over, Spoon spatula · serve plate, Spoons In Pot, Spraying For Bugs, Spraying Fruit Trees, Stabilize Bottle, Stack Bowls, Stack Cubes, Stack White Mugs, Stack Yellow On Red, Stack blocks three, Stack blocks two, Stack bowls three, Stack bowls two, Stack3 Rubiks Cube, Stacking Wood, Stamp seal, Stand Bulb, Stand Pages, Staples marker · draw smile, Staples marker · write c, Store Batteries, Store Produce, Storing Food, Sweeping Garage, Take Mugs Off Of Shelf, Take Spatula Off Shelf, Thawing Frozen Food, Throw Away Snacks, Tidying Bathroom, Tidying Bedroom, Tidying Living Room, Tool Organization, Tool Organization Both, Tools Picking All Hammers, Turning Out All Lights Before Sleep, Unloading The Car, Unstack Rubiks Cube, Utensils In Mug, Vacuuming Floors, Wash A Baseball Cap, Wash Dog Toys, Water Into Mug, White Mugs In Bin, Xingshu · Chun, Xingshu · Feng, Xingshu · He, Xingshu · Huifenghechang, Xingshu · Jian, Xingshu · Maolinxiuzhu, Xingshu · Nian, Xingshu · Ren, Xingshu · Shan, Xingshu · Tianlangqiqing, Xingshu · Tong, Xingshu · Xian, Xingshu · Xin, Xingshu · Yong, Xingshu · Yu, Xingshu · Zhi, Yellow And White Objects In Bin

Control & Coordination¶

control-coordination

Manipulation/control, contact-rich execution, locomotion control, multi-arm or whole-body coordination.

Rolls up BEHAVIOR skills: hand over, pick up from, place in, place on, place under, place in next to, place on next to, hold, release, push to, sweep surface, tip over, open door, open drawer, open lid, close door, close drawer, close lid, insert, attach, hang, press, wipe hard, pour, turn on switch, turn off switch, turn to, ignite, chop, spray

Feedback & Adaptation¶

feedback-adaptation

Closed-loop correction, failure recovery, online adaptation, exploration based on feedback.

Tagged on 69 task(s): Assembling Gift Baskets, Boxing Books Up for Storage, Boxing Food After Dinner, Bringing Paper To Recycling, Bringing Water, Can Meat, Canning Food, Carrying in Groceries, Chopping Wood, Clean Up Broken Glass, Clean Up Toys, Clean Up Your Desk, Clean Your Rusty Garden Tools, Clear Organic Objects, Collecting Aluminum Cans, Collecting Children's Toys, Composting Waste, Cook A Brisket, Cook A Frozen Pie, Cook Bacon, Cook Brussels Sprouts, Cook Cabbage, Cook Hot Dogs, Cubes And Blocks In Bin, Differentiate Cubes, Dispose Of Batteries, Dispose Of Glass, Dynamic3D · Dynamo, Freeze Fruit, Freeze Pies, Fruits On Plate, Getting Organized For Work, Halve An Egg, Loading The Car, Locomotion · Balance hard, Locomotion · Balance simple, Locomotion · Sit hard, Locomotion · Sit simple, Make Gift Bags For Baby Showers, Make Pizza, Manipulation · Basketball, Manipulation · Highbar simple, Manipulation · find_unseen_object, Moving Boxes To Storage, Organizing School Stuff, Outfit A Basic Toolbox, Picking Up Toys, Picking Up Trash, Preparing Lunch Box, Put bottles dustbin, Putting Away Toys, Putting Dishes Away After Cleaning, Putting Up Christmas Decorations Inside, RB-Y1 · Navigate to, Set Up A Coffee Station In Your Kitchen, Setting The Table, Setup A Bar For A Cocktail Party, Slicing Vegetables, Sorting Books On Shelf, Sorting Bottles Cans And Paper, Sorting Household Items, Sorting Vegetables, Store Batteries, Store Honey, Storing Food, Thawing Frozen Food, Turning Out All Lights Before Sleep, Unloading The Car, Wash Dog Toys

Task Domain¶

What kind of task it is: where the robot works and what its body has to do. Take every one that applies.

Manipulation¶

manipulation

Fixed-base object manipulation, grasping, pushing, insertion, tool use, articulated/deformable object interaction.

Locomotion & Stability¶

locomotion-stability

Walking, running, balancing, terrain traversal, dynamic stabilization.

Tagged on 9 task(s): Locomotion · Balance hard, Locomotion · Balance simple, Locomotion · Hurdle, Locomotion · Maze, Locomotion · Reach, Locomotion · Sit hard, Locomotion · Sit simple, Locomotion · Stair, Locomotion · Walk

navigation-exploration

Navigation, target search, exploration, spatial memory.

Tagged on 41 task(s): Boxing Food After Dinner, Bringing Paper To Recycling, Bringing Water, Bringing in Wood, Carrying Out Garden Furniture, Carrying in Groceries, Chopping Wood, Clean Up Broken Glass, Clean Your Rusty Garden Tools, Clean a Patio, Cleaning Up Branches And Twigs, Composting Waste, Cook Brussels Sprouts, Dispose Of Batteries, Dispose Of Glass, Dynamic3D · Dynamo, Hiding Easter Eggs, Installing A Fax Machine, Installing A Modem, Installing Smoke Detectors, Laying Tile Floors, Loading The Car, Locomotion · Maze, Make Cabinet Doors, Manipulation · Door, Moving Boxes To Storage, Picking Up Trash, Putting Away Toys, Putting Dirty Dishes In Sink, Putting Up Christmas Decorations Inside, RB-Y1 · Navigate to, Rearrange Your Room, Setting The Table, Sorting Household Items, Spraying For Bugs, Spraying Fruit Trees, Stacking Wood, Store Batteries, Store Honey, Turning Out All Lights Before Sleep, Unloading The Car

Mobile / Whole-body Manipulation¶

mobile-whole-body-manipulation

Tasks combining navigation/locomotion with manipulation or requiring coordinated whole-body motion.

Rolls up BEHAVIOR skills: move to

Tagged on 151 task(s): Assembling Gift Baskets, Attach a Camera to a Tripod, Boxing Books Up for Storage, Boxing Food After Dinner, Bringing Paper To Recycling, Bringing Water, Bringing in Wood, Can Meat, Canning Food, Carrying Out Garden Furniture, Carrying in Groceries, Chop an Onion, Chopping Wood, Clean A Keyboard, Clean Boxing Gloves, Clean Up Broken Glass, Clean Up Your Desk, Clean Your Rusty Garden Tools, Clean a Patio, Clean a Trumpet, Cleaning Up Branches And Twigs, Cleaning Up Plates and Food, Clearing Food from Table into Fridge, Collecting Aluminum Cans, Collecting Children's Toys, Composite · AddLemonToFish, Composite · AssembleCookingArray, Composite · BeverageSorting, Composite · CandleCleanup, Composite · CookieDoughPrep, Composite · DeliverBrewedCoffee, Composite · DisplayMeatVariety, Composite · DrinkwareConsolidation, Composite · MakeIceLemonade, Composite · MakeLoadedPotato, Composite · MealPrepStaging, Composite · MicrowaveDefrostMeat, Composite · PlateSteakMeal, Composite · ReheatMeal, Composite · RetrieveMeat, Composite · SanitizePrepCuttingBoard, Composite · SearingMeat, Composite · SeasoningSpiceSetup, Composite · SetUpCuttingStation, Composite · StockingBreakfastFoods, Composite · SweetenCoffee, Composite · TongBuffetSetup, Composting Waste, Cook A Brisket, Cook A Frozen Pie, Cook Bacon, Cook Broccolini, Cook Brussels Sprouts, Cook Cabbage, Cook Hot Dogs, Dispose Of Batteries, Dispose Of Glass, Dynamic3D · Balance beam, Dynamic3D · Constrained cupboard, Dynamic3D · Dynamo, Dynamic3D · Scoop pour, Dynamic3D · Shelf, Dynamic3D · Sort blocks, Dynamic3D · Sweep into drawer, Dynamic3D · Sweep simple, Dynamic3D · Tossing, Freeze Fruit, Freeze Pies, Getting Organized For Work, Halve An Egg, Hanging Pictures, Hiding Easter Eggs, Installing A Fax Machine, Installing A Modem, Installing A Scanner, Installing Smoke Detectors, Kinematic3D · Obstruction 3D, Kinematic3D · Packing 3D, Laying Tile Floors, Loading The Car, Locomotion · Reach, Locomotion · Sit hard, Make Cabinet Doors, Make Gift Bags For Baby Showers, Make Microwave Popcorn, Make Pizza, Make Rose Centerpieces, Manipulation · Basketball, Manipulation · Bookshelf simple, Manipulation · Cabinet, Manipulation · Cube, Manipulation · Door, Manipulation · Highbar simple, Manipulation · Insert normal, Manipulation · Kitchen, Manipulation · Package, Manipulation · Powerlift, Manipulation · Push, Manipulation · Room, Manipulation · Spoon, Manipulation · Truck, Manipulation · Window, Moving Boxes To Storage, Organizing Art Supplies, Organizing School Stuff, Outfit A Basic Toolbox, Packing Meal For Delivery, Picking Up Toys, Picking Up Trash, Polishing Shoes, Preparing Lunch Box, Put Together A Basic Pruning Kit, Putting Away Halloween Decorations, Putting Away Toys, Putting Dirty Dishes In Sink, Putting Dishes Away After Cleaning, Putting Shoes On Rack, Putting Up Christmas Decorations Inside, RB-Y1 · Open door, Re Shelving Library Books, Rearrange Your Room, Rearranging Kitchen Furniture, Scrubbing Bathroom Floor, Set Up A Coffee Station In Your Kitchen, Setting Mousetraps, Setting The Fire, Setting The Table, Setup A Bar For A Cocktail Party, Slicing Vegetables, Sorting Books On Shelf, Sorting Bottles Cans And Paper, Sorting Household Items, Sorting Vegetables, Spraying For Bugs, Spraying Fruit Trees, Stacking Wood, Store Batteries, Store Honey, Store Produce, Storing Food, Sweeping Garage, Thawing Frozen Food, Tidying Bathroom, Tidying Bedroom, Tidying Living Room, Turning On Radio, Turning Out All Lights Before Sleep, Unloading The Car, Vacuuming Floors, Wash A Baseball Cap, Wash Dog Toys

Interaction & Collaboration¶

interaction-collaboration

Human-robot interaction, multi-robot collaboration, handover, cooperative tasks.

Tagged on 3 task(s): Aloha Hand Over, Handover block, Handover microphone

Detailed labels¶

The shared vocabulary for the labels: field in state/tasks/<benchmark>.yml, written to span benchmarks rather than describe one. Each facet below is a question you ask about a task; the labels under it are the answers. scripts/migrate_tag_groups.py derived the first display tags from them.

Where these came from, and how settled they are

Each benchmark publishes its own idea of capability and they do not agree: BEHAVIOR a formal goal language and an object ontology, RoboLab an attribute list about how the goal is worded, RoboWits a paragraph of prose per task, HumanoidBench nothing but a reward function. Those are theirs. The labels below are ours, fitted to fourteen suites — see the benchmark landscape — under one rule: a label earns its place only if knowing it changes what you predict about the task.

Where a label can be derived from something a benchmark already publishes, state/taxonomy.yml records the rule under derived_from, so the derivation is inspectable and scripts/suggest_labels.py can propose labels for a task instead of making anyone read the whole list. Labels showing a skill roll-up map onto BEHAVIOR's own 31 primitives.

Treat this as a starting point to argue with, not a settled standard — it has not been through a group discussion. Several labels are reached by nothing we have today: they are the axes a suite with legs, hands or people in it would occupy, and they are here so that adding one does not mean rewriting the vocabulary.

Scale¶

How far the robot has to go. Navigation is not a separate label because every rung above tabletop already implies it. — ladder: take the highest rung that applies

Tabletop¶

scale-tabletop

Everything is inside one workspace and the base never moves. The floor of the ladder, and a real finding when a whole suite sits here.

Why it exists: LIBERO, Meta-World+, RoboLab, RoboWits, RoboCasa-GR1, VLABench and most of RoboTwin 2.0 never move a base — about half the surveyed corpus.

Tagged on 234 task(s): Adjust bottle, Aero Cube Rotate Z Axis, Align Blocks, Align Chopsticks, Aloha Hand Over, Aloha Single Peg Insertion, Animals In Bin, Apple And Yogurt In Bowl, Bagels On Plate, Balance Board, Ball Into Bottle, Ball Into Jar, Ball Onto Tower, Banana In Bowl, Banana On Plate, Banana Then Rubiks Cube, Bananas In Bin One More, Bananas In Bin Three Total, Bananas In Crate, Bananas Out Of Bin, Bbq Sauce In Bin, Beat block hammer, Big Pumpkin In Bin, Black Items In Bin, Block Stacking Order Agnostic, Block Stacking Specified Order, Blocks In Bin, Blocks ranking rgb, Blocks ranking size, Blue brush · sweep forward, Blue brush · sweep right, Bowl In Bin, Bowl Stacking Left On Right, Bowl Stacking Right On Left, Butter Above Raisin, Canned Food In Bin, Clamp In Right Bin, Claw hammer · swing down, Claw hammer · swing side, Clean Up Toys, Clear Organic Objects, Click alarm clock, Click bell, Clutter Plastic, Clutter Pumpkin, Coffee Pot In Bin, Collect Screws, Condiments In Bin, Cooking Clear Plate, Cooking Pick Pasta Tool, Cover With Lid, Cubes And Blocks In Bin, Cylinder Through Hole, Differentiate Cubes, Dishes In Bin, Dominos, Dump bin big bin, Electronics In Bin, Flat eraser · wipe c, Flat eraser · wipe smile, Flat spatula · flip over, Flat spatula · serve plate, Food Packing By Color, Food Packing1 Boxes, Food Packing1 Cans, Food Packing2 Boxes, Food Packing2 Cans, Food Packing3 Boxes, Food Packing3 Cans, Fruits Green Limes On Plate, Fruits Moving, Fruits Moving Orange Or Lime, Fruits On Plate, Fruits On Plate3, Fruits Onion, Fruits Onion To Plate, Fruits Oranges On Plate, Gap Retrieve, Grab A Bagel, Grab A Fruit, Grab roller, Green Spoons In Pot, Hammers In Left Bin, Handle eraser · wipe c, Handle eraser · wipe smile, Handover block, Handover microphone, Hanging mug, Hold Cup, Jugs On Shelf, Keyboard Out Of Bin, Larger Object Raisin Box In Bin, Leap Cube Reorient, Leap Cube Rotate Z Axis, Lift pot, Long screwdriver · spin horizontal, Long screwdriver · spin vertical, Mallet hammer · swing down, Mallet hammer · swing side, Marker In Mug, Mouse On Keyboard, Move Banana To Bagel Plate, Move Cube, Move can pot, Move pill bottle pad, Move playing card away, Move stapler pad, Mustard Above Raisin, Mustard In Left Bin, Mustard In Right Bin, Non Hammer Tools In Right Bin, One Bottle In Square Pail, One Bottle On Shelf, Open laptop, Open microwave, Panda Open Cabinet, Panda Pick Cube, Panda Pick Cube Cartesian, Panda Pick Cube Orientation, Panda Robotiq Push Cube, Phone Or Remote In Bin, Pick Drill, Pick Glasses, Pick Orange Object, Pick Up Blue Pitcher, Pick Up Green Object, Pick diverse bottles, Pick dual bottles, Pinch Card, Pink Spoon In Pot, Place Book, Place Tall Box, Place a to b left, Place a to b right, Place bread basket, Place bread skillet, Place burger fries, Place can basket, Place cans plastic box, Place container plate, Place dual shoes, Place empty cup, Place fan, Place mouse pad, Place object basket, Place object scale, Place object stand, Place phone stand, Place shoe, Plastic Bottles In Square Pail, Press stapler, Put Bowl On Shelf Top, Put Mugs On Shelf, Put Two Mugs On Shelf, Put bottles dustbin, Put object cabinet, Raise Platform, Recycle Carton, Recycle Cartons On Box, Recycle Cartons Vertical Crate, Red Dishes In Bin, Red Items In Bin, Red brush · sweep forward, Red brush · sweep right, Reorient All Mugs, Reorient Jug, Reorient Red Mug, Reorient White Mugs, Retrieve Cube, Retrieve Roll, Roll Up Ball, Rotate QR code, Round Dough Sheet, Rubiks Cube, Rubiks Cube And Banana, Rubiks Cube Behind Bowl, Rubiks Cube In Front Of Bowl, Rubiks Cube Left Of Bowl, Rubiks Cube Or Banana, Rubiks Cube Right Of Bowl, Rubiks Cube Then Banana, Rubiks Cubes In Bin, Sauce Bottles Crate, Scan object, Seal Colander, Separate Marbles And Sand, Shake bottle, Shake bottle horizontally, Sharpie marker · draw smile, Sharpie marker · write c, Short screwdriver · spin horizontal, Short screwdriver · spin vertical, Small Pumpkin In Bin, Smaller Object Butter In Bin, Smartphone In Bin, Spoon In Mug, Spoon spatula · flip over, Spoon spatula · serve plate, Spoons In Pot, Stabilize Bottle, Stack Bowls, Stack Cubes, Stack White Mugs, Stack Yellow On Red, Stack blocks three, Stack blocks two, Stack bowls three, Stack bowls two, Stack3 Rubiks Cube, Stamp seal, Stand Bulb, Stand Pages, Staples marker · draw smile, Staples marker · write c, Take Measuring Spoon Out, Take Mugs Off Of Shelf, Take Spatula Off Shelf, Throw Away Apple, Throw Away Snacks, Tool Organization, Tool Organization Both, Tools Picking All Hammers, Tools Picking Drill, Tools Picking Hammer, Toy In Bin, Turn switch, Unstack Rubiks Cube, Utensils In Mug, Water Into Mug, White Mug In Center Of Table, White Mugs In Bin, Wood Spatula To Bowl, Yellow And White Objects In Bin, Yogurt In Bowl

Room¶

scale-room

The base must move within one room or workspace to reach what the task needs.

Why it exists: RoboCasa's kitchens; RoboCasa365's NavigateKitchen; HumanoidBench maze.

Rolls up BEHAVIOR skills: move to

Tagged on 63 task(s): Assembling Gift Baskets, Attach a Camera to a Tripod, Boxing Books Up for Storage, Can Meat, Canning Food, Chop an Onion, Clean A Keyboard, Clean Boxing Gloves, Clean Up Your Desk, Clean a Trumpet, Cleaning Up Plates and Food, Clearing Food from Table into Fridge, Collecting Aluminum Cans, Collecting Children's Toys, Cook A Brisket, Cook A Frozen Pie, Cook Bacon, Cook Broccolini, Cook Cabbage, Cook Hot Dogs, Freeze Fruit, Freeze Pies, Getting Organized For Work, Halve An Egg, Hanging Pictures, Installing A Scanner, Make Gift Bags For Baby Showers, Make Microwave Popcorn, Make Pizza, Make Rose Centerpieces, Organizing Art Supplies, Organizing School Stuff, Outfit A Basic Toolbox, Packing Meal For Delivery, Picking Up Toys, Polishing Shoes, Preparing Lunch Box, Put Together A Basic Pruning Kit, Putting Away Halloween Decorations, Putting Dishes Away After Cleaning, Putting Shoes On Rack, Re Shelving Library Books, Rearranging Kitchen Furniture, Scrubbing Bathroom Floor, Set Up A Coffee Station In Your Kitchen, Setting Mousetraps, Setting The Fire, Setup A Bar For A Cocktail Party, Slicing Vegetables, Sorting Books On Shelf, Sorting Bottles Cans And Paper, Sorting Vegetables, Store Produce, Storing Food, Sweeping Garage, Thawing Frozen Food, Tidying Bathroom, Tidying Bedroom, Tidying Living Room, Turning On Radio, Vacuuming Floors, Wash A Baseball Cap, Wash Dog Toys

Building¶

scale-building

The task spans rooms or floors, so the layout has to be traversed.

Why it exists: BEHAVIOR picking_up_trash — cans in the living room, bin in the kitchen; 29 of its 100 tasks span rooms.

Tagged on 25 task(s): Boxing Food After Dinner, Bringing Water, Carrying in Groceries, Clean Up Broken Glass, Composting Waste, Cook Brussels Sprouts, Dispose Of Batteries, Dispose Of Glass, Installing A Fax Machine, Installing A Modem, Laying Tile Floors, Loading The Car, Make Cabinet Doors, Moving Boxes To Storage, Picking Up Trash, Putting Away Toys, Putting Dirty Dishes In Sink, Putting Up Christmas Decorations Inside, Rearrange Your Room, Setting The Table, Sorting Household Items, Store Batteries, Store Honey, Turning Out All Lights Before Sleep, Unloading The Car

Outdoor¶

scale-outdoor

Part of the task happens outdoors or in unstructured space.

Why it exists: BEHAVIOR's garden and patio activities; the legged suites' outdoor terrain.

Tagged on 12 task(s): Bringing Paper To Recycling, Bringing in Wood, Carrying Out Garden Furniture, Chopping Wood, Clean Your Rusty Garden Tools, Clean a Patio, Cleaning Up Branches And Twigs, Hiding Easter Eggs, Installing Smoke Detectors, Spraying For Bugs, Spraying Fruit Trees, Stacking Wood

Body¶

What the body must do beyond moving a single arm. Tag only what the task cannot be solved without.

Legged locomotion¶

legged-locomotion

Moving is itself the control problem — a gait to maintain, a commanded velocity to track, balance to keep while moving. A wheeled base on a flat floor does not qualify.

Why it exists: MuJoCo Playground's joystick suites on Go1, Spot, Barkour, G1, H1, T1, Apollo, Op3 and Berkeley Humanoid; HumanoidBench walk, stand, run.

Not yet tagged on any task.

Rough terrain¶

rough-terrain

Getting there can fail, not just take time — stairs, slopes, gaps, low passages, obstacles that must be climbed or crawled under.

Why it exists: Playground's RoughTerrain variants; HumanoidBench stair, hurdle, crawl, slide, pole.

Not yet tagged on any task.

Dynamic balance¶

dynamic-balance

Staying upright is at risk, or getting back up is the task — recovery from a fall, a seat or a slip, and acrobatics where momentum has to be managed.

Why it exists: Playground Go1Getup, SpotGetup, Go1Handstand, Go1Footstand; HumanoidBench balance, sit, stand, highbar.

Not yet tagged on any task.

Whole-body coordination¶

whole-body

Legs or base, torso and arms must act as one — to reach past the arm's workspace, brace against a load, lift something heavy, lean or squat.

Why it exists: HumanoidBench's fifteen whole-body manipulation tasks (truck unloading, powerlift, door, bookshelf); RoboCasa365's mobile manipulation.

Tagged on 3 task(s): Getting Organized For Work, Setting Mousetraps, Sorting Household Items

Bimanual¶

bimanual

Two end effectors must cooperate on the same object or motion — hold and open, hand over, lift together, carry what one hand cannot.

Why it exists: RoboTwin 2.0 Handover Block, Handover Mic, Lift Pot, Pick Dual Bottles; Playground AlohaHandOver and AlohaSinglePegInsertion. Not RoboWits by default: our own reference solution for stack_cubes used one arm.

Rolls up BEHAVIOR skills: hand over

Tagged on 27 task(s): Aloha Hand Over, Aloha Single Peg Insertion, Blocks ranking size, Bringing in Wood, Carrying Out Garden Furniture, Dump bin big bin, Grab roller, Handover block, Handover microphone, Hanging mug, Lift pot, Moving Boxes To Storage, Pick diverse bottles, Pick dual bottles, Place bread basket, Place burger fries, Place can basket, Place cans plastic box, Place dual shoes, Place object basket, Put bottles dustbin, Put object cabinet, Scan object, Stabilize Bottle, Stack blocks three, Stack blocks two, Stand Pages

In-hand manipulation¶

in-hand

Multi-finger control a parallel jaw cannot do — reorient in hand, regrasp a tool into the pose its use needs, move fingers independently. A five-finger hand picking up a cup does not qualify.

Why it exists: DexToolBench's 24 tool tasks, where the tool must be rotated in-hand into a functional grasp; Playground LeapCubeReorient, LeapCubeRotateZAxis; HumanoidBench cube.

Tagged on 9 task(s): Aero Cube Rotate Z Axis, Flat spatula · flip over, Leap Cube Reorient, Leap Cube Rotate Z Axis, Long screwdriver · spin horizontal, Long screwdriver · spin vertical, Short screwdriver · spin horizontal, Short screwdriver · spin vertical, Spoon spatula · flip over

Contact¶

What the manipulation physics demands.

Grasp and transport¶

transport

Pick something up and put it somewhere. The floor of this facet: tag it even when richer labels apply, so a task whose only contact label is this one is exactly "plain pick-and-place".

Why it exists: 94 of BEHAVIOR's 100 goals, 105 of RoboLab's 120, 18 of RoboCasa-GR1's 24, most of LIBERO. Roughly 70% of the surveyed corpus — kept because its absence is what marks out the locomotion and in-hand suites.

Rolls up BEHAVIOR skills: pick up from, place in, place on, place under, place in next to, place on next to, hold, release

Tagged on 319 task(s): Adjust bottle, Align Blocks, Align Chopsticks, Aloha Hand Over, Aloha Single Peg Insertion, Animals In Bin, Apple And Yogurt In Bowl, Assembling Gift Baskets, Attach a Camera to a Tripod, Bagels On Plate, Balance Board, Ball Into Bottle, Ball Into Jar, Ball Onto Tower, Banana In Bowl, Banana On Plate, Banana Then Rubiks Cube, Bananas In Bin One More, Bananas In Bin Three Total, Bananas In Crate, Bananas Out Of Bin, Bbq Sauce In Bin, Beat block hammer, Big Pumpkin In Bin, Black Items In Bin, Block Stacking Order Agnostic, Block Stacking Specified Order, Blocks In Bin, Blocks ranking rgb, Blocks ranking size, Blue brush · sweep forward, Blue brush · sweep right, Bowl In Bin, Bowl Stacking Left On Right, Bowl Stacking Right On Left, Boxing Books Up for Storage, Boxing Food After Dinner, Bringing Paper To Recycling, Bringing Water, Bringing in Wood, Butter Above Raisin, Can Meat, Canned Food In Bin, Canning Food, Carrying Out Garden Furniture, Carrying in Groceries, Chop an Onion, Chopping Wood, Clamp In Right Bin, Claw hammer · swing down, Claw hammer · swing side, Clean A Keyboard, Clean Boxing Gloves, Clean Up Broken Glass, Clean Up Toys, Clean Up Your Desk, Clean Your Rusty Garden Tools, Clean a Patio, Clean a Trumpet, Cleaning Up Branches And Twigs, Cleaning Up Plates and Food, Clear Organic Objects, Clearing Food from Table into Fridge, Clutter Plastic, Clutter Pumpkin, Coffee Pot In Bin, Collect Screws, Collecting Aluminum Cans, Collecting Children's Toys, Composting Waste, Condiments In Bin, Cook A Brisket, Cook A Frozen Pie, Cook Bacon, Cook Broccolini, Cook Brussels Sprouts, Cook Cabbage, Cook Hot Dogs, Cooking Clear Plate, Cooking Pick Pasta Tool, Cover With Lid, Cubes And Blocks In Bin, Cylinder Through Hole, Differentiate Cubes, Dishes In Bin, Dispose Of Batteries, Dispose Of Glass, Dump bin big bin, Electronics In Bin, Flat eraser · wipe c, Flat eraser · wipe smile, Flat spatula · flip over, Flat spatula · serve plate, Food Packing By Color, Food Packing1 Boxes, Food Packing1 Cans, Food Packing2 Boxes, Food Packing2 Cans, Food Packing3 Boxes, Food Packing3 Cans, Freeze Fruit, Freeze Pies, Fruits Green Limes On Plate, Fruits Moving, Fruits Moving Orange Or Lime, Fruits On Plate, Fruits On Plate3, Fruits Onion, Fruits Onion To Plate, Fruits Oranges On Plate, Gap Retrieve, Getting Organized For Work, Grab A Bagel, Grab A Fruit, Grab roller, Green Spoons In Pot, Halve An Egg, Hammers In Left Bin, Handle eraser · wipe c, Handle eraser · wipe smile, Handover block, Handover microphone, Hanging Pictures, Hanging mug, Hiding Easter Eggs, Hold Cup, Installing A Fax Machine, Installing A Modem, Installing A Scanner, Installing Smoke Detectors, Jugs On Shelf, Keyboard Out Of Bin, Larger Object Raisin Box In Bin, Laying Tile Floors, Lift pot, Loading The Car, Long screwdriver · spin horizontal, Long screwdriver · spin vertical, Make Cabinet Doors, Make Gift Bags For Baby Showers, Make Microwave Popcorn, Make Pizza, Make Rose Centerpieces, Mallet hammer · swing down, Mallet hammer · swing side, Marker In Mug, Mouse On Keyboard, Move Banana To Bagel Plate, Move Cube, Move can pot, Move pill bottle pad, Move playing card away, Move stapler pad, Moving Boxes To Storage, Mustard Above Raisin, Mustard In Left Bin, Mustard In Right Bin, Non Hammer Tools In Right Bin, One Bottle In Square Pail, One Bottle On Shelf, Organizing Art Supplies, Organizing School Stuff, Outfit A Basic Toolbox, Packing Meal For Delivery, Panda Pick Cube, Panda Pick Cube Cartesian, Panda Pick Cube Orientation, Phone Or Remote In Bin, Pick Drill, Pick Glasses, Pick Orange Object, Pick Up Blue Pitcher, Pick Up Green Object, Pick diverse bottles, Pick dual bottles, Picking Up Toys, Picking Up Trash, Pinch Card, Pink Spoon In Pot, Place Book, Place Tall Box, Place a to b left, Place a to b right, Place bread basket, Place bread skillet, Place burger fries, Place can basket, Place cans plastic box, Place container plate, Place dual shoes, Place empty cup, Place fan, Place mouse pad, Place object basket, Place object scale, Place object stand, Place phone stand, Place shoe, Plastic Bottles In Square Pail, Polishing Shoes, Preparing Lunch Box, Put Bowl On Shelf Top, Put Mugs On Shelf, Put Together A Basic Pruning Kit, Put Two Mugs On Shelf, Put bottles dustbin, Put object cabinet, Putting Away Halloween Decorations, Putting Away Toys, Putting Dirty Dishes In Sink, Putting Dishes Away After Cleaning, Putting Shoes On Rack, Putting Up Christmas Decorations Inside, Raise Platform, Re Shelving Library Books, Rearrange Your Room, Rearranging Kitchen Furniture, Recycle Carton, Recycle Cartons On Box, Recycle Cartons Vertical Crate, Red Dishes In Bin, Red Items In Bin, Red brush · sweep forward, Red brush · sweep right, Reorient All Mugs, Reorient Jug, Reorient Red Mug, Reorient White Mugs, Retrieve Cube, Retrieve Roll, Roll Up Ball, Rotate QR code, Round Dough Sheet, Rubiks Cube, Rubiks Cube And Banana, Rubiks Cube Behind Bowl, Rubiks Cube In Front Of Bowl, Rubiks Cube Left Of Bowl, Rubiks Cube Or Banana, Rubiks Cube Right Of Bowl, Rubiks Cube Then Banana, Rubiks Cubes In Bin, Sauce Bottles Crate, Scrubbing Bathroom Floor, Seal Colander, Separate Marbles And Sand, Set Up A Coffee Station In Your Kitchen, Setting Mousetraps, Setting The Fire, Setting The Table, Setup A Bar For A Cocktail Party, Shake bottle, Shake bottle horizontally, Sharpie marker · draw smile, Sharpie marker · write c, Short screwdriver · spin horizontal, Short screwdriver · spin vertical, Slicing Vegetables, Small Pumpkin In Bin, Smaller Object Butter In Bin, Smartphone In Bin, Sorting Books On Shelf, Sorting Bottles Cans And Paper, Sorting Household Items, Sorting Vegetables, Spoon In Mug, Spoon spatula · flip over, Spoon spatula · serve plate, Spoons In Pot, Spraying For Bugs, Spraying Fruit Trees, Stabilize Bottle, Stack Bowls, Stack Cubes, Stack White Mugs, Stack Yellow On Red, Stack blocks three, Stack blocks two, Stack bowls three, Stack bowls two, Stack3 Rubiks Cube, Stacking Wood, Stamp seal, Stand Bulb, Stand Pages, Staples marker · draw smile, Staples marker · write c, Store Batteries, Store Honey, Store Produce, Storing Food, Sweeping Garage, Take Measuring Spoon Out, Take Mugs Off Of Shelf, Take Spatula Off Shelf, Thawing Frozen Food, Throw Away Apple, Throw Away Snacks, Tidying Bathroom, Tidying Bedroom, Tidying Living Room, Tool Organization, Tool Organization Both, Tools Picking All Hammers, Tools Picking Drill, Tools Picking Hammer, Toy In Bin, Unloading The Car, Unstack Rubiks Cube, Utensils In Mug, Vacuuming Floors, Wash A Baseball Cap, Wash Dog Toys, Water Into Mug, White Mug In Center Of Table, White Mugs In Bin, Wood Spatula To Bowl, Yellow And White Objects In Bin, Yogurt In Bowl

Push, slide, sweep¶

nonprehensile

Move something without holding it. Tag when the solution does not go through a grasp.

Why it exists: Meta-World push, plate-slide, sweep; Playground PandaRobotiqPushCube; HumanoidBench push and package; RoboWits move_cube.

Rolls up BEHAVIOR skills: push to, sweep surface

Tagged on 12 task(s): Align Blocks, Align Chopsticks, Clean a Patio, Collect Screws, Dominos, Move Cube, Panda Robotiq Push Cube, Pinch Card, Place Book, Retrieve Cube, Roll Up Ball, Sweeping Garage

Momentum and timing¶

dynamics

The object leaves contact, or keeps moving after it, and timing or momentum decides the outcome — throw, catch, hit, topple, roll a chain reaction. A slower, more careful version fails.

Why it exists: HumanoidBench basketball, which its authors class as dynamic manipulation; Meta-World basketball and soccer; RoboWits dominos and roll_up_ball.

Rolls up BEHAVIOR skills: tip over

Tagged on 8 task(s): Aero Cube Rotate Z Axis, Align Chopsticks, Beat block hammer, Dominos, Leap Cube Rotate Z Axis, Move Cube, Shake bottle, Shake bottle horizontally

Open and close¶

articulated

A hinge, drawer, lid or slider dictates the path, so the trajectory is set by the object rather than chosen by the planner.

Why it exists: RoboCasa365's doors, drawers, lids and racks; Meta-World's door, drawer, window and box tasks; 22 BEHAVIOR goals; RoboTwin Open Laptop, Open Microwave.

Rolls up BEHAVIOR skills: open door, open drawer, open lid, close door, close drawer, close lid

Tagged on 47 task(s): Boxing Food After Dinner, Bringing Paper To Recycling, Bringing Water, Can Meat, Canning Food, Carrying in Groceries, Clean Boxing Gloves, Clean Up Your Desk, Clean Your Rusty Garden Tools, Cleaning Up Branches And Twigs, Cleaning Up Plates and Food, Clearing Food from Table into Fridge, Cook A Brisket, Cook A Frozen Pie, Cook Bacon, Cook Brussels Sprouts, Cook Cabbage, Cook Hot Dogs, Freeze Fruit, Freeze Pies, Halve An Egg, Installing A Modem, Installing A Scanner, Loading The Car, Make Gift Bags For Baby Showers, Make Microwave Popcorn, Make Pizza, Open laptop, Open microwave, Outfit A Basic Toolbox, Panda Open Cabinet, Preparing Lunch Box, Put Together A Basic Pruning Kit, Put object cabinet, Putting Away Halloween Decorations, Putting Dishes Away After Cleaning, Rearranging Kitchen Furniture, Setting The Table, Slicing Vegetables, Store Batteries, Store Honey, Store Produce, Storing Food, Thawing Frozen Food, Unloading The Car, Wash A Baseball Cap, Wash Dog Toys

Insert and assemble¶

precision-fit

Make one object hold onto another with little clearance — insert, thread, hang, fasten, nest. The tolerance is the work, not the transport.

Why it exists: Playground AlohaSinglePegInsertion; Meta-World peg-insert-side, assembly, disassemble; HumanoidBench insert; RoboCasa's coffee-pod insertion; RoboWits cylinder_through_hole.

Rolls up BEHAVIOR skills: insert, attach, hang

Tagged on 28 task(s): Aloha Single Peg Insertion, Attach a Camera to a Tripod, Ball Into Bottle, Ball Onto Tower, Bowl Stacking Left On Right, Bowl Stacking Right On Left, Can Meat, Cook Brussels Sprouts, Cover With Lid, Cylinder Through Hole, Green Spoons In Pot, Hanging Pictures, Hanging mug, Installing Smoke Detectors, Make Cabinet Doors, Marker In Mug, Pink Spoon In Pot, Place can basket, Place object stand, Place phone stand, Retrieve Roll, Spoon In Mug, Spoons In Pot, Stack Bowls, Stack White Mugs, Stand Pages, Thawing Frozen Food, Utensils In Mug

Stack and balance¶

stability

The result must stay standing once released. The difficulty is the settling, not the reach.

Why it exists: RoboTwin Stack Blocks/Bowls; RoboWits stack_cubes, ball_onto_tower, balance_board; RoboLab's stacking tasks; 8 of RoboWits' 30 insights are about the centre of mass.

Tagged on 24 task(s): Balance Board, Ball Onto Tower, Block Stacking Order Agnostic, Block Stacking Specified Order, Hold Cup, Moving Boxes To Storage, Raise Platform, Reorient All Mugs, Reorient Jug, Reorient Red Mug, Reorient White Mugs, Sorting Books On Shelf, Stabilize Bottle, Stack Cubes, Stack White Mugs, Stack Yellow On Red, Stack blocks three, Stack blocks two, Stack bowls three, Stack bowls two, Stack3 Rubiks Cube, Stacking Wood, Stand Bulb, Stand Pages

Regulated force¶

force

Sustained or controlled force against resistance — press, pry, hold a load, keep a tool on a surface. Position control alone fails.

Why it exists: HumanoidBench powerlift and window; Meta-World handle-press and lever-pull; BEHAVIOR's wiping activities.

Rolls up BEHAVIOR skills: press, wipe hard

Tagged on 13 task(s): Ball Into Jar, Clean A Keyboard, Clean Your Rusty Garden Tools, Clean a Trumpet, Open microwave, Pinch Card, Place Book, Polishing Shoes, Press stapler, Roll Up Ball, Round Dough Sheet, Scrubbing Bathroom Floor, Stamp seal

Deformable¶

deformable

Cloth, rope, dough, foam, paper — the shape is part of the state.

Why it exists: RoboWits round_dough_sheet and ball_into_jar; BEHAVIOR's soft bodies. Rare across the field — no surveyed suite does cloth or rope, which is itself worth being able to see.

Tagged on 2 task(s): Ball Into Jar, Round Dough Sheet

Fluids and granular media¶

fluid-granular

Liquid, powder, sand or many small particles must be moved, contained or kept in place.

Why it exists: RoboWits water_into_mug, seal_colander; VLABench MakeJuice; BEHAVIOR's filled goals.

Rolls up BEHAVIOR skills: pour

Tagged on 10 task(s): Canning Food, Chop an Onion, Cook Cabbage, Differentiate Cubes, Dump bin big bin, Make Microwave Popcorn, Seal Colander, Separate Marbles And Sand, Stabilize Bottle, Water Into Mug

Use a tool¶

tool-use

An object must be used on another to do what the hand cannot — including grasping it in the pose its use requires.

Why it exists: DexToolBench in full (hammer, marker, eraser, brush, spatula, screwdriver); Meta-World hammer, stick-push, stick-pull; HumanoidBench window and spoon; VLABench's Tool Use skill; RoboLab's affordance tag.

Tagged on 39 task(s): Align Blocks, Align Chopsticks, Ball Into Bottle, Beat block hammer, Canning Food, Chop an Onion, Chopping Wood, Clean A Keyboard, Clean Your Rusty Garden Tools, Clean a Patio, Clean a Trumpet, Collect Screws, Cook Brussels Sprouts, Cook Cabbage, Differentiate Cubes, Dominos, Halve An Egg, Hold Cup, Move Cube, Pinch Card, Place Book, Polishing Shoes, Retrieve Roll, Roll Up Ball, Round Dough Sheet, Scan object, Scrubbing Bathroom Floor, Seal Colander, Separate Marbles And Sand, Setting The Fire, Slicing Vegetables, Spraying For Bugs, Spraying Fruit Trees, Stabilize Bottle, Stamp seal, Stand Bulb, Sweeping Garage, Vacuuming Floors, Water Into Mug

Object state¶

What must change about an object beyond its pose. Success is not a position.

Operate a device¶

device-operation

A switch, button, knob, lever, handle or faucet must be operated.

Why it exists: RoboCasa365's knobs, levers and buttons; Meta-World's four button-press tasks, dial-turn and lever-pull; RoboTwin Click Alarmclock, Turn Switch.

Rolls up BEHAVIOR skills: press, turn on switch, turn off switch, turn to, ignite

Tagged on 23 task(s): Clean Boxing Gloves, Click alarm clock, Click bell, Cook A Brisket, Cook A Frozen Pie, Cook Bacon, Cook Broccolini, Cook Brussels Sprouts, Cook Cabbage, Cook Hot Dogs, Installing A Fax Machine, Installing A Modem, Installing A Scanner, Make Microwave Popcorn, Make Pizza, Press stapler, Setting The Fire, Thawing Frozen Food, Turn switch, Turning On Radio, Turning Out All Lights Before Sleep, Wash A Baseball Cap, Wash Dog Toys

Heat, cook, freeze¶

thermal

Cooking, heating, freezing, melting, boiling or igniting.

Why it exists: 9 BEHAVIOR goals; RoboCasa365's boiling, toasting and steaming; VLABench HeatFoodWithMicrowave.

Rolls up BEHAVIOR skills: ignite

Tagged on 12 task(s): Cook A Brisket, Cook A Frozen Pie, Cook Bacon, Cook Broccolini, Cook Brussels Sprouts, Cook Cabbage, Cook Hot Dogs, Freeze Pies, Make Microwave Popcorn, Make Pizza, Setting The Fire, Thawing Frozen Food

Cut¶

cutting

Tool-mediated division — slice, dice, chop, halve.

Why it exists: 9 BEHAVIOR goals, built from half__* synsets; RoboCasa's chopping activities.

Rolls up BEHAVIOR skills: chop

Tagged on 7 task(s): Canning Food, Chop an Onion, Chopping Wood, Cook Brussels Sprouts, Cook Cabbage, Halve An Egg, Slicing Vegetables

Treat a surface¶

surface-treatment

Particles must be put onto or taken off a surface — wipe, sweep, spray, soak, erase.

Why it exists: 13 BEHAVIOR goals; HumanoidBench window; DexToolBench eraser and brush.

Rolls up BEHAVIOR skills: wipe hard, sweep surface, spray

Tagged on 13 task(s): Clean A Keyboard, Clean Boxing Gloves, Clean Your Rusty Garden Tools, Clean a Patio, Clean a Trumpet, Polishing Shoes, Scrubbing Bathroom Floor, Spraying For Bugs, Spraying Fruit Trees, Sweeping Garage, Vacuuming Floors, Wash A Baseball Cap, Wash Dog Toys

Goal¶

What makes the goal hard to satisfy, or hard to read.

Geometric constraint¶

geometric-constraint

Success is a constraint over a whole configuration rather than a place to put one thing — fit through, pack into, route around, thread between, keep clear of. The work is finding a feasible arrangement, not finding the target. Distinct from relational-goal, which names which object.

Why it exists: The largest cluster in RoboWits — 15 of its 30 published insights are about geometry and clearance — and what Meta-World's peg and assembly tasks and RoboCasa's packing really test. No suite annotates it.

Tagged on 52 task(s): Align Blocks, Align Chopsticks, Assembling Gift Baskets, Balance Board, Ball Into Bottle, Ball Into Jar, Ball Onto Tower, Cover With Lid, Cylinder Through Hole, Dominos, Food Packing By Color, Food Packing3 Boxes, Food Packing3 Cans, Gap Retrieve, Getting Organized For Work, Jugs On Shelf, Laying Tile Floors, Moving Boxes To Storage, One Bottle On Shelf, Packing Meal For Delivery, Pinch Card, Place Book, Place Tall Box, Polishing Shoes, Put Bowl On Shelf Top, Put Mugs On Shelf, Put Two Mugs On Shelf, Putting Away Halloween Decorations, Putting Dishes Away After Cleaning, Putting Shoes On Rack, Putting Up Christmas Decorations Inside, Raise Platform, Rearranging Kitchen Furniture, Recycle Cartons Vertical Crate, Retrieve Cube, Retrieve Roll, Roll Up Ball, Round Dough Sheet, Seal Colander, Separate Marbles And Sand, Set Up A Coffee Station In Your Kitchen, Setting The Table, Setup A Bar For A Cocktail Party, Sorting Books On Shelf, Sorting Household Items, Stack Bowls, Stack Cubes, Stand Bulb, Stand Pages, Storing Food, Take Mugs Off Of Shelf, Take Spatula Off Shelf

Spatial reference¶

relational-goal

The goal names its target by a relation — left of, behind, between, second from the right, centred on — so it needs a frame of reference to ground.

Why it exists: LIBERO-Spatial, where position is the only thing telling two identical bowls apart; VLABench's nth-from-left tasks; 29 RoboLab tasks.

Tagged on 57 task(s): Bananas Out Of Bin, Blocks ranking rgb, Blocks ranking size, Bowl Stacking Left On Right, Bowl Stacking Right On Left, Butter Above Raisin, Clamp In Right Bin, Cooking Pick Pasta Tool, Food Packing By Color, Getting Organized For Work, Hammers In Left Bin, Handover block, Hiding Easter Eggs, Installing A Modem, Installing A Scanner, Keyboard Out Of Bin, Laying Tile Floors, Move can pot, Move playing card away, Mustard Above Raisin, Mustard In Left Bin, Mustard In Right Bin, Non Hammer Tools In Right Bin, Place a to b left, Place a to b right, Place dual shoes, Place fan, Polishing Shoes, Put Bowl On Shelf Top, Put Mugs On Shelf, Put Two Mugs On Shelf, Putting Away Halloween Decorations, Putting Shoes On Rack, Putting Up Christmas Decorations Inside, Recycle Cartons Vertical Crate, Rotate QR code, Rubiks Cube Behind Bowl, Rubiks Cube In Front Of Bowl, Rubiks Cube Left Of Bowl, Rubiks Cube Right Of Bowl, Scan object, Set Up A Coffee Station In Your Kitchen, Setting Mousetraps, Setting The Table, Setup A Bar For A Cocktail Party, Sorting Household Items, Spoon In Mug, Stacking Wood, Take Spatula Off Shelf, Tidying Bedroom, Tool Organization, Tool Organization Both, Tools Picking All Hammers, Tools Picking Drill, Tools Picking Hammer, Unloading The Car, White Mug In Center Of Table

Attribute reference¶

attribute-goal

Objects are named by colour, size or a category that several objects share — "the black items", "all green fruit", "the smaller toys" — so the right one has to be picked out.

Why it exists: LIBERO-Object; RoboLab's colour and size tags; RoboTwin Blocks Ranking RGB. RoboLab's semantics tag (60 tasks) is deliberately not mapped: naming an object by its category is how instructions normally read.

Tagged on 67 task(s): Apple And Yogurt In Bowl, Bbq Sauce In Bin, Big Pumpkin In Bin, Black Items In Bin, Block Stacking Specified Order, Blocks In Bin, Blocks ranking rgb, Blocks ranking size, Clean Up Toys, Clear Organic Objects, Clutter Plastic, Condiments In Bin, Cooking Clear Plate, Cooking Pick Pasta Tool, Cover With Lid, Cubes And Blocks In Bin, Dishes In Bin, Electronics In Bin, Food Packing By Color, Food Packing1 Boxes, Food Packing1 Cans, Food Packing2 Boxes, Food Packing2 Cans, Food Packing3 Boxes, Food Packing3 Cans, Fruits Green Limes On Plate, Fruits Moving, Green Spoons In Pot, Hammers In Left Bin, Larger Object Raisin Box In Bin, Move stapler pad, Non Hammer Tools In Right Bin, Pick Orange Object, Pick Up Blue Pitcher, Pick Up Green Object, Pick diverse bottles, Pink Spoon In Pot, Place a to b left, Place a to b right, Place empty cup, Place fan, Place mouse pad, Plastic Bottles In Square Pail, Recycle Carton, Recycle Cartons On Box, Recycle Cartons Vertical Crate, Red Dishes In Bin, Red Items In Bin, Reorient Red Mug, Reorient White Mugs, Sauce Bottles Crate, Small Pumpkin In Bin, Smaller Object Butter In Bin, Sorting Books On Shelf, Sorting Bottles Cans And Paper, Sorting Household Items, Sorting Vegetables, Spoons In Pot, Stack White Mugs, Stamp seal, Throw Away Snacks, Tool Organization, Tools Picking Hammer, White Mug In Center Of Table, White Mugs In Bin, Yellow And White Objects In Bin, Yogurt In Bowl

Counting and quantifiers¶

quantified-goal

"Exactly two", "all three", "one more", "any shelf". Miscounting, or failing to commit to one choice consistently, fails outright.

Why it exists: BEHAVIOR's forn goals; RoboLab bananas_in_bin_three_total; RoboTwin Stack Blocks Three.

Tagged on 42 task(s): Assembling Gift Baskets, Bananas In Bin One More, Bananas In Bin Three Total, Bananas In Crate, Can Meat, Canning Food, Cleaning Up Plates and Food, Clearing Food from Table into Fridge, Collecting Children's Toys, Freeze Fruit, Freeze Pies, Fruits Moving, Fruits Moving Orange Or Lime, Fruits On Plate3, Fruits Oranges On Plate, Hiding Easter Eggs, Make Gift Bags For Baby Showers, One Bottle In Square Pail, One Bottle On Shelf, Packing Meal For Delivery, Phone Or Remote In Bin, Picking Up Toys, Place bread basket, Place cans plastic box, Put Mugs On Shelf, Put Two Mugs On Shelf, Putting Away Halloween Decorations, Putting Away Toys, Putting Dishes Away After Cleaning, Putting Up Christmas Decorations Inside, Re Shelving Library Books, Rearrange Your Room, Rubiks Cube Or Banana, Setting Mousetraps, Setting The Table, Setup A Bar For A Cocktail Party, Sorting Books On Shelf, Sorting Bottles Cans And Paper, Sorting Vegetables, Store Batteries, Store Produce, Storing Food

Implicit intent¶

implicit-goal

The instruction states a need rather than an action, or is vague enough to admit several readings — "tidy this up", "I'm thirsty". The agent must choose a reading and commit.

Why it exists: VLABench's Semantic Instruction track, built on implicit needs in dialogue; RoboLab ships a vague wording for every task.

Tagged on 7 task(s): Cooking Pick Pasta Tool, Dishes In Bin, Fruits On Plate, Fruits On Plate3, Fruits Onion To Plate, Fruits Oranges On Plate, Recycle Carton

World knowledge¶

knowledge-goal

Facts that are not in the scene are needed — which fruit has vitamin C, the rules of the game, which tube holds the acid, what a latte is. Perception and planning cannot get there.

Why it exists: VLABench's Common Sense & World Knowledge track and its composites (PlayTexasHoldem, GetLatteCoffee, SelectChemistryTube) — the only suite in the survey testing this, which is why the label exists.

Tagged on 4 task(s): Clear Organic Objects, Recycle Carton, Recycle Cartons On Box, Recycle Cartons Vertical Crate

Structure and risk¶

How much has to go right, and whether a mistake can be undone. The two flags below apply at any horizon. — ladder: take the highest rung that applies

Atomic¶

horizon-atomic

One skill on one object; seconds of motion. Failure means retry.

Why it exists: Meta-World, LIBERO's three short suites, most of RoboLab and RoboTwin.

Tagged on 105 task(s): Adjust bottle, Aero Cube Rotate Z Axis, Balance Board, Ball Into Jar, Banana In Bowl, Banana On Plate, Bananas In Bin One More, Big Pumpkin In Bin, Bowl In Bin, Bowl Stacking Left On Right, Bowl Stacking Right On Left, Butter Above Raisin, Canned Food In Bin, Clamp In Right Bin, Click alarm clock, Click bell, Coffee Pot In Bin, Cooking Pick Pasta Tool, Cover With Lid, Cylinder Through Hole, Food Packing By Color, Food Packing1 Boxes, Food Packing1 Cans, Fruits Moving, Fruits Moving Orange Or Lime, Fruits Onion, Fruits Onion To Plate, Grab A Bagel, Grab A Fruit, Grab roller, Hanging Pictures, Keyboard Out Of Bin, Larger Object Raisin Box In Bin, Leap Cube Reorient, Leap Cube Rotate Z Axis, Lift pot, Marker In Mug, Mouse On Keyboard, Move can pot, Move pill bottle pad, Move playing card away, Move stapler pad, Mustard Above Raisin, Mustard In Left Bin, Mustard In Right Bin, One Bottle In Square Pail, One Bottle On Shelf, Open laptop, Panda Open Cabinet, Panda Pick Cube, Panda Pick Cube Cartesian, Panda Pick Cube Orientation, Panda Robotiq Push Cube, Phone Or Remote In Bin, Pick Drill, Pick Glasses, Pick Orange Object, Pick Up Blue Pitcher, Pick Up Green Object, Pick diverse bottles, Pick dual bottles, Pink Spoon In Pot, Place a to b left, Place a to b right, Place bread skillet, Place container plate, Place empty cup, Place fan, Place mouse pad, Place object scale, Place object stand, Place phone stand, Place shoe, Press stapler, Put Bowl On Shelf Top, Reorient Jug, Reorient Red Mug, Rotate QR code, Rubiks Cube, Rubiks Cube Behind Bowl, Rubiks Cube In Front Of Bowl, Rubiks Cube Left Of Bowl, Rubiks Cube Or Banana, Rubiks Cube Right Of Bowl, Sauce Bottles Crate, Shake bottle, Shake bottle horizontally, Small Pumpkin In Bin, Smaller Object Butter In Bin, Smartphone In Bin, Spoon In Mug, Stack Bowls, Stamp seal, Store Honey, Take Measuring Spoon Out, Take Spatula Off Shelf, Throw Away Apple, Tools Picking Drill, Tools Picking Hammer, Toy In Bin, Turn switch, Turning On Radio, White Mug In Center Of Table, Wood Spatula To Bowl, Yogurt In Bowl

Composite¶

horizon-composite

Two to five dependent steps; an early mistake can strand the later ones.

Why it exists: LIBERO-Long's 3–4 subgoals; RoboWits, where a tool must be fetched before it is used.

Tagged on 194 task(s): Align Blocks, Align Chopsticks, Aloha Hand Over, Aloha Single Peg Insertion, Animals In Bin, Apple And Yogurt In Bowl, Attach a Camera to a Tripod, Bagels On Plate, Ball Into Bottle, Ball Onto Tower, Banana Then Rubiks Cube, Bananas In Bin Three Total, Bananas In Crate, Bananas Out Of Bin, Bbq Sauce In Bin, Beat block hammer, Black Items In Bin, Block Stacking Order Agnostic, Block Stacking Specified Order, Blocks In Bin, Blocks ranking rgb, Blocks ranking size, Blue brush · sweep forward, Blue brush · sweep right, Boxing Food After Dinner, Bringing Paper To Recycling, Bringing Water, Bringing in Wood, Can Meat, Carrying Out Garden Furniture, Carrying in Groceries, Chop an Onion, Claw hammer · swing down, Claw hammer · swing side, Clean A Keyboard, Clean Boxing Gloves, Clean Up Broken Glass, Clean Your Rusty Garden Tools, Clean a Patio, Clean a Trumpet, Cleaning Up Branches And Twigs, Cleaning Up Plates and Food, Clearing Food from Table into Fridge, Clutter Plastic, Clutter Pumpkin, Collect Screws, Composting Waste, Condiments In Bin, Cook A Brisket, Cook A Frozen Pie, Cook Broccolini, Cook Cabbage, Cook Hot Dogs, Cooking Clear Plate, Differentiate Cubes, Dishes In Bin, Dispose Of Batteries, Dispose Of Glass, Dominos, Dump bin big bin, Electronics In Bin, Flat eraser · wipe c, Flat eraser · wipe smile, Flat spatula · flip over, Flat spatula · serve plate, Food Packing2 Boxes, Food Packing2 Cans, Food Packing3 Boxes, Food Packing3 Cans, Freeze Fruit, Freeze Pies, Fruits Green Limes On Plate, Fruits On Plate3, Fruits Oranges On Plate, Gap Retrieve, Green Spoons In Pot, Halve An Egg, Hammers In Left Bin, Handle eraser · wipe c, Handle eraser · wipe smile, Handover block, Handover microphone, Hanging mug, Hiding Easter Eggs, Hold Cup, Installing A Fax Machine, Installing A Modem, Installing A Scanner, Installing Smoke Detectors, Jugs On Shelf, Laying Tile Floors, Long screwdriver · spin horizontal, Long screwdriver · spin vertical, Make Cabinet Doors, Make Gift Bags For Baby Showers, Make Microwave Popcorn, Make Rose Centerpieces, Mallet hammer · swing down, Mallet hammer · swing side, Move Banana To Bagel Plate, Move Cube, Moving Boxes To Storage, Non Hammer Tools In Right Bin, Open microwave, Organizing Art Supplies, Packing Meal For Delivery, Picking Up Trash, Pinch Card, Place Book, Place Tall Box, Place bread basket, Place burger fries, Place can basket, Place cans plastic box, Place dual shoes, Place object basket, Plastic Bottles In Square Pail, Polishing Shoes, Put Mugs On Shelf, Put Together A Basic Pruning Kit, Put Two Mugs On Shelf, Put object cabinet, Putting Away Halloween Decorations, Putting Dirty Dishes In Sink, Putting Shoes On Rack, Raise Platform, Re Shelving Library Books, Rearrange Your Room, Rearranging Kitchen Furniture, Recycle Carton, Recycle Cartons On Box, Recycle Cartons Vertical Crate, Red Dishes In Bin, Red Items In Bin, Red brush · sweep forward, Red brush · sweep right, Reorient All Mugs, Reorient White Mugs, Retrieve Cube, Retrieve Roll, Roll Up Ball, Round Dough Sheet, Rubiks Cube And Banana, Rubiks Cube Then Banana, Rubiks Cubes In Bin, Scan object, Scrubbing Bathroom Floor, Seal Colander, Separate Marbles And Sand, Setting Mousetraps, Setting The Fire, Sharpie marker · draw smile, Sharpie marker · write c, Short screwdriver · spin horizontal, Short screwdriver · spin vertical, Spoon spatula · flip over, Spoon spatula · serve plate, Spoons In Pot, Spraying For Bugs, Spraying Fruit Trees, Stabilize Bottle, Stack Cubes, Stack White Mugs, Stack Yellow On Red, Stack blocks three, Stack blocks two, Stack bowls three, Stack bowls two, Stack3 Rubiks Cube, Stacking Wood, Stand Bulb, Stand Pages, Staples marker · draw smile, Staples marker · write c, Store Batteries, Store Produce, Sweeping Garage, Take Mugs Off Of Shelf, Throw Away Snacks, Tidying Bathroom, Tidying Bedroom, Tidying Living Room, Tool Organization, Tool Organization Both, Tools Picking All Hammers, Turning Out All Lights Before Sleep, Unloading The Car, Unstack Rubiks Cube, Utensils In Mug, Vacuuming Floors, Wash A Baseball Cap, Water Into Mug, White Mugs In Bin, Yellow And White Objects In Bin

Activity¶

horizon-activity

Six or more dependent steps, often minutes of motion and several rooms or fixtures. Recovery matters more than any single skill.

Why it exists: BEHAVIOR goals running to 10 clauses and 869 s; RoboCasa365's composites at up to 15 subtasks; VLABench's 500+ step composites.

Tagged on 35 task(s): Assembling Gift Baskets, Boxing Books Up for Storage, Canning Food, Chopping Wood, Clean Up Toys, Clean Up Your Desk, Clear Organic Objects, Collecting Aluminum Cans, Collecting Children's Toys, Cook Bacon, Cook Brussels Sprouts, Cubes And Blocks In Bin, Fruits On Plate, Getting Organized For Work, Loading The Car, Make Pizza, Organizing School Stuff, Outfit A Basic Toolbox, Picking Up Toys, Preparing Lunch Box, Put bottles dustbin, Putting Away Toys, Putting Dishes Away After Cleaning, Putting Up Christmas Decorations Inside, Set Up A Coffee Station In Your Kitchen, Setting The Table, Setup A Bar For A Cocktail Party, Slicing Vegetables, Sorting Books On Shelf, Sorting Bottles Cans And Paper, Sorting Household Items, Sorting Vegetables, Storing Food, Thawing Frozen Food, Wash Dog Toys

Required order¶

ordered

The goal fixes the order of the steps, not just the end state.

A flag, not a rung: it can apply at any level of this ladder.

Why it exists: HumanoidBench bookshelf ("in a given order"); RoboTwin Blocks Ranking.

Tagged on 66 task(s): Banana Then Rubiks Cube, Block Stacking Specified Order, Blocks ranking rgb, Blocks ranking size, Blue brush · sweep forward, Blue brush · sweep right, Boxing Food After Dinner, Bringing Paper To Recycling, Bringing Water, Can Meat, Canning Food, Carrying in Groceries, Claw hammer · swing down, Claw hammer · swing side, Clean Up Your Desk, Clean Your Rusty Garden Tools, Cleaning Up Plates and Food, Clearing Food from Table into Fridge, Cook Bacon, Cook Brussels Sprouts, Dump bin big bin, Flat eraser · wipe c, Flat eraser · wipe smile, Flat spatula · flip over, Flat spatula · serve plate, Freeze Fruit, Freeze Pies, Handle eraser · wipe c, Handle eraser · wipe smile, Handover block, Handover microphone, Hanging mug, Loading The Car, Long screwdriver · spin horizontal, Long screwdriver · spin vertical, Mallet hammer · swing down, Mallet hammer · swing side, Outfit A Basic Toolbox, Place can basket, Place object basket, Preparing Lunch Box, Put Together A Basic Pruning Kit, Put bottles dustbin, Put object cabinet, Putting Away Halloween Decorations, Putting Dishes Away After Cleaning, Rearranging Kitchen Furniture, Red brush · sweep forward, Red brush · sweep right, Rubiks Cube Then Banana, Setting The Fire, Sharpie marker · draw smile, Sharpie marker · write c, Short screwdriver · spin horizontal, Short screwdriver · spin vertical, Slicing Vegetables, Sorting Books On Shelf, Spoon spatula · flip over, Spoon spatula · serve plate, Stack Yellow On Red, Stack blocks three, Stack blocks two, Stack bowls three, Staples marker · draw smile, Staples marker · write c, Thawing Frozen Food

Irreversible¶

irreversible

A wrong action cannot be undone — cut, cooked, spilled, burnt, dropped somewhere unreachable. The episode is lost rather than delayed, so an agent that explores by trying things is penalised in a way retries do not show.

A flag, not a rung: it can apply at any level of this ladder.

Why it exists: Cutting and cooking in BEHAVIOR and RoboCasa; DexToolBench's marker tasks. Named for the coding agent's failure mode, not the robot's.

Tagged on 15 task(s): Canning Food, Chop an Onion, Chopping Wood, Cook A Brisket, Cook A Frozen Pie, Cook Bacon, Cook Broccolini, Cook Brussels Sprouts, Cook Cabbage, Cook Hot Dogs, Dump bin big bin, Halve An Egg, Make Pizza, Setting The Fire, Slicing Vegetables

Perception¶

What has to be perceived that is not the geometry in front of the camera. Judging a state the goal already names is not listed here — the state facet covers it, and counting it twice is how a taxonomy stops being informative.

Partial observability¶

partial-observability

What the task needs is not visible from the start pose — inside a cabinet, behind something, in another room. The robot must move or open something before it can even see the problem.

Why it exists: BEHAVIOR exposes no simulator state at evaluation, so anything outside the opening view has to be found; RoboCasa's objects inside appliances; VLABench's Explore skill.

Tagged on 36 task(s): Boxing Food After Dinner, Bringing Paper To Recycling, Bringing Water, Can Meat, Canning Food, Carrying in Groceries, Clean Up Broken Glass, Clean Your Rusty Garden Tools, Composting Waste, Cook A Brisket, Cook A Frozen Pie, Cook Bacon, Cook Brussels Sprouts, Cook Cabbage, Cook Hot Dogs, Dispose Of Batteries, Dispose Of Glass, Freeze Fruit, Freeze Pies, Halve An Egg, Loading The Car, Make Gift Bags For Baby Showers, Make Pizza, Moving Boxes To Storage, Picking Up Trash, Preparing Lunch Box, Putting Away Toys, Setting The Table, Slicing Vegetables, Sorting Household Items, Store Batteries, Store Honey, Thawing Frozen Food, Turning Out All Lights Before Sleep, Unloading The Car, Wash Dog Toys

Act to find out¶

active-perception

The information needed cannot be seen at all, only produced — weigh it, float it, push it to see if it is fixed, open it to check. Perception requires a physical experiment.

Why it exists: RoboWits differentiate_cubes, where wood and metal look the same and the robot has to float them. The rarest thing in the survey and the one closest to what a coding agent does when it probes an unfamiliar API.

Tagged on 1 task(s): Differentiate Cubes

Fine visual detail¶

fine-detail

Recognition turns on print, texture or small geometry rather than shape or colour — a card's face, a book's title, a code, a painting.

Why it exists: VLABench's whole Mesh & Texture dimension (SelectPainting, SelectBook, poker); RoboTwin Rotate QRcode and Scan Object.

Tagged on 5 task(s): Align Chopsticks, Move Cube, Rotate QR code, Sorting Books On Shelf, Spoons In Pot

Generalisation¶

What the benchmark varies between episodes, plus one flag for language, which does not nest with the rest. This is the facet that separates a capability test from a replay test, and four of the surveyed suites exist mainly to measure it. — ladder: take the highest rung that applies

Nothing varies¶

gen-fixed

One fixed initial state. A recorded trajectory replays; the task measures execution, not capability. Worth recording plainly rather than politely.

Why it exists: RoboLab ships one initial state per task and randomises nothing.

Tagged on 144 task(s): Animals In Bin, Apple And Yogurt In Bowl, Bagels On Plate, Banana In Bowl, Banana On Plate, Banana Then Rubiks Cube, Bananas In Bin One More, Bananas In Bin Three Total, Bananas In Crate, Bananas Out Of Bin, Bbq Sauce In Bin, Big Pumpkin In Bin, Black Items In Bin, Block Stacking Order Agnostic, Block Stacking Specified Order, Blocks In Bin, Blue brush · sweep forward, Blue brush · sweep right, Bowl In Bin, Bowl Stacking Left On Right, Bowl Stacking Right On Left, Butter Above Raisin, Canned Food In Bin, Clamp In Right Bin, Claw hammer · swing down, Claw hammer · swing side, Clean Up Toys, Clear Organic Objects, Clutter Plastic, Clutter Pumpkin, Coffee Pot In Bin, Condiments In Bin, Cooking Clear Plate, Cooking Pick Pasta Tool, Cubes And Blocks In Bin, Dishes In Bin, Electronics In Bin, Flat eraser · wipe c, Flat eraser · wipe smile, Flat spatula · flip over, Flat spatula · serve plate, Food Packing By Color, Food Packing1 Boxes, Food Packing1 Cans, Food Packing2 Boxes, Food Packing2 Cans, Food Packing3 Boxes, Food Packing3 Cans, Fruits Green Limes On Plate, Fruits Moving, Fruits Moving Orange Or Lime, Fruits On Plate, Fruits On Plate3, Fruits Onion, Fruits Onion To Plate, Fruits Oranges On Plate, Grab A Bagel, Grab A Fruit, Green Spoons In Pot, Hammers In Left Bin, Handle eraser · wipe c, Handle eraser · wipe smile, Jugs On Shelf, Keyboard Out Of Bin, Larger Object Raisin Box In Bin, Long screwdriver · spin horizontal, Long screwdriver · spin vertical, Mallet hammer · swing down, Mallet hammer · swing side, Marker In Mug, Mouse On Keyboard, Move Banana To Bagel Plate, Mustard Above Raisin, Mustard In Left Bin, Mustard In Right Bin, Non Hammer Tools In Right Bin, One Bottle In Square Pail, One Bottle On Shelf, Phone Or Remote In Bin, Pick Drill, Pick Glasses, Pick Orange Object, Pick Up Blue Pitcher, Pick Up Green Object, Pink Spoon In Pot, Plastic Bottles In Square Pail, Put Bowl On Shelf Top, Put Mugs On Shelf, Put Two Mugs On Shelf, Recycle Carton, Recycle Cartons On Box, Recycle Cartons Vertical Crate, Red Dishes In Bin, Red Items In Bin, Red brush · sweep forward, Red brush · sweep right, Reorient All Mugs, Reorient Jug, Reorient Red Mug, Reorient White Mugs, Rubiks Cube, Rubiks Cube And Banana, Rubiks Cube Behind Bowl, Rubiks Cube In Front Of Bowl, Rubiks Cube Left Of Bowl, Rubiks Cube Or Banana, Rubiks Cube Right Of Bowl, Rubiks Cube Then Banana, Rubiks Cubes In Bin, Sauce Bottles Crate, Sharpie marker · draw smile, Sharpie marker · write c, Short screwdriver · spin horizontal, Short screwdriver · spin vertical, Small Pumpkin In Bin, Smaller Object Butter In Bin, Smartphone In Bin, Spoon In Mug, Spoon spatula · flip over, Spoon spatula · serve plate, Spoons In Pot, Stack White Mugs, Stack Yellow On Red, Stack3 Rubiks Cube, Staples marker · draw smile, Staples marker · write c, Take Measuring Spoon Out, Take Mugs Off Of Shelf, Take Spatula Off Shelf, Throw Away Apple, Throw Away Snacks, Tool Organization, Tool Organization Both, Tools Picking All Hammers, Tools Picking Drill, Tools Picking Hammer, Toy In Bin, Unstack Rubiks Cube, Utensils In Mug, White Mug In Center Of Table, White Mugs In Bin, Wood Spatula To Bowl, Yellow And White Objects In Bin, Yogurt In Bowl

Layout varies¶

gen-layout

Object placements or initial states differ between episodes.

Why it exists: Meta-World's parametric variation; LIBERO-Spatial; RoboWits, whose paper states objects are randomly placed per trial; VLABench's ±5 cm noise.

Tagged on 190 task(s): Adjust bottle, Aero Cube Rotate Z Axis, Align Blocks, Align Chopsticks, Aloha Hand Over, Aloha Single Peg Insertion, Assembling Gift Baskets, Attach a Camera to a Tripod, Balance Board, Ball Into Bottle, Ball Into Jar, Ball Onto Tower, Beat block hammer, Blocks ranking rgb, Blocks ranking size, Boxing Books Up for Storage, Boxing Food After Dinner, Bringing Paper To Recycling, Bringing Water, Bringing in Wood, Can Meat, Canning Food, Carrying Out Garden Furniture, Carrying in Groceries, Chop an Onion, Chopping Wood, Clean A Keyboard, Clean Boxing Gloves, Clean Up Broken Glass, Clean Up Your Desk, Clean Your Rusty Garden Tools, Clean a Patio, Clean a Trumpet, Cleaning Up Branches And Twigs, Cleaning Up Plates and Food, Clearing Food from Table into Fridge, Click alarm clock, Click bell, Collect Screws, Collecting Aluminum Cans, Collecting Children's Toys, Composting Waste, Cook A Brisket, Cook A Frozen Pie, Cook Bacon, Cook Broccolini, Cook Brussels Sprouts, Cook Cabbage, Cook Hot Dogs, Cover With Lid, Cylinder Through Hole, Differentiate Cubes, Dispose Of Batteries, Dispose Of Glass, Dominos, Dump bin big bin, Freeze Fruit, Freeze Pies, Gap Retrieve, Getting Organized For Work, Grab roller, Halve An Egg, Handover block, Handover microphone, Hanging Pictures, Hanging mug, Hiding Easter Eggs, Hold Cup, Installing A Fax Machine, Installing A Modem, Installing A Scanner, Installing Smoke Detectors, Laying Tile Floors, Leap Cube Reorient, Leap Cube Rotate Z Axis, Lift pot, Loading The Car, Make Cabinet Doors, Make Gift Bags For Baby Showers, Make Microwave Popcorn, Make Pizza, Make Rose Centerpieces, Move Cube, Move can pot, Move pill bottle pad, Move playing card away, Move stapler pad, Moving Boxes To Storage, Open laptop, Open microwave, Organizing Art Supplies, Organizing School Stuff, Outfit A Basic Toolbox, Packing Meal For Delivery, Panda Open Cabinet, Panda Pick Cube, Panda Pick Cube Cartesian, Panda Pick Cube Orientation, Panda Robotiq Push Cube, Pick diverse bottles, Pick dual bottles, Picking Up Toys, Picking Up Trash, Pinch Card, Place Book, Place Tall Box, Place a to b left, Place a to b right, Place bread basket, Place bread skillet, Place burger fries, Place can basket, Place cans plastic box, Place container plate, Place dual shoes, Place empty cup, Place fan, Place mouse pad, Place object basket, Place object scale, Place object stand, Place phone stand, Place shoe, Polishing Shoes, Preparing Lunch Box, Press stapler, Put Together A Basic Pruning Kit, Put bottles dustbin, Put object cabinet, Putting Away Halloween Decorations, Putting Away Toys, Putting Dirty Dishes In Sink, Putting Dishes Away After Cleaning, Putting Shoes On Rack, Putting Up Christmas Decorations Inside, Raise Platform, Re Shelving Library Books, Rearrange Your Room, Rearranging Kitchen Furniture, Retrieve Cube, Retrieve Roll, Roll Up Ball, Rotate QR code, Round Dough Sheet, Scan object, Scrubbing Bathroom Floor, Seal Colander, Separate Marbles And Sand, Set Up A Coffee Station In Your Kitchen, Setting Mousetraps, Setting The Fire, Setting The Table, Setup A Bar For A Cocktail Party, Shake bottle, Shake bottle horizontally, Slicing Vegetables, Sorting Books On Shelf, Sorting Bottles Cans And Paper, Sorting Household Items, Sorting Vegetables, Spraying For Bugs, Spraying Fruit Trees, Stabilize Bottle, Stack Bowls, Stack Cubes, Stack blocks three, Stack blocks two, Stack bowls three, Stack bowls two, Stacking Wood, Stamp seal, Stand Bulb, Stand Pages, Store Batteries, Store Honey, Store Produce, Storing Food, Sweeping Garage, Thawing Frozen Food, Tidying Bathroom, Tidying Bedroom, Tidying Living Room, Turn switch, Turning On Radio, Turning Out All Lights Before Sleep, Unloading The Car, Vacuuming Floors, Wash A Baseball Cap, Wash Dog Toys, Water Into Mug

Appearance varies¶

gen-appearance

Lighting, textures, background or camera pose change while the task does not.

Why it exists: THE COLOSSEUM, which found distractor count, target colour and lighting the three most damaging factors; RoboTwin 2.0's five randomisation axes; VLABench's Unseen Texture track.

Not yet tagged on any task.

Objects vary¶

gen-objects

Instances or categories at test time were not in the training set.

Why it exists: LIBERO-Object; VLABench's Cross-Category track over 2,000+ objects; RoboTwin's 731 objects; RoboCasa365's 3,200.

Not yet tagged on any task.

Scenes vary¶

gen-scenes

The same task is posed in different rooms or buildings — different geometry, not just textures.

Why it exists: RoboCasa's 120 kitchens and RoboCasa365's 2,500; BEHAVIOR's 50 scenes.

Not yet tagged on any task.

Robots vary¶

gen-embodiment

The same task is posed on more than one robot, so no kinematic chain can be assumed.

Why it exists: RoboTwin 2.0's five embodiments; Playground's joystick task across eight robots; VLABench's arms, dual arms, humanoids and quadrupeds.

Not yet tagged on any task.

Wording varies¶

gen-language

The instruction is paraphrased, pitched at another level of detail, or corrupted, while the task stays the same. A flag, not a rung: it does not nest with the visual ladder.

A flag, not a rung: it can apply at any level of this ladder.

Why it exists: LIBERO-PRO, where corrupted instructions left policies producing identical trajectories; RoboTwin's language axis; RoboLab's three wordings per task.

Tagged on 120 task(s): Animals In Bin, Apple And Yogurt In Bowl, Bagels On Plate, Banana In Bowl, Banana On Plate, Banana Then Rubiks Cube, Bananas In Bin One More, Bananas In Bin Three Total, Bananas In Crate, Bananas Out Of Bin, Bbq Sauce In Bin, Big Pumpkin In Bin, Black Items In Bin, Block Stacking Order Agnostic, Block Stacking Specified Order, Blocks In Bin, Bowl In Bin, Bowl Stacking Left On Right, Bowl Stacking Right On Left, Butter Above Raisin, Canned Food In Bin, Clamp In Right Bin, Clean Up Toys, Clear Organic Objects, Clutter Plastic, Clutter Pumpkin, Coffee Pot In Bin, Condiments In Bin, Cooking Clear Plate, Cooking Pick Pasta Tool, Cubes And Blocks In Bin, Dishes In Bin, Electronics In Bin, Food Packing By Color, Food Packing1 Boxes, Food Packing1 Cans, Food Packing2 Boxes, Food Packing2 Cans, Food Packing3 Boxes, Food Packing3 Cans, Fruits Green Limes On Plate, Fruits Moving, Fruits Moving Orange Or Lime, Fruits On Plate, Fruits On Plate3, Fruits Onion, Fruits Onion To Plate, Fruits Oranges On Plate, Grab A Bagel, Grab A Fruit, Green Spoons In Pot, Hammers In Left Bin, Jugs On Shelf, Keyboard Out Of Bin, Larger Object Raisin Box In Bin, Marker In Mug, Mouse On Keyboard, Move Banana To Bagel Plate, Mustard Above Raisin, Mustard In Left Bin, Mustard In Right Bin, Non Hammer Tools In Right Bin, One Bottle In Square Pail, One Bottle On Shelf, Phone Or Remote In Bin, Pick Drill, Pick Glasses, Pick Orange Object, Pick Up Blue Pitcher, Pick Up Green Object, Pink Spoon In Pot, Plastic Bottles In Square Pail, Put Bowl On Shelf Top, Put Mugs On Shelf, Put Two Mugs On Shelf, Recycle Carton, Recycle Cartons On Box, Recycle Cartons Vertical Crate, Red Dishes In Bin, Red Items In Bin, Reorient All Mugs, Reorient Jug, Reorient Red Mug, Reorient White Mugs, Rubiks Cube, Rubiks Cube And Banana, Rubiks Cube Behind Bowl, Rubiks Cube In Front Of Bowl, Rubiks Cube Left Of Bowl, Rubiks Cube Or Banana, Rubiks Cube Right Of Bowl, Rubiks Cube Then Banana, Rubiks Cubes In Bin, Sauce Bottles Crate, Small Pumpkin In Bin, Smaller Object Butter In Bin, Smartphone In Bin, Spoon In Mug, Spoons In Pot, Stack White Mugs, Stack Yellow On Red, Stack3 Rubiks Cube, Take Measuring Spoon Out, Take Mugs Off Of Shelf, Take Spatula Off Shelf, Throw Away Apple, Throw Away Snacks, Tool Organization, Tool Organization Both, Tools Picking All Hammers, Tools Picking Drill, Tools Picking Hammer, Toy In Bin, Unstack Rubiks Cube, Utensils In Mug, White Mug In Center Of Table, White Mugs In Bin, Wood Spatula To Bowl, Yellow And White Objects In Bin, Yogurt In Bowl

Label taxonomy¶

Capability¶

Perception & Understanding¶

Planning & Reasoning¶

Control & Coordination¶

Feedback & Adaptation¶

Task Domain¶

Manipulation¶

Locomotion & Stability¶

Navigation & Exploration¶

Mobile / Whole-body Manipulation¶

Interaction & Collaboration¶

Detailed labels¶

Scale¶

Tabletop¶

Room¶

Building¶

Outdoor¶

Body¶

Legged locomotion¶

Rough terrain¶

Dynamic balance¶

Whole-body coordination¶

Bimanual¶

In-hand manipulation¶

Contact¶

Grasp and transport¶

Push, slide, sweep¶

Momentum and timing¶

Open and close¶

Insert and assemble¶

Stack and balance¶

Regulated force¶

Deformable¶

Fluids and granular media¶

Use a tool¶

Object state¶

Operate a device¶

Heat, cook, freeze¶

Cut¶

Treat a surface¶

Goal¶

Geometric constraint¶

Spatial reference¶

Attribute reference¶

Counting and quantifiers¶

Implicit intent¶

World knowledge¶

Structure and risk¶

Atomic¶

Composite¶

Activity¶

Required order¶

Irreversible¶

Perception¶

Partial observability¶

Act to find out¶

Fine visual detail¶

Generalisation¶

Nothing varies¶

Layout varies¶

Appearance varies¶

Objects vary¶

Scenes vary¶

Robots vary¶

Wording varies¶