activity-class.md

Activity Class Definitions

Data was annotated by three expert annotators by reference to the RGB images. Definition of activity class was made with the help of industrial engineers. In the latest version, three types of annotations are available: (1) work operations, (2) action, and (3) outliers.

1. Table of Contents

• 1. Table of Contents
• 2. Work Operations (OPENPACK_OPERATIONS)
• 3. Actions (OPENPACK_ACTIONS)
• 4. Outlier Events (OPENPACK_OUTLIERS)

2. Work Operations (OPENPACK_OPERATIONS)

Operation clases are defined based on an instruction document used in an actual logistics center. The document specifies a sequence of work operation performbed by workers. In the data collection, the subjects basically followed the instructions for packing. Ten operations written in the document are annotated. Here are definitions of the ten operations.

ID	Operation	Description	ignore
100	Picking	Check the order-sheet and go to the back table to pick up the items.
200	Relocate Item Label	Peel off the label from the items and place it on the bottom margin of the order sheet. Check the item names and quantity on the list and label with a ballpoint pen. (When picking is done in one round trip, this action is set as the beginning of the box.)
300	Assemble Box	Assemble cardboard boxes that match the size of the items.
400	Insert Items	Put the goods into the box. Fill the box with air cushion.
500	Close Box	Close the box with craft tape.
600	Attach Box Label	Attach the box number label to the side of the box.
700	Scan Label	Scan barcodes with the handy scanner in the following order: (1) order sheet, (2) box number sticker, (3) item label. Then press the "ESC" button on the handy scanner twice. Next, scan barcodes with the scanner of the label printer in the following order: (1) order sheet, (2) code on the label printer.
800	Attach Shipping Label	Attach the shipping label printed out from the label printer to the box.
900	Put on Back Table	Move the completed packing box under the back table.
1000	Fill out Order	Write their signatures in the confirmation column of the order sheet and insert it into the tray for completed order sheets.
8100	Null	For the operation label, ID8200 is translated into this label (ID8100).	True

In the opnepack-toolkit, this operation class set is implemented as openpack_toolkit.OPENPACK_OPERATIONS. Also, a list of work operations are available in the YAML format. See openpack-toolkit:configs/dataset/annotation/openpack-operations.yaml

3. Actions (OPENPACK_ACTIONS)

In an instruction document, each operation are explained in details. For example, a description of the “relocate item label” operation is given as “Remove the label from the items and place it on the bottom margin of the packaging list. Check the product name and quantity on the list and label with a ballpoint pen.” As you can see, each operation consists of mutiple actions. Many actions are similar to each other but some of them are unique to a specific operatoin. In general, to recognize complex activities such as work operation, understanding the relation among small actions are important. Therefore, to enable research to recognize hierarchical activity classes, we annotated the actions as follows.

ID	Operation	Action	Description	ignore
101	Picking	Pick Up Sheet	Pick up order sheet from the workbench.
103	Picking	Pick Up Item from Box	Pick items out of a box that contains items picked in batches.
107	Picking	Pick Up Box Sheet	Pick up an order sheet and packed boxes at the same time. (This label is used for moving packed boxes and picking items for the next box at the same time.)
108	Picking	Walk to Work Bench	Lift the item and turn back toward the workbench. (When picking is done in one round trip, this action is set as the beginning of the box.)
109	Picking	Put Packed Box	Go to the back table and place the packed box under the table.
201	Relocate Item Label	Remove Item Label	Remove the sticker with the barcode from the each item.
202	Relocate Item Label	Attach to Order Sheet	Attach the label on the bottom margin of the order sheet.
203	Relocate Item Label	Hold Pen
204	Relocate Item Label	Write Check Mark	The subject's checkes items name and quantity.
205	Relocate Item Label	Put Item Small Bag	Insert small items (<10cm) into the small paper bag and close it with tape. The action is almost the same as ID0404 but the timing is different.
301	Assemble Box	Pick Cardboard	Select and pick up boxes that match the size of the items.
302	Assemble Box	Bend Flap	Bend "bottom" 4 flaps of the box.
303	Assemble Box	Attach Tape	Apply craft tape to close the "bottom" of the box.
304	Assemble Box	Turn Over Box	Turn the box upside down so that the side with the craft tape is on the bottom.
305	Assemble Box	Pick Up Assembled Box	Pick up a pre-assembled box. (Used in Scenario.3-5)
401	Insert Items	Insert Item into Box	Subject grabs individual items and inserts them in the assembled box
402	Insert Items	Air Cushion	Insert air cushion to the box.
403	Insert Items	Separate Air Cushion	Tear off the connected cushioning.
404	Insert Items	Put Item Small Bag	Insert small items (<10cm) into the small paper bag and close it with tape. The action is almost the same as ID0205 but the timing is different.
501	Close Box	Bend Flap	Bend 4 flaps in the upper side of the box.
502	Close Box	Attach Tape	Close the box with craft tape.
601	Attach Box Label	Attach Box Label	Place the box number label on the side of the box.
701	Scan Label	Pick Up HT	Pick up handy scanner (HT).
702	Scan Label	Scan Order Sheet	Scan the order ID code in the upper left corner of the order sheet.
703	Scan Label	Scan Box	Scan the box number label.
704	Scan Label	Scan Item	Scan each item number on the label on the order sheet one by one.
705	Scan Label	Hold Scanner	Pick up white scanner (for label printer).
706	Scan Label	Scan Order Sheet	Scan the order ID code in the upper left corner of the order sheet with the white scanner.
707	Scan Label	Scan Printer	Scan the code on the label printer to print out sheiiping labels.
801	Attach Shipping Label	Pick Up Shipping Label	Pick up shipping label from the label printer.
802	Attach Shipping Label	Attach Shipping Label	Attach the printed shipping label to the upper side of the box.
901	Put on Back Table	Pick Up Packed Box	Pick up prepared box from the workbench.
902	Put on Back Table	Put Packed Box	Go to the back table and place the packed box under the table.
1001	Fill out Order	Pick Up Pen	The subject's hand touches the pen.
1002	Fill out Order	Write Sign	Sign each field in the column next to the item list on the order sheet.
1003	Fill out Order	Push Order Sheet into Tray	Insert order sheet into the tray for completed order sheets.
8101	Null	Others		True
8102	Null	System Error		True
8103	Null	Ignore		True
8104	Null	Unknown		True
8105	Null	Null	Segment without any specific actions. This class is prepared for action detection.
8106	Null	Exclude	Segments with this label should be excluded from evaluation. This class is prepared for action detection.	True
8201	System Error	System Error		True

In the opnepack-toolkit, this action class set is implemented as openpack_toolkit.configs.dataset.annotation.OPENPACK_ACTIONS. Also, a list of action labels are available in the YAML format. See openpack-toolkit:configs/dataset/annotation/openpack-actions.yaml

4. Outlier Events (OPENPACK_OUTLIERS)

In principle, a worker repetitively performs a sequence of predefined operations, but mistakes and other unexpected actions occur. Because these outliers can decrease product quality and productivity, managers of industrial sites want to detect such harmful activities. Therefore, OpenPack annotated 45 outlying events over seven categories that are not specified in the work instructions.

We categorize outlying activities as follows:

• (A-a) Incident: This category is assigned to a time segment when a worker makes a mistake or suffers from an accident, e.g., dropping dry cells on the ground.
• (A-b) Investigation: This category is assigned to a segment when a worker investigates the current situation to avoid additional mistakes or accidents after the incident occurs, e.g., a segment when a subject re-confirms the work procedures.
• (A-c) Recovery: Activities to correct errors after a mistake or an accident, e.g., when a worker changes the shipping box size and assembles it again.
• (B) Struggling: Workers sometimes struggle to complete an operation, e.g., the barcode on an item is difficult to scan.
• (C) Confused: When workers lose the next procedure, they sometimes stop working. Segments corresponding to a work stoppage are classified in this category.
• (D) Parallel: Skilled workers can perform two tasks in parallel. This category is assigned to segments with overlapping operation/action labels.
• (E) Additional: Activities that are not included in the work-instruction documents but are necessary to complete shipping orders, such as preparing a new adhesive tape when the old one gets used up.

Note that the outlier categories are defined to classify events that may affect the work activity recognition performance, including mistakes. Some categories are similar to each other; therefore, the differences among the outlier categories are noted below.

• (A-a) Incident, (A-b) investigation, and (A-c) recovery are categories related to mistakes or errors. These categories were assigned to each phase of mistakes, from the action that caused the mistake, confirmation of the situation, to recovery.
• Both (B) struggling and (C) confused are categories for a state, in which the work progress is stalled; however, "struggling" is a category for a state, in which the subject continues to move, while "confused" is a /category for a state, in which the movement has stopped.
• (A-c) Recovery and (E) additional are classified based on whether they are related to work errors or not.
• Actions included in (E) additional are those that are not part of the job description but are important for efficient work, such as cleaning the workbench. There is a difference in the way these actions are executed between skilled workers and others. However, because they are large movements, and thus affect the sensor data, we annotated these actions.

The following table shows the list of annotated outlier events. The above outlying labels contain auxiliary information regarding types of related events as "Tags", in the incident category facilitating analysis of outlying activities.

ID	Category	Event	Description	Tags	ignore	Note
Aa01	Incident	Others	(no specific events)	LOW, HIGH, Single,
Aa02	Incident	Miss	A worker makes a mistake.	LOW, HIGH, Single,
Aa03	Incident	Accident	A worker suffers from an accident.	LOW, HIGH, Single,
Aa04	Incident	System Error	Problems occurred in the experimental environment or in the sensors device side.
Aa05	Incident	Talk to Staff	Talking to staff to inform of problems they faced. For example, box stock is runnning out.
Aa06	Incident	Break Label	The item label is broken when it is peeled off.
Aa07	Incident	Wrong Choice of Box Size	A worker is picking up a box that is not sufficiently large to put all the items in it.	S, M, L, LL,
Ab01	Investigation	Others	(no specific events)	LOW, HIGH,
Ab02	Investigation	System Error	Checking sensor devices or tools such as handheld scanners.
Ab03	Investigation	Check Box Size	Checking to see if all the items fit into the box by, for example, placing the item to the side of the box.	Item, Box-Close, Box-Open, Box-In,
Ab04	Investigation	Check Item Size	Holding items to see their sizes.
Ab05	Investigation	Talk to Staff	Stopping their work and asking how they can recover from mistakes or accidents.
Ac01	Recovery	Others	(no specific events)	LOW, HIGH,
Ac02	Recovery	System Error	Fixing device errors etc.
Ac03	Recovery	Take out Inserted Items from Box	Taking out items from the box to change to a different size box.
Ac04	Recovery	Put Away Box for Resizing	Moving the box to the side of the workbench to discard the box which is of the wrong size.	S, M, L, LL,
Ac05	Recovery	Disassemble Box for Resizing	Removing the attached tapes and disassembling box to use a different sized box	S, M, L, LL,
Ac06	Recovery	Fold Box to Clean Up	Fold the box back into the stock before the bend flap operation.
Ac07	Recovery	Return Box	Put the folded box on the box stock.
Ac08	Recovery	Talk to Staff	Correcting their mistakes while asking the staff what to do.
Ac09	Recovery	Item Label	Correction of errors related to the attachment and removal of item labels.
B01	Struggling	Others	(no specific events)	LOW, HIGH,
B02	Struggling	Look for Items	Stopping in front of the back table and looking for items to pack.
B03	Struggling	Label	Struggling to remove the labels.	Item, Box, Shipping,
B04	Struggling	Scan	Struggling to scan labels.	HT, LP,
B05	Struggling	Air Cushion	Struggling to cut air cushion, or struggling to put air cushions into the box.
B06	Struggling	Insert Items	Struggling to insert items into the box.
B07	Struggling	Tape	Struggling to attach tape to the box.
C01	Confused	Others	(no specific events)	LOW, HIGH,
C02	Confused	Box Size	Stop their movements and think about which box size to pick.
C04	Confused	Process	Almost doing something else that is out of the expected work procedure.
C05	Confused	Double Check	Double check by pointing items with their finger.
D01	Parallel	Others	(no specific events)	LOW, HIGH,
D02	Parallel	Action&Action	Overlap of two action labels.	01011003, 01080201, 03020303, 03030304, 03030401, 03030501, 04010403, 07061001, 07071001, 08011003, 1002*1003,
D03	Parallel	Action&Additional	Overlap of an action label and an outlier/additional label.	0101G06, 0108G06, 0301B03, 0301G05, 0301G06, 03030304G07, 0303G06, 0304G07, 0401G07, 0401G08, 0502G07, 0502G08, 0601G03, 0601G08, 0701G06, 0701G08, 0702G06, 0703G08, 0704G08, 0707G06, 0707G08, 0801G03, 0801G06, 0801G08, 0901G06, 1001G06, 1003G06,
D04	Parallel	Action&Others	Overlap of an action label and an other label in any category.	107, 030, 109, 011, 001,
D05	Parallel	Additional&Additional	Overlap of two outlier/additional labels.	B01G07, C03C04, G06G08, G03G06,
D06	Parallel	Additional&Others	Overlap of an outlier/additional label and an other label/s.
E01	Additional	Others	(no specific events)	LOW, HIGH,
E02	Additional	Talk to Staff	Stopping their work and talking with staff.
E03	Additional	Dispose	Discarding the trash of box labels.
E05	Additional	Refill Boxe Stock	Move additional box stock to the workbench.	S-D, S, M, L, LL,
E06	Additional	Clean Up the Workbench	Clean up the workbench. For example, organizing order sheets.
E07	Additional	Lift Up Box	Lifting up the assembled box.
E08	Additional	Move Box	Turning and moving the assembled box on the workbench.

In the opnepack-toolkit, this outlier event set is implemented as openpack_toolkit.configs.dataset.annotation.OPENPACK_OUTLIERS. Also, a list of action labels are available in the YAML format. See openpack-toolkit:configs/dataset/annotation/openpack-actions.yaml