Synthetic Action Simulation Platform for Elderly Action Data Generation
The world’s elderly population growth emphasizes the necessity of eldercare technologies and underlines the role of action recognition tasks to comprehend elders’ activities of daily living. However, most public datasets used in human action recognition either differ from or have limited coverage of elders’ activities in many aspects. Moreover, data acquisition of elders’ ADL is challenging due to the privacy and physical limitations of the elderly.
We introduce ElderSim, a synthetic action simulation platform that can generate synthetic data on elders’ daily activities. For 55 kinds of frequent daily activities of the elders, ElderSim generates realistic motions of synthetic characters with several customizable data-generating options and provides several output modalities. We also provide KIST SynADL dataset which is generated from our simulation platform.
Development
Unreal Engine 4
We chose the Unreal Engine 4 (UE4) as our main rendering engine for real-time photorealistic rendering.
Background
We have modeled four residential houses based on their indoor measurements using Maya. The household background has become visually more realistic by using physics-based materials and the Post-Process Volume function of UE4.
Character
Fifteen subjects, including thirteen elders, have been recruited to sufficiently represent a variety of body shapes and appearances. Their body shapes have been captured from Kinect and utilized to design the body shape of synthetic characters in Maya.
Motion
In ElderSim, we provide motions for 55 activity classes. Based on the Motion Capture data obtained from the subjects, motion data have been rigged in Maya. The motions for synthetic characters are then generated by adjusting the template’s kinematic parameters to those of each character and playing the constructed movements.
| ID | Action descripction | ID | Action descripction |
|---|---|---|---|
| 1 | eating food with a fork | 29 | hanging out laundry |
| 2 | pouring water into a cup | 30 | looking around for something |
| 3 | taking medicine | 31 | using a remote control |
| 4 | drinking water | 32 | reading a book |
| 5 | putting food in the fridge/taking food from the fridge | 33 | reading a newspaper |
| 6 | trimming vegetables | 34 | handwriting |
| 7 | peeling fruit | 35 | talking on the phone |
| 8 | using a gas stove | 36 | playing with a mobile phone |
| 9 | cutting vegetable on the cutting board | 37 | using a computer |
| 10 | brushing teeth | 38 | smoking |
| 11 | washing hands | 39 | clapping |
| 12 | washing face | 40 | rubbing face with hands |
| 13 | wiping face with a towel | 41 | doing freehand exercise |
| 14 | putting on cosmetics | 42 | doing neck roll exercise |
| 15 | putting on lipstick | 43 | massaging a shoulder oneself |
| 16 | brushing hair | 44 | taking a bow |
| 17 | blow drying hair | 45 | talking to each other |
| 18 | putting on a jacket | 46 | handshaking |
| 19 | taking off a jacket | 47 | hugging each other |
| 20 | putting on/taking off shoes | 48 | fighting each other |
| 21 | putting on/taking off glasses | 49 | waving a hand |
| 22 | washing the dishes | 50 | flapping a hand up and down (beckoning) |
| 23 | vacuumming the floor | 51 | pointing with a finger |
| 24 | scrubbing the floor with a rag | 52 | opening the door and walking in |
| 25 | wipping off the dinning table | 53 | fallen on the floor |
| 26 | rubbing up furniture | 54 | sitting up/standing up |
| 27 | spreading bedding/folding bedding | 55 | lying down |
| 28 | washing a towel by hands |
Viewpoint
ElderSim contains viewpoints reflecting eldercare applications. To implement the various viewpoints, we define UE4 splines that contain several vertical cameras and position these splines based on the user parameters.
Lighting
To simulate the effect of sunlight over time, we utilize the SkySphere Blueprint function of UE4 and provide an adjustable time parameter in 100 levels to vary sunlight. Indoor light sources are placed according to lighting layouts of actual houses. Finer rendering effects are applied by the Post-Process Volume effect of UE4.
Features
- Customizable parameters
ElderSim is provided with an intuitive graphical user interface (GUI) to select data-generating options as needed. - Large-scale labeled data generation
The data are generated and recorded according to all possible combinations of options in ElderSim. The total number of samples can be calculated with a simple multiplication.
Total Samples=actions×subjects×backgrounds×viewpoints×light conditions×objects - Various modality outputs
ElderSim provides video resolutions and frame rates of up to 1920x1080 and 60 FPS, respectively. Modalities of RGB video, 2D, and 3D skeleton are provided. - KIST SynADL Dataset
Based on the above developmental features of ElderSim, we generate KIST SynADL, a large-scale synthetic dataset of elders’ activities considering eldercare applications. KIST SynADL provides 462k RGB videos and 2D, 3D skeleton data, covering 55 action classes, 28 camera viewpoints, 15 characters, five lighting conditions, four backgrounds, and a single object.
| Feature | Number |
|---|---|
| Samples | 462,200 |
| Classes | 55 |
| Subjects | 15 |
| Viewpoints | 28 |
| Backgrounds | 4 |
| Objects | 1 |
Publications
If you use either the platform or the dataset, please refer to the following technical report: PUBLICATION (arXiv)
Download
● ElderSim
The ElderSim platform can be downloaded by clicking here.
● KIST SynADL
The KIST SynADL dataset can be downloaded by clicking here.
| Modality | Details | Format | Size |
|---|---|---|---|
| RGB Video | 640x360 resolution | AVI | 433 GB |
| 2D Skeleton | 25 joints (OpenPose) | JSON | 18.4 GB |
| 2D Skeleton | 25 joints (Kinect) | JSON | 24.4 GB |
| 3D Skeleton | 25 joints (OpenPose) | JSON | 18.5 GB |
| 3D Skeleton | 25 joints (Kinect) | JSON | 24.4 GB |
| Total | 518.7 GB |
Contact
Please contact jhcho@kist.re.kr if you have any questions or comments.
Acknowledgment
- This work was supported by the Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 2017-0-00162, Development of Human-care Robot Technology for Aging Society)