Event file restructuring¶
This tutorial works through the process of restructuring event files using the HED event remodeling tools. The tools, which are written in Python, are designed to be run on an entire BIDS dataset. The tools can be called in Jupyter notebook or run via command-line scripts.
-
Add structure column Docs in process
Add structure events Docs in process
Add structure numbers Docs in process
What is event file restructuring?¶
The event files in an experiment provide a crucial link between what happens in the experiment and the experimental data by providing identified time markers linked to the timeline of the experiment.
Event files are often initially created using information in the logs files generated by the experiment’s presentation software or other control software. These event files are then used to identify portions of the data corresponding to particular points or blocks of data to be analyzed or compared.
Event file restructuring refers to creating, modifying, and
reorganizing the event markers in tabular files in order to
disambiguate or clarify the information for distribution and analysis.
Restructuring can occur at several stages during the acquisition and processing
of experimental data as shown in this schematic diagram:
.
In addition to restructuring during initial creation of the tabular event files, restructuring may be required when the event files do not conform to the requirements of a particular analysis. Thus, restructuring is an iterative process, which is supported by the HED remodeling tools for datasets with tabular event files.
Table 1 gives a summary of the tools available in the HED remodeling toolbox.
Category |
Command |
Example use case |
|---|---|---|
clean-up |
||
remove_columns |
Remove temporary columns created during restructuring. |
|
remove_rows |
Remove rows with n/a values in a specified column. |
|
rename_columns |
Make columns names consistent across a dataset. |
|
reorder_columns |
Make column order consistent across a dataset. |
|
factor |
||
factor_column |
Extract factor vectors from a column of condition variables. |
|
factor_hed_tags |
Extract factor vectors from search queries of HED annotations. |
|
factor_hed_types |
Extract design matrices and/or condition variables. |
|
restructure |
||
remap_columns |
Create m columns from values in n columns (for recoding). |
|
split_event |
Split trial-encoded rows into multiple events. |
|
merge_consecutive |
Replace multiple consecutive events of the same type |
|
add_structure_column |
Add a column with condition names. |
|
add_structure_rows |
Add a row representing the start of a block or trial. |
|
add_structure_numbers |
Add a column with trial or block numbers. |
The clean-up commands are used at various phases of restructuring to assure consistency across event files in the dataset.
The factor commands produce column vectors of the same length as the events file that encode condition variables, design matrices, or satisfaction of other search criteria. See the HED conditions and design matrices for more information on factoring and analysis.
The restructure commands modify the way that events are encoded.
Installing remodeling tools¶
Currently, the remodeling tools are available in the GitHub hed-curation repository along with other tools for data cleaning and curation. These tools rely on the hedtools library which is available on PyPI and can be installed via PIP.
Once the HED remodeling tools are available in final form, the tools will be moved to the GitHub hed-python repository and be available for installation via PyPI. At that time, versions for single event files will become available as a web-service and through a web-interface on the HED online tools. A docker version is also under development.
In the meantime, if you want to run the latest version of the tools, you will need to install the development branch of both hed-python and hed-curation using:
pip install git+https://github.com/hed-standard/hed-python/@develop
pip install git+https://github.com/hed-standard/hed-curation/@develop
Running remodeling scripts¶
Remodeling consists of applying a list of commands to an events file to restructure or modify it in some way.
The following diagram shows a schematic of the remodeling process.
Initially, the user creates a backup of the event files. Restructuring applies a sequence of remodeling commands given in a JSON transformation file to produce a final result. The transformation file provides a record of the operations performed on the file. If the user detects a mistake in the transformation, he/she can correct the transformation file and restore the backup to rerun.
The remodeling toolbox provides several scripts to apply the transformations to the files in a BIDS-formatted dataset. The basic scripts are summarized in Table 2.
Script name |
Arguments |
Purpose |
|---|---|---|
run_backup |
bids_dir |
Backup the event files. |
run_remodel |
bids_dir |
Remodel the event files. |
run_restore |
bids_dir |
Restore the event files. |
run_remove |
bids_dir |
Remove the backup event files. |
All the scripts have a required parameter which is the full path of the BIDS dataset root. The -t task_name option specifies which task in the dataset the remodeling should apply to. Often, when a dataset includes multiple tasks, the event files are structured differently for each task and thus require different transformation files.
The other -e exclude_dirs gives a list of directories to ignore in searching for event file.
In BIDS, typical directories to exclude are derivatives, code, and stimulus_files.
Backing up the events¶
Before any remodeling is performed, you should always back up the event files. Usually this is just done once, before any remodeling is done. There are two strategies for doing the backup: in-place and full-tree.
The in-place strategy makes a copy of each event file in the same directory
as the event file using the suffix _eventsorig.tsv to distinguish
these backup files from the active event files (which end in _events.tsv).
The in-place strategy is simple, but the extra files prevent the dataset from
validating with the BIDS validator.
The full-tree strategy creates a complete tree backup in the
code/event_backup directory of the BIDS dataset.
This strategy is more expensive, but does not prevent validation
or interfere with other processing.
Example command to backup the events.
python run_backup.py t:\ds002790-data -b full-tree -e derivatives code simulus_files
Remodeling the events¶
The event remodeling process is given by the following example:
Example command to remodel the events.
python run_remodel.py t:\ds002790-data -m derivatives/models/simple_rmdl.json -e derivatives code simulus_files
Example remodeling file with commands to remove columns and reorder the rest.
[
{
"command": "remove_columns",
"description": "Get rid of the sample and the value columns.",
"parameters": {
"remove_names": ["sample", "value"],
"ignore_missing": true
}
},
{
"command": "reorder_columns",
"description": "Want event_type and task_role columns after onset and duration.",
"parameters": {
"column_order": ["onset", "duration", "event_type", "task_role"],
"ignore_missing": false,
"keep_others": true
}
}
]
Remodeling operations¶
The examples in this tutorial use the following excerpt of the stop-go task from sub-0013 of the AOMIC-PIOP2 dataset available on OpenNeuro as ds002790. The full events file is sub-0013_task-stopsignal_acq-seq_events.tsv.
Table 3: Excerpt from an event file from the stop-go task of AOMIC-PIOP2 (ds002790).
onset |
duration |
trial_type |
stop_signal_delay |
response_time |
response_accuracy |
response_hand |
sex |
|---|---|---|---|---|---|---|---|
0.0776 |
0.5083 |
go |
n/a |
0.565 |
correct |
right |
|
5.5774 |
0.5083 |
unsuccesful_stop |
0.2 |
0.49 |
correct |
right |
female |
9.5856 |
0.5084 |
go |
n/a |
0.45 |
correct |
right |
female |
13.5939 |
0.5083 |
succesful_stop |
0.2 |
n/a |
n/a |
n/a |
female |
17.1021 |
0.5083 |
unsuccesful_stop |
0.25 |
0.633 |
correct |
left |
male |
21.6103 |
0.5083 |
go |
n/a |
0.443 |
correct |
left |
male |
The factor_hed_types and factor_hed_tags also require HED annotations of the events and require a JSON sidecar that includes HED annotations. The tutorial uses the following JSON excerpt to illustrate these operations. The full JSON file can be found at: task-stopsiqnal_acq-seq_events.json. These HED commands also require the HED schema. The tutorials use the latest version that is downloaded from the web.
Excerpt of JSON sidecar with HED annotations for the stop-go task of AOMIC-PIOP2.
{
"trial_type": {
"HED": {
"succesful_stop": "Sensory-presentation, Visual-presentation, Correct-action, Image, Label/succesful_stop",
"unsuccesful_stop": "Sensory-presentation, Visual-presentation, Incorrect-action, Image, Label/unsuccesful_stop",
"go": "Sensory-presentation, Visual-presentation, Image, Label/go"
}
},
"stop_signal_delay": {
"HED": "(Auditory-presentation, Delay/# s)"
},
"sex": {
"HED": {
"male": "Def/Male-image-cond",
"female": "Def/Female-image-cond"
}
},
"hed_defs": {
"HED": {
"def_male": "(Definition/Male-image-cond, (Condition-variable/Image-sex, (Male, (Image, Face))))",
"def_female": "(Definition/Female-image-cond, (Condition-variable/Image-sex, (Female, (Image, Face))))"
}
}
}
Add structure column¶
NOT WRITTEN - PLACEHOLDER
Add a column of numbers corresponding to a structure elements such as trials or blocks.
Table 4: Parameters for the add_structure_column command.
Parameter |
Type |
Description |
|---|---|---|
column_name |
str |
The name of the column to be created or modified. |
source_columns |
list |
Names of the columns to be used for remapping. |
mapping |
dict |
The keys are the values to be placed in the derived columns and the values are each an array |
The add_structure_column command in the following example specifies …
An example add_structure_column command.
{
"column_name": "add_structure_column",
"source_columns": ["response_accuracy", "response_hand"],
"mapping": {
"left": [["correct", "left"], ["incorrect", "right"]],
"right": [["correct", "right"], ["incorrect", "left"]]
}
}
The results of executing this add_structure_column command on the sample events file are:
Table 5: Results of the previous add_structure_column command.
onset |
duration |
trial_type |
stop_signal_delay |
response_time |
response_accuracy |
response_hand |
sex |
|---|---|---|---|---|---|---|---|
0.0776 |
0.5083 |
go |
n/a |
0.565 |
correct |
right |
female |
5.5774 |
0.5083 |
unsuccesful_stop |
0.2 |
0.49 |
correct |
right |
female |
9.5856 |
0.5084 |
go |
n/a |
0.45 |
correct |
right |
female |
13.5939 |
0.5083 |
succesful_stop |
0.2 |
n/a |
n/a |
n/a |
female |
17.1021 |
0.5083 |
unsuccesful_stop |
0.25 |
0.633 |
correct |
left |
male |
21.6103 |
0.5083 |
go |
n/a |
0.443 |
correct |
left |
male |
Add structure events¶
NOT WRITTEN - PLACEHOLDER
Add events representing the start of a structural element such as a trial or a block.
Table 6: Parameters for the add_structure_events command.
Parameter |
Type |
Description |
|---|---|---|
column_name |
str |
The name of the column to be created or modified. |
source_columns |
list |
Names of the columns to be used for remapping. |
mapping |
dict |
The keys are the values to be placed in the derived columns and the values are each an array |
The add_structure_events command in the following example specifies …
An example add_structure_events command.
{
"column_name": "add_structure_events",
"source_columns": ["response_accuracy", "response_hand"],
"mapping": {
"left": [["correct", "left"], ["incorrect", "right"]],
"right": [["correct", "right"], ["incorrect", "left"]]
}
}
The results of executing this add_structure_events command on the sample events file are:
Table 7: Results of the previous add_structure_events command.
onset |
duration |
trial_type |
stop_signal_delay |
response_time |
response_accuracy |
response_hand |
sex |
|---|---|---|---|---|---|---|---|
0.0776 |
0.5083 |
go |
n/a |
0.565 |
correct |
right |
female |
5.5774 |
0.5083 |
unsuccesful_stop |
0.2 |
0.49 |
correct |
right |
female |
9.5856 |
0.5084 |
go |
n/a |
0.45 |
correct |
right |
female |
13.5939 |
0.5083 |
succesful_stop |
0.2 |
n/a |
n/a |
n/a |
female |
17.1021 |
0.5083 |
unsuccesful_stop |
0.25 |
0.633 |
correct |
left |
male |
21.6103 |
0.5083 |
go |
n/a |
0.443 |
correct |
left |
male |
Add structure numbers¶
NOT WRITTEN - PLACEHOLDER
Add a column with numbers corresponding to a structural element.
TODO clarify the difference between add_structure_numbers and add_structure_column.
Table 8: Parameters for the add_structure_numbers command.
Parameter |
Type |
Description |
|---|---|---|
column_name |
str |
The name of the column to be created or modified. |
source_columns |
list of str |
A list of columns to be used for remapping. |
mapping |
dict |
The keys are the values to be placed in the derived columns and the values are each an array |
The add_structure_numbers command in the following example specifies …
An example add_structure_numbers command.
{
"command": "add_structure_numbers"
"description": "xxx"
"parameters": {
"column_name": "match_side",
"source_columns": ["response_accuracy", "response_hand"],
"mapping": {
"left": [["correct", "left"], ["incorrect", "right"]],
"right": [["correct", "right"], ["incorrect", "left"]]
}
}
}
The results of executing this add_structure_numbers command on the sample events file are:
Table 9: Results of executing the previous add_structure_numbers command.
onset |
duration |
trial_type |
match_side |
stop_signal_delay |
response_time |
response_accuracy |
response_hand |
sex |
|---|---|---|---|---|---|---|---|---|
0.0776 |
0.5083 |
go |
right |
n/a |
0.565 |
correct |
right |
female |
5.5774 |
0.5083 |
unsuccesful_stop |
right |
0.2 |
0.49 |
correct |
right |
female |
9.5856 |
0.5084 |
go |
n/a |
0.45 |
correct |
right |
female |
|
13.5939 |
0.5083 |
succesful_stop |
0.2 |
n/a |
n/a |
right |
female |
|
17.1021 |
0.5083 |
unsuccesful_stop |
0.25 |
0.633 |
correct |
left |
male |
|
21.6103 |
0.5083 |
go |
n/a |
0.443 |
correct |
left |
male |
Factor column¶
For each of the specified values in the indicated column create a column containing 1’s and 0’s indicating presence or absence of the value. If no values are specified, all unique values in that column are factored.
Table 10: Parameters for the factor_column command.
Parameter |
Type |
Description |
|---|---|---|
column_name |
str |
The name of the column to be factored. |
factor_values |
list |
Column values to be included as factors. |
factor_names |
list |
Column names for created factors. Must be the same length as factor_values. |
ignore_missing |
bool |
If true, include columns for factor values not in events. |
overwrite_existing |
bool |
If true, existing factor columns are overwritten. |
The factor_column command in the following example specifies that factor columns should be created for succesful_stop and unsuccesful_stop of the trial_type column. The resulting columns are called stopped and stop_failed, respectively.
If the factor_values is an empty list, factors are created for all unique values in the column_name column. The factor_names parameters must be the same length as factor_values. If factor_names is empty, the newly created columns are of the form column_name.factor_value.
A sample factor_column command.
{
"command": "factor_column"
"description": "Create factors for the succesful_stop and unsuccesful_stop values."
"parameters": {
"column_name": "trial_type",
"factor_values": ["succesful_stop", "unsuccesful_stop"],
"factor_names": ["stopped", "stop_failed"],
"ignore_missing": false,
"overwrite_existing": true
}
}
The results of executing this factor_column command on the sample events file are:
Table 11: Results of factoring column XXX.
onset |
duration |
trial_type |
stop_signal_delay |
response_time |
response_accuracy |
response_hand |
sex |
stopped |
stop_failed |
|---|---|---|---|---|---|---|---|---|---|
0.0776 |
0.5083 |
go |
n/a |
0.565 |
correct |
right |
female |
0 |
0 |
5.5774 |
0.5083 |
unsuccesful_stop |
0.2 |
0.49 |
correct |
right |
female |
0 |
1 |
9.5856 |
0.5084 |
go |
n/a |
0.45 |
correct |
right |
female |
0 |
0 |
13.5939 |
0.5083 |
succesful_stop |
0.2 |
n/a |
n/a |
n/a |
female |
1 |
0 |
17.1021 |
0.5083 |
unsuccesful_stop |
0.25 |
0.633 |
correct |
left |
male |
0 |
1 |
21.6103 |
0.5083 |
go |
n/a |
0.443 |
correct |
left |
male |
0 |
0 |
Factor HED type¶
The factor HED type operation produces factor columns in an event file based on values of the specified HED type tag. The most common type is the HED Condition-variable tag, which corresponds to factor vectors based in the experimental design. Other commonly use type tags include Task, Control-variable, and Time-block.
We assume that the dataset has been annotated using HED tags to properly document the experiment conditions and focus on how such an annotated dataset can be used with event remodeling to produce factor columns in the event file corresponding to these condition variables.
For additional information on how to encode experimental designs using HED please see HED conditions and design matrices.
Table 14: Parameters for factor_hed_type command.
Parameter |
Type |
Description |
|---|---|---|
type_tag |
str |
HED tag used to find the factors (most commonly Condition-variable). |
type_values |
list |
Values to factor for the type_tag. |
overwrite_existing |
bool |
If true, existing factor columns are overwritten. |
factor_encoding |
str |
Indicates type of encoding. Valid encodings are ‘categorical’ and ‘one-hot’. |
To simplifyThe factor_hed_type command in the following example specifies …
Example factor_hed_type command.
{
"command": "factor_hed_type"
"description": "Factor based on the sex of the images being presented."
"parameters": {
"type_tag": "Condition-variable",
"type_values": [],
"overwrite_existing": true,
"factor_encoding": "one-hot"
}
}
In order to use the JSON file. The full file is at:
The results of executing this factor_hed-tags command on the sample events file using the sample sidecar file for HED annotations are:
Table 15: Results of factor_hed_type.
onset |
duration |
trial_type |
stop_signal_delay |
response_time |
response_accuracy |
response_hand |
sex |
Image-sex.Female-image-cond |
Image-sex.Male-image-cond |
|---|---|---|---|---|---|---|---|---|---|
0.0776 |
0.5083 |
go |
n/a |
0.565 |
correct |
right |
female |
1 |
0 |
5.5774 |
0.5083 |
unsuccesful_stop |
0.2 |
0.49 |
correct |
right |
female |
1 |
0 |
9.5856 |
0.5084 |
go |
n/a |
0.45 |
correct |
right |
female |
1 |
0 |
13.5939 |
0.5083 |
succesful_stop |
0.2 |
n/a |
n/a |
n/a |
female |
1 |
0 |
17.1021 |
0.5083 |
unsuccesful_stop |
0.25 |
0.633 |
correct |
left |
male |
0 |
1 |
21.6103 |
0.5083 |
go |
n/a |
0.443 |
correct |
left |
male |
0 |
1 |
Merge consecutive¶
Sometimes in experimental logs, a single long event is represented by multiple repeat events. Merges these same events occurring consecutively into one event with duration of the new event updated to encompass the extent of the merged events..
Table 16: Parameters for the merge_consecutive command.
Parameter |
Type |
Description |
|---|---|---|
column_name |
str |
The name of the column to be merged. |
event_code |
str, int, float |
The value in column_name . |
match_columns |
list |
Columns of that must match to collapse events. |
set_durations |
bool |
If True, set durations based on merged events. |
ignore_missing |
bool |
If True, missing column_name or match_columns does not raise an error. |
The merge_consecutive command in the following example specifies …
A sample merge_consecutive command.
{
"command": "merge_consecutive"
"description": "Merge consecutive *succesful_stop* events that match the *match_columns."
"parameters": {
"column_name": "trial_type",
"event_code": "succesful_stop",
"match_columns": ["stop_signal_delay", "response_hand", "sex"],
"set_durations": true,
"ignore_missing": true
}
}
The follo
Table 17: Input file for a merge_consecutive command.
onset |
duration |
trial_type |
stop_signal_delay |
response_hand |
sex |
|---|---|---|---|---|---|
0.0776 |
0.5083 |
go |
n/a |
right |
female |
5.5774 |
0.5083 |
unsuccesful_stop |
0.2 |
right |
female |
9.5856 |
0.5084 |
go |
n/a |
right |
female |
13.5939 |
0.5083 |
succesful_stop |
0.2 |
n/a |
female |
14.2 |
0.5083 |
succesful_stop |
0.2 |
n/a |
female |
15.3 |
0.5083 |
succesful_stop |
0.2 |
n/a |
female |
17.3 |
0.5083 |
succesful_stop |
0.25 |
n/a |
female |
19.0 |
0.5083 |
succesful_stop |
0.25 |
n/a |
female |
21.1021 |
0.5083 |
unsuccesful_stop |
0.25 |
left |
male |
22.6103 |
0.5083 |
go |
n/a |
left |
male |
The results of executing the previous merge_events command on the sample events file are:
Table 18: The results of the merge_events command.
| onset | duration | trial_type | stop_signal_delay | response_hand | sex | | —– | ——– | ———- | —————— | ——– —- | — | | 0.0776 | 0.5083 | go | n/a | right | female | | 5.5774 | 0.5083 | unsuccesful_stop | 0.2 | right | female | | 9.5856 | 0.5084 | go | n/a | right | female | | 13.5939 | 2.2144 | succesful_stop | 0.2 | n/a | female | | 17.3 | 2.2083 | succesful_stop | 0.25 | n/a | female | | 21.1021 | 0.5083 | unsuccesful_stop | 0.25 | left | male | | 22.6103 | 0.5083 | go | n/a | left | male]
Remap columns¶
This command maps the values that appear in a specified m columns of an event file into values in n columns using a defined mapping. This command is often used for recoding the event file. The mapping should have targets for all combinations of values that appear in the m columns.
Table 19: Parameters for the remap_columns command.
Parameter |
Type |
Description |
|---|---|---|
source_columns |
list |
Columns with combinations of values to be mapped. |
destination_columns |
list |
Columns to be mapped into. |
map_list |
list |
A list. Each element consists n + m elements |
ignore_missing |
bool |
If false, a combination of |
The map_list parameter specifies how each unique combination of values from the source columns will be mapped into the destination columns. If there are m source columns and n destination columns, then each entry in *map_list must be a list with n + m elements.
The remap_columns command in the following example creates a new column called response_type based on the unique values in the combination of columns response_accuracy and response_hand.
An example remap_columns command.
{
"command": "remap_columns",
"description": "Map response_accuracy and response hand into a single column.",
"parameters": {
"source_columns": ["response_accuracy", "response_hand"],
"destination_columns": ["response_type"],
"map_list": [["correct", "left", "correct_left"],
["correct", "right", "correct_right"],
["incorrect", "left", "incorrect_left"],
["incorrect", "right", "incorrect_left"],
["n/a", "n/a", "n/a"]],
"ignore_missing": true
}
}
The results of executing the previous remap_column command on the sample events file are:
Table 20: Mapping columns response_accuracy and response_hand into a response_type column.
onset |
duration |
trial_type |
stop_signal_delay |
response_time |
response_accuracy |
response_hand |
sex |
response_type |
|---|---|---|---|---|---|---|---|---|
0.0776 |
0.5083 |
go |
n/a |
0.565 |
correct |
right |
female |
correct_right |
5.5774 |
0.5083 |
unsuccesful_stop |
0.2 |
0.49 |
correct |
right |
female |
correct_right |
9.5856 |
0.5084 |
go |
n/a |
0.45 |
correct |
right |
female |
correct_right |
13.5939 |
0.5083 |
succesful_stop |
0.2 |
n/a |
n/a |
n/a |
female |
n/a |
17.1021 |
0.5083 |
unsuccesful_stop |
0.25 |
0.633 |
correct |
left |
male |
correct_left |
21.6103 |
0.5083 |
go |
n/a |
0.443 |
correct |
left |
male |
correct_left |
Typically, the remap_columns command is used often used to map single codes in the experimental log into multiple columns in the final events file.
Remove columns¶
Remove the specified columns if present.
If one of the specified columns is not in the file and the ignore_missing
parameter is false, a KeyError is raised for missing column.
Table 21: Parameters for the remove_columns operation.
Parameter |
Type |
Description |
|---|---|---|
remove_names |
list of str |
A list of columns to remove. |
ignore_missing |
boolean |
If true, missing columns are ignored, otherwise raise an error. |
The remove_column command in the following example removes the stop_signal_delay and response_accuracy columns. The face column is not in the dataframe, but it is ignored, since ignore_missing is True.
An example remove_columns command.
{
"command": "remove_columns",
"description": "Remove columns before the next step.",
"parameters": {
"remove_names": ["stop_signal_delay", "response_accuracy", "face"],
"ignore_missing": true
}
}
The results of executing this command on the
sample events file
are shown below.
Although face is not the name of a column in the dataframe,
it is ignored because ignore_missing is true.
If ignore_missing had been false, a KeyError would have been generated.
Table 22: Results of executing the remove_column.
onset |
duration |
trial_type |
response_time |
response_hand |
sex |
|---|---|---|---|---|---|
0.0776 |
0.5083 |
go |
0.565 |
right |
female |
5.5774 |
0.5083 |
unsuccesful_stop |
0.49 |
right |
female |
9.5856 |
0.5084 |
go |
0.45 |
right |
female |
13.5939 |
0.5083 |
succesful_stop |
n/a |
n/a |
female |
17.1021 |
0.5083 |
unsuccesful_stop |
0.633 |
left |
male |
21.6103 |
0.5083 |
go |
0.443 |
left |
male |
Remove rows¶
Remove rows in which the named column has one of the specified values.
Table 23: Parameters for remove_rows.
Parameter |
Type |
Description |
|---|---|---|
column_name |
str |
The name of the column to be tested. |
remove_values |
list |
A list of values to be tested for removal. |
The following example remove_rows command removes the rows whose trial_type column has either succesful_stop or unsuccesful_stop.
Sample remove_rows command.
{
"command": "remove_rows",
"description": "Remove rows where trial_type is either succesful_stop or unsuccesful_stop.",
"parameters": {
"column_name": "trial_type",
"remove_values": ["succesful_stop", "unsuccesful_stop"]
}
}
The results of executing the previous remove_rows command on the sample events file are:
Table 24: The results of executing the previous remove_rows command.
onset |
duration |
trial_type |
stop_signal_delay |
response_time |
response_accuracy |
response_hand |
sex |
|---|---|---|---|---|---|---|---|
0.0776 |
0.5083 |
go |
n/a |
0.565 |
correct |
right |
female |
9.5856 |
0.5084 |
go |
n/a |
0.45 |
correct |
right |
female |
21.6103 |
0.5083 |
go |
n/a |
0.443 |
correct |
left |
male |
After removing rows with trial_type equal to succesful_stop or unsuccesful_stop only the three go trials remain.
Rename columns¶
Rename columns by providing a dictionary of old names to new names.
Table 25: Parameters for rename_columns.
Parameter |
Type |
Description |
|---|---|---|
column_mapping |
dict |
The keys are the old column names and the values are the new names. |
ignore_missing |
bool |
If false, a key error is raised if a dictionary key is not a column name. |
The rename_columns command in the following example specifies that response_hand column be
renamed hand_used and that the sex column be renamed image_sex.
The face entry in the mapping will be ignored because ignore_missing is true.
If ignore_missing is false, a KeyError exception is raised if a column specified in
the mapping does not correspond to a column name in the dataframe.
Example rename_columns command.
{
"command": "rename_columns",
"description": "Remove columns before splitting events.",
"parameters": {
"column_mapping": {
"random_column": "new_random_column",
"stop_signal_delay": "stop_delay",
"response_hand": "hand_used",
"sex": "image_sex"
},
"ignore_missing": true
}
}
The results of executing the previous rename_columns command on the sample events file are:
Table 26: After the rename_columns command is executed, the sample events file is:
onset |
duration |
trial_type |
stop_delay |
response_time |
response_accuracy |
hand_used |
image_sex |
|---|---|---|---|---|---|---|---|
0.0776 |
0.5083 |
go |
n/a |
0.565 |
correct |
right |
female |
5.5774 |
0.5083 |
unsuccesful_stop |
0.2 |
0.49 |
correct |
right |
female |
9.5856 |
0.5084 |
go |
n/a |
0.45 |
correct |
right |
female |
13.5939 |
0.5083 |
succesful_stop |
0.2 |
n/a |
n/a |
n/a |
female |
17.1021 |
0.5083 |
unsuccesful_stop |
0.25 |
0.633 |
correct |
left |
male |
21.6103 |
0.5083 |
go |
n/a |
0.443 |
correct |
left |
male |
Reorder columns¶
Reorder the columns in the specified order. If ignore_missing is true, and items in the reorder list do not exist in the file, these columns are ignored. On the other hand, if ignore_missing is false, a column name not appearing in the reorder list causes a ValueError to be raised.
The keep_others parameter controls whether or not columns in the dataframe that do not appear in the reorder list are dropped (keep_others is false) or put at the end of the dataframe in the order they appear (keep_others is true).
Table 27: Parameters for reorder_columns.
Parameter |
Type |
Description |
|---|---|---|
column_order |
list |
A list of columns in the order they should appear in the data. |
ignore_missing |
bool |
If true and a column in column_order does not appear in the dataframe |
keep_others |
bool |
If true, existing columns that aren’t in column_order |
The reorder_columns command in the following example specifies that the first four columns of the dataset should be: onset, duration, trial_type, response_hand, and response_time. Since ignore_missing is true, these will be the only columns retained.
Example reorder_columns command.
{
"command": "reorder_columns",
"description": "Reorder columns.",
"parameters": {
"column_order": ["onset", "duration", "response_time", "trial_type"],
"ignore_missing": true,
"keep_others": false
}
}
The results of executing the previous reorder_columns command on the sample events file are:
Table 28: Results of reorder_columns.
onset |
duration |
response_time |
trial_type |
|---|---|---|---|
0.0776 |
0.5083 |
0.565 |
go |
5.5774 |
0.5083 |
0.49 |
unsuccesful_stop |
9.5856 |
0.5084 |
0.45 |
go |
13.5939 |
0.5083 |
n/a |
succesful_stop |
17.1021 |
0.5083 |
0.633 |
unsuccesful_stop |
21.6103 |
0.5083 |
0.443 |
go |
Split event¶
The split_event, which is one of the more complicated remodeling operations, is often used to convert event files from using trial-level encoding to event-level encoding. In trial-level encoding each row of the event file represents all the events in a single trial (usually some variation of the cue-stimulus-response-feedback-ready sequence). In event-level encoding, each row represents the marker for a single event. In this case a trial consists of a sequence of multiple events.
The split_event command requires an anchor_column, which could be an existing column or a column that must be added to the dataframe. The purpose of the anchor_column is to hold the codes for the new events.
The new_events dictionary has the new events to be created. The keys are the new event codes to be inserted into the anchor_column. The values in new_events are themselves dictionaries. Each of these dictionaries has three keys:
onset_source, a list specifying the items to be added to the onset of the event row being split. These items can be any combination of numerical values and column names.
duration a list of any combinations of numerical values and column names whose values are to be added to compute the duration.
copy_columns a list of column names whose values should be copied into the new events. Unlisted columns are filled with n/a.
Table 29: Parameters for split_event.
Parameter |
Type |
Description |
|---|---|---|
anchor_event |
str |
The name of the column that will be used for split-event codes. |
new_events |
dict |
Dictionary whose keys are the codes to be inserted as new events |
remove_parent_event |
bool |
If true, remove parent event. |
The split_event command in the following example specifies that new rows should be added to encode the response and stop signal. The anchor column is still trial_type. In a full processing example, it might make sense to rename trial_type to be event_type and to delete the response_time and the stop_signal_delay since these items have been unfolded into separate events.
An example split_event command.
{
"command": "split_events",
"description": "add response events to the trials.",
"parameters": {
"anchor_column": "trial_type",
"new_events": {
"response": {
"onset_source": ["response_time"],
"duration": [0],
"copy_columns": ["response_accuracy", "response_hand", "sex", "trial_number"]
},
"stop_signal": {
"onset_source": ["stop_signal_delay"],
"duration": [0.5],
"copy_columns": ["trial_number"]
}
},
"remove_parent_event": false
}
}
The results of executing this split_event command on the sample events file are:
Table 30: Results of the previous split_event command.
onset |
duration |
trial_type |
stop_signal_delay |
response_time |
response_accuracy |
response_hand |
sex |
|---|---|---|---|---|---|---|---|
0.0776 |
0.5083 |
go |
n/a |
0.565 |
correct |
right |
female |
0.6426 |
0 |
response |
n/a |
n/a |
correct |
right |
female |
5.5774 |
0.5083 |
unsuccesful_stop |
0.2 |
0.49 |
correct |
right |
female |
5.7774 |
0.5 |
stop_signal |
n/a |
n/a |
n/a |
n/a |
n/a |
6.0674 |
0 |
response |
n/a |
n/a |
correct |
right |
female |
9.5856 |
0.5084 |
go |
n/a |
0.45 |
correct |
right |
female |
10.0356 |
0 |
response |
n/a |
n/a |
correct |
right |
female |
13.5939 |
0.5083 |
succesful_stop |
0.2 |
n/a |
n/a |
n/a |
female |
13.7939 |
0.5 |
stop_signal |
n/a |
n/a |
n/a |
n/a |
n/a |
17.1021 |
0.5083 |
unsuccesful_stop |
0.25 |
0.633 |
correct |
left |
male |
17.3521 |
0.5 |
stop_signal |
n/a |
n/a |
n/a |
n/a |
n/a |
17.7351 |
0 |
response |
n/a |
n/a |
correct |
left |
male |
21.6103 |
0.5083 |
go |
n/a |
0.443 |
correct |
left |
male |
22.0533 |
0 |
response |
n/a |
n/a |
correct |
left |
male |
Note that the event numbers are added before the splitting and then copied as the new events are created. This strategy results in a trial number column associated with the events, an alternative to the more complicated process of adding a structure column after splitting.