Event file restructuring

This tutorial works through the process of restructuring event files using the HED event remodeling tools. The tools, which are written in Python, are designed to be run on an entire BIDS dataset. The tools can be called in Jupyter notebook or run via command-line scripts.

What is event file restructuring?

The event files in an experiment provide a crucial link between what happens in the experiment and the experimental data by providing identified time markers linked to the timeline of the experiment.

Event files are often initially created using information in the logs files generated by the experiment’s presentation software or other control software. These event files are then used to identify portions of the data corresponding to particular points or blocks of data to be analyzed or compared.

Event file restructuring refers to creating, modifying, and reorganizing the event markers in tabular files in order to disambiguate or clarify the information for distribution and analysis. Restructuring can occur at several stages during the acquisition and processing of experimental data as shown in this schematic diagram:
schematic diagram.

In addition to restructuring during initial creation of the tabular event files, restructuring may be required when the event files do not conform to the requirements of a particular analysis. Thus, restructuring is an iterative process, which is supported by the HED remodeling tools for datasets with tabular event files.

Table 1 gives a summary of the tools available in the HED remodeling toolbox.

Table 1: Summary of the HED remodeling tools for tabular files.

Category

Command

Example use case

clean-up

remove_columns

Remove temporary columns created during restructuring.

remove_rows

Remove rows with n/a values in a specified column.

rename_columns

Make columns names consistent across a dataset.

reorder_columns

Make column order consistent across a dataset.

factor

factor_column

Extract factor vectors from a column of condition variables.

factor_hed_tags

Extract factor vectors from search queries of HED annotations.

factor_hed_types

Extract design matrices and/or condition variables.

restructure

remap_columns

Create m columns from values in n columns (for recoding).

split_event

Split trial-encoded rows into multiple events.

merge_consecutive

Replace multiple consecutive events of the same type
with one event of longer duration.

add_structure_column

Add a column with condition names.

add_structure_rows

Add a row representing the start of a block or trial.

add_structure_numbers

Add a column with trial or block numbers.

The clean-up commands are used at various phases of restructuring to assure consistency across event files in the dataset.

The factor commands produce column vectors of the same length as the events file that encode condition variables, design matrices, or satisfaction of other search criteria. See the HED conditions and design matrices for more information on factoring and analysis.

The restructure commands modify the way that events are encoded.

Installing remodeling tools

Currently, the remodeling tools are available in the GitHub hed-curation repository along with other tools for data cleaning and curation. These tools rely on the hedtools library which is available on PyPI and can be installed via PIP.

Once the HED remodeling tools are available in final form, the tools will be moved to the GitHub hed-python repository and be available for installation via PyPI. At that time, versions for single event files will become available as a web-service and through a web-interface on the HED online tools. A docker version is also under development.

In the meantime, if you want to run the latest version of the tools, you will need to install the development branch of both hed-python and hed-curation using:

pip install git+https://github.com/hed-standard/hed-python/@develop
pip install git+https://github.com/hed-standard/hed-curation/@develop

Running remodeling scripts

Remodeling consists of applying a list of commands to an events file to restructure or modify it in some way.

The following diagram shows a schematic of the remodeling process.

Event remodeling process

Initially, the user creates a backup of the event files. Restructuring applies a sequence of remodeling commands given in a JSON transformation file to produce a final result. The transformation file provides a record of the operations performed on the file. If the user detects a mistake in the transformation, he/she can correct the transformation file and restore the backup to rerun.

The remodeling toolbox provides several scripts to apply the transformations to the files in a BIDS-formatted dataset. The basic scripts are summarized in Table 2.

Table 2: Summary of the remodeling scripts.

Script name

Arguments

Purpose

run_backup

bids_dir
-t task_name
-b backup-type
-e exclude_dirs

Backup the event files.

run_remodel

bids_dir
-t task_name
-m model-path
-e exclude_dirs

Remodel the event files.

run_restore

bids_dir
-t task_name
-b backup-type
-e exclude_dirs

Restore the event files.

run_remove

bids_dir
-t task_name
-b backup-type
-e exclude_dirs

Remove the backup event files.

All the scripts have a required parameter which is the full path of the BIDS dataset root. The -t task_name option specifies which task in the dataset the remodeling should apply to. Often, when a dataset includes multiple tasks, the event files are structured differently for each task and thus require different transformation files.

The other -e exclude_dirs gives a list of directories to ignore in searching for event file. In BIDS, typical directories to exclude are derivatives, code, and stimulus_files.

Backing up the events

Before any remodeling is performed, you should always back up the event files. Usually this is just done once, before any remodeling is done. There are two strategies for doing the backup: in-place and full-tree.

The in-place strategy makes a copy of each event file in the same directory as the event file using the suffix _eventsorig.tsv to distinguish these backup files from the active event files (which end in _events.tsv). The in-place strategy is simple, but the extra files prevent the dataset from validating with the BIDS validator.

The full-tree strategy creates a complete tree backup in the code/event_backup directory of the BIDS dataset. This strategy is more expensive, but does not prevent validation or interfere with other processing.

Example command to backup the events.

python run_backup.py t:\ds002790-data -b full-tree -e derivatives code simulus_files

Remodeling the events

The event remodeling process is given by the following example:

Example command to remodel the events.

python run_remodel.py t:\ds002790-data -m derivatives/models/simple_rmdl.json -e derivatives code simulus_files

Example remodeling file with commands to remove columns and reorder the rest.

[
   {
       "command": "remove_columns",
       "description": "Get rid of the sample and the value columns.",
       "parameters": {
           "remove_names": ["sample", "value"],
           "ignore_missing": true
       }
   },
   {
       "command": "reorder_columns",
       "description": "Want event_type and task_role columns after onset and duration.",
       "parameters": {
           "column_order": ["onset", "duration", "event_type", "task_role"],
           "ignore_missing": false,
            "keep_others": true
       }
   }
]

Remodeling operations

The examples in this tutorial use the following excerpt of the stop-go task from sub-0013 of the AOMIC-PIOP2 dataset available on OpenNeuro as ds002790. The full events file is sub-0013_task-stopsignal_acq-seq_events.tsv.

Table 3: Excerpt from an event file from the stop-go task of AOMIC-PIOP2 (ds002790).

onset

duration

trial_type

stop_signal_delay

response_time

response_accuracy

response_hand

sex

0.0776

0.5083

go

n/a

0.565

correct

right

5.5774

0.5083

unsuccesful_stop

0.2

0.49

correct

right

female

9.5856

0.5084

go

n/a

0.45

correct

right

female

13.5939

0.5083

succesful_stop

0.2

n/a

n/a

n/a

female

17.1021

0.5083

unsuccesful_stop

0.25

0.633

correct

left

male

21.6103

0.5083

go

n/a

0.443

correct

left

male

The factor_hed_types and factor_hed_tags also require HED annotations of the events and require a JSON sidecar that includes HED annotations. The tutorial uses the following JSON excerpt to illustrate these operations. The full JSON file can be found at: task-stopsiqnal_acq-seq_events.json. These HED commands also require the HED schema. The tutorials use the latest version that is downloaded from the web.

Excerpt of JSON sidecar with HED annotations for the stop-go task of AOMIC-PIOP2.

{
    "trial_type": {
        "HED": {
            "succesful_stop": "Sensory-presentation, Visual-presentation, Correct-action, Image, Label/succesful_stop",
            "unsuccesful_stop": "Sensory-presentation, Visual-presentation, Incorrect-action, Image, Label/unsuccesful_stop",
            "go": "Sensory-presentation, Visual-presentation, Image, Label/go"
        }
    },
    "stop_signal_delay": {
        "HED": "(Auditory-presentation, Delay/# s)"
        },
    "sex": {
        "HED": {
            "male": "Def/Male-image-cond",
            "female": "Def/Female-image-cond"
        }
    },
    "hed_defs": {
        "HED": {
            "def_male": "(Definition/Male-image-cond, (Condition-variable/Image-sex, (Male, (Image, Face))))",
            "def_female": "(Definition/Female-image-cond, (Condition-variable/Image-sex, (Female, (Image, Face))))"
        }
    }
}

Add structure column

NOT WRITTEN - PLACEHOLDER

Add a column of numbers corresponding to a structure elements such as trials or blocks.

Table 4: Parameters for the add_structure_column command.

Parameter

Type

Description

column_name

str

The name of the column to be created or modified.

source_columns

list

Names of the columns to be used for remapping.

mapping

dict

The keys are the values to be placed in the derived columns and the values are each an array

The add_structure_column command in the following example specifies …

An example add_structure_column command.

{ 
    "column_name": "add_structure_column",
    "source_columns": ["response_accuracy", "response_hand"],
    "mapping": {
        "left": [["correct", "left"], ["incorrect", "right"]],
        "right": [["correct", "right"], ["incorrect", "left"]]
    }
}

The results of executing this add_structure_column command on the sample events file are:

Table 5: Results of the previous add_structure_column command.

onset

duration

trial_type

stop_signal_delay

response_time

response_accuracy

response_hand

sex

0.0776

0.5083

go

n/a

0.565

correct

right

female

5.5774

0.5083

unsuccesful_stop

0.2

0.49

correct

right

female

9.5856

0.5084

go

n/a

0.45

correct

right

female

13.5939

0.5083

succesful_stop

0.2

n/a

n/a

n/a

female

17.1021

0.5083

unsuccesful_stop

0.25

0.633

correct

left

male

21.6103

0.5083

go

n/a

0.443

correct

left

male

Add structure events

NOT WRITTEN - PLACEHOLDER

Add events representing the start of a structural element such as a trial or a block.

Table 6: Parameters for the add_structure_events command.

Parameter

Type

Description

column_name

str

The name of the column to be created or modified.

source_columns

list

Names of the columns to be used for remapping.

mapping

dict

The keys are the values to be placed in the derived columns and the values are each an array

The add_structure_events command in the following example specifies …

An example add_structure_events command.

{ 
    "column_name": "add_structure_events",
    "source_columns": ["response_accuracy", "response_hand"],
    "mapping": {
        "left": [["correct", "left"], ["incorrect", "right"]],
        "right": [["correct", "right"], ["incorrect", "left"]]
    }
}

The results of executing this add_structure_events command on the sample events file are:

Table 7: Results of the previous add_structure_events command.

onset

duration

trial_type

stop_signal_delay

response_time

response_accuracy

response_hand

sex

0.0776

0.5083

go

n/a

0.565

correct

right

female

5.5774

0.5083

unsuccesful_stop

0.2

0.49

correct

right

female

9.5856

0.5084

go

n/a

0.45

correct

right

female

13.5939

0.5083

succesful_stop

0.2

n/a

n/a

n/a

female

17.1021

0.5083

unsuccesful_stop

0.25

0.633

correct

left

male

21.6103

0.5083

go

n/a

0.443

correct

left

male

Add structure numbers

NOT WRITTEN - PLACEHOLDER

Add a column with numbers corresponding to a structural element.

TODO clarify the difference between add_structure_numbers and add_structure_column.

Table 8: Parameters for the add_structure_numbers command.

Parameter

Type

Description

column_name

str

The name of the column to be created or modified.

source_columns

list of str

A list of columns to be used for remapping.

mapping

dict

The keys are the values to be placed in the derived columns and the values are each an array

The add_structure_numbers command in the following example specifies …

An example add_structure_numbers command.

{ 
    "command": "add_structure_numbers"
    "description": "xxx"
    "parameters": {
        "column_name": "match_side",
        "source_columns": ["response_accuracy", "response_hand"],
        "mapping": {
            "left": [["correct", "left"], ["incorrect", "right"]],
            "right": [["correct", "right"], ["incorrect", "left"]]
        }
    }
}

The results of executing this add_structure_numbers command on the sample events file are:

Table 9: Results of executing the previous add_structure_numbers command.

onset

duration

trial_type

match_side

stop_signal_delay

response_time

response_accuracy

response_hand

sex

0.0776

0.5083

go

right

n/a

0.565

correct

right

female

5.5774

0.5083

unsuccesful_stop

right

0.2

0.49

correct

right

female

9.5856

0.5084

go

n/a

0.45

correct

right

female

13.5939

0.5083

succesful_stop

0.2

n/a

n/a

right

female

17.1021

0.5083

unsuccesful_stop

0.25

0.633

correct

left

male

21.6103

0.5083

go

n/a

0.443

correct

left

male

Factor column

For each of the specified values in the indicated column create a column containing 1’s and 0’s indicating presence or absence of the value. If no values are specified, all unique values in that column are factored.

Table 10: Parameters for the factor_column command.

Parameter

Type

Description

column_name

str

The name of the column to be factored.

factor_values

list

Column values to be included as factors.

factor_names

list

Column names for created factors. Must be the same length as factor_values.

ignore_missing

bool

If true, include columns for factor values not in events.

overwrite_existing

bool

If true, existing factor columns are overwritten.

The factor_column command in the following example specifies that factor columns should be created for succesful_stop and unsuccesful_stop of the trial_type column. The resulting columns are called stopped and stop_failed, respectively.

If the factor_values is an empty list, factors are created for all unique values in the column_name column. The factor_names parameters must be the same length as factor_values. If factor_names is empty, the newly created columns are of the form column_name.factor_value.

A sample factor_column command.

{ 
    "command": "factor_column"
    "description": "Create factors for the succesful_stop and unsuccesful_stop values."
    "parameters": {
        "column_name": "trial_type",
        "factor_values": ["succesful_stop", "unsuccesful_stop"],
        "factor_names": ["stopped", "stop_failed"],
        "ignore_missing": false,
        "overwrite_existing": true
    }
}

The results of executing this factor_column command on the sample events file are:

Table 11: Results of factoring column XXX.

onset

duration

trial_type

stop_signal_delay

response_time

response_accuracy

response_hand

sex

stopped

stop_failed

0.0776

0.5083

go

n/a

0.565

correct

right

female

0

0

5.5774

0.5083

unsuccesful_stop

0.2

0.49

correct

right

female

0

1

9.5856

0.5084

go

n/a

0.45

correct

right

female

0

0

13.5939

0.5083

succesful_stop

0.2

n/a

n/a

n/a

female

1

0

17.1021

0.5083

unsuccesful_stop

0.25

0.633

correct

left

male

0

1

21.6103

0.5083

go

n/a

0.443

correct

left

male

0

0

Factor HED tags

Produce a one-hot factor based on a HED tag search. The search is based on a list of query filters, which are applied in succession to the assembled HED strings for the event file. Only events that satisfy each query filter as applied in succession will have 1 for the factors. If an event fails one of the queries it does not get a factor

Table 12: Parameters for factor_hed_tags command.

Parameter

Type

Description

factor_name

str

Name of the column to create for the factor.

remove_types

list

Structural HED tags to be removed (usually Condition-variable and Task).

filter_queries

list

Queries to be applied in succession to filter.

overwrite_existing

bool

Overwrite the contents of factor_name column.

The factor_hed-tags command in the following example specifies …

Example factor_hed_tags command.

{ 
    "command": "factor_hed_tags"
    "description": "xxx"
    "parameters": {
        "column_name": "match_side",
        "source_columns": ["response_accuracy", "response_hand"],
        "mapping": {
            "left": [["correct", "left"], ["incorrect", "right"]],
            "right": [["correct", "right"], ["incorrect", "left"]]
        }
    }
}

The results of executing this factor_hed-tags command on the sample events file using the sample sidecar file for HED annotations is:

Table 13: Results of factor_hed_tags.

onset

duration

trial_type

stop_signal_delay

response_time

response_accuracy

response_hand

sex

0.0776

0.5083

go

n/a

0.565

correct

right

female

5.5774

0.5083

unsuccesful_stop

right

0.2

0.49

correct

right

9.5856

0.5084

go

n/a

0.45

correct

right

female

13.5939

0.5083

succesful_stop

0.2

n/a

n/a

n/a

female

17.1021

0.5083

unsuccesful_stop

0.25

0.633

correct

left

male

21.6103

0.5083

go

n/a

0.443

correct

left

male

Factor HED type

The factor HED type operation produces factor columns in an event file based on values of the specified HED type tag. The most common type is the HED Condition-variable tag, which corresponds to factor vectors based in the experimental design. Other commonly use type tags include Task, Control-variable, and Time-block.

We assume that the dataset has been annotated using HED tags to properly document the experiment conditions and focus on how such an annotated dataset can be used with event remodeling to produce factor columns in the event file corresponding to these condition variables.

For additional information on how to encode experimental designs using HED please see HED conditions and design matrices.

Table 14: Parameters for factor_hed_type command.

Parameter

Type

Description

type_tag

str

HED tag used to find the factors (most commonly Condition-variable).

type_values

list

Values to factor for the type_tag.
If empty all values of that type_tag are used.

overwrite_existing

bool

If true, existing factor columns are overwritten.

factor_encoding

str

Indicates type of encoding. Valid encodings are ‘categorical’ and ‘one-hot’.

To simplifyThe factor_hed_type command in the following example specifies …

Example factor_hed_type command.

{ 
    "command": "factor_hed_type"
    "description": "Factor based on the sex of the images being presented."
    "parameters": {
        "type_tag": "Condition-variable",
        "type_values": [],
        "overwrite_existing": true,
        "factor_encoding": "one-hot"
    }
}

In order to use the JSON file. The full file is at:

The results of executing this factor_hed-tags command on the sample events file using the sample sidecar file for HED annotations are:

Table 15: Results of factor_hed_type.

onset

duration

trial_type

stop_signal_delay

response_time

response_accuracy

response_hand

sex

Image-sex.Female-image-cond

Image-sex.Male-image-cond

0.0776

0.5083

go

n/a

0.565

correct

right

female

1

0

5.5774

0.5083

unsuccesful_stop

0.2

0.49

correct

right

female

1

0

9.5856

0.5084

go

n/a

0.45

correct

right

female

1

0

13.5939

0.5083

succesful_stop

0.2

n/a

n/a

n/a

female

1

0

17.1021

0.5083

unsuccesful_stop

0.25

0.633

correct

left

male

0

1

21.6103

0.5083

go

n/a

0.443

correct

left

male

0

1

Merge consecutive

Sometimes in experimental logs, a single long event is represented by multiple repeat events. Merges these same events occurring consecutively into one event with duration of the new event updated to encompass the extent of the merged events..

Table 16: Parameters for the merge_consecutive command.

Parameter

Type

Description

column_name

str

The name of the column to be merged.

event_code

str, int, float

The value in column_name .

match_columns

list

Columns of that must match to collapse events.

set_durations

bool

If True, set durations based on merged events.

ignore_missing

bool

If True, missing column_name or match_columns does not raise an error.

The merge_consecutive command in the following example specifies …

A sample merge_consecutive command.

{ 
    "command": "merge_consecutive"
    "description": "Merge consecutive *succesful_stop* events that match the *match_columns."
    "parameters": {
        "column_name": "trial_type",
        "event_code": "succesful_stop",
        "match_columns": ["stop_signal_delay", "response_hand", "sex"],
        "set_durations": true,
        "ignore_missing": true
    }
}

The follo

Table 17: Input file for a merge_consecutive command.

onset

duration

trial_type

stop_signal_delay

response_hand

sex

0.0776

0.5083

go

n/a

right

female

5.5774

0.5083

unsuccesful_stop

0.2

right

female

9.5856

0.5084

go

n/a

right

female

13.5939

0.5083

succesful_stop

0.2

n/a

female

14.2

0.5083

succesful_stop

0.2

n/a

female

15.3

0.5083

succesful_stop

0.2

n/a

female

17.3

0.5083

succesful_stop

0.25

n/a

female

19.0

0.5083

succesful_stop

0.25

n/a

female

21.1021

0.5083

unsuccesful_stop

0.25

left

male

22.6103

0.5083

go

n/a

left

male

The results of executing the previous merge_events command on the sample events file are:

Table 18: The results of the merge_events command.

| onset | duration | trial_type | stop_signal_delay | response_hand | sex | | —– | ——– | ———- | —————— | ——– —- | — | | 0.0776 | 0.5083 | go | n/a | right | female | | 5.5774 | 0.5083 | unsuccesful_stop | 0.2 | right | female | | 9.5856 | 0.5084 | go | n/a | right | female | | 13.5939 | 2.2144 | succesful_stop | 0.2 | n/a | female | | 17.3 | 2.2083 | succesful_stop | 0.25 | n/a | female | | 21.1021 | 0.5083 | unsuccesful_stop | 0.25 | left | male | | 22.6103 | 0.5083 | go | n/a | left | male]

Remap columns

This command maps the values that appear in a specified m columns of an event file into values in n columns using a defined mapping. This command is often used for recoding the event file. The mapping should have targets for all combinations of values that appear in the m columns.

Table 19: Parameters for the remap_columns command.

Parameter

Type

Description

source_columns

list

Columns with combinations of values to be mapped.

destination_columns

list

Columns to be mapped into.

map_list

list

A list. Each element consists n + m elements
corresponding to the lengths of the source and destination lists respectively.

ignore_missing

bool

If false, a combination of

The map_list parameter specifies how each unique combination of values from the source columns will be mapped into the destination columns. If there are m source columns and n destination columns, then each entry in *map_list must be a list with n + m elements.

The remap_columns command in the following example creates a new column called response_type based on the unique values in the combination of columns response_accuracy and response_hand.

An example remap_columns command.

{ 
    "command": "remap_columns",
    "description": "Map response_accuracy and response hand into a single column.",
    "parameters": {
        "source_columns": ["response_accuracy", "response_hand"],
        "destination_columns": ["response_type"],
        "map_list": [["correct", "left", "correct_left"],
                     ["correct", "right", "correct_right"],
                     ["incorrect", "left", "incorrect_left"],
                     ["incorrect", "right", "incorrect_left"],
                     ["n/a", "n/a", "n/a"]],
        "ignore_missing": true
    }
}

The results of executing the previous remap_column command on the sample events file are:

Table 20: Mapping columns response_accuracy and response_hand into a response_type column.

onset

duration

trial_type

stop_signal_delay

response_time

response_accuracy

response_hand

sex

response_type

0.0776

0.5083

go

n/a

0.565

correct

right

female

correct_right

5.5774

0.5083

unsuccesful_stop

0.2

0.49

correct

right

female

correct_right

9.5856

0.5084

go

n/a

0.45

correct

right

female

correct_right

13.5939

0.5083

succesful_stop

0.2

n/a

n/a

n/a

female

n/a

17.1021

0.5083

unsuccesful_stop

0.25

0.633

correct

left

male

correct_left

21.6103

0.5083

go

n/a

0.443

correct

left

male

correct_left

Typically, the remap_columns command is used often used to map single codes in the experimental log into multiple columns in the final events file.

Remove columns

Remove the specified columns if present. If one of the specified columns is not in the file and the ignore_missing parameter is false, a KeyError is raised for missing column.

Table 21: Parameters for the remove_columns operation.

Parameter

Type

Description

remove_names

list of str

A list of columns to remove.

ignore_missing

boolean

If true, missing columns are ignored, otherwise raise an error.

The remove_column command in the following example removes the stop_signal_delay and response_accuracy columns. The face column is not in the dataframe, but it is ignored, since ignore_missing is True.

An example remove_columns command.

{   
    "command": "remove_columns",
    "description": "Remove columns before the next step.",
    "parameters": {
        "remove_names": ["stop_signal_delay", "response_accuracy", "face"],
        "ignore_missing": true
    }
}

The results of executing this command on the sample events file are shown below. Although face is not the name of a column in the dataframe, it is ignored because ignore_missing is true. If ignore_missing had been false, a KeyError would have been generated.

Table 22: Results of executing the remove_column.

onset

duration

trial_type

response_time

response_hand

sex

0.0776

0.5083

go

0.565

right

female

5.5774

0.5083

unsuccesful_stop

0.49

right

female

9.5856

0.5084

go

0.45

right

female

13.5939

0.5083

succesful_stop

n/a

n/a

female

17.1021

0.5083

unsuccesful_stop

0.633

left

male

21.6103

0.5083

go

0.443

left

male

Remove rows

Remove rows in which the named column has one of the specified values.

Table 23: Parameters for remove_rows.

Parameter

Type

Description

column_name

str

The name of the column to be tested.

remove_values

list

A list of values to be tested for removal.

The following example remove_rows command removes the rows whose trial_type column has either succesful_stop or unsuccesful_stop.

Sample remove_rows command.

{   
    "command": "remove_rows",
    "description": "Remove rows where trial_type is either succesful_stop or unsuccesful_stop.",
    "parameters": {
        "column_name": "trial_type",
        "remove_values": ["succesful_stop", "unsuccesful_stop"]
    }
}

The results of executing the previous remove_rows command on the sample events file are:

Table 24: The results of executing the previous remove_rows command.

onset

duration

trial_type

stop_signal_delay

response_time

response_accuracy

response_hand

sex

0.0776

0.5083

go

n/a

0.565

correct

right

female

9.5856

0.5084

go

n/a

0.45

correct

right

female

21.6103

0.5083

go

n/a

0.443

correct

left

male

After removing rows with trial_type equal to succesful_stop or unsuccesful_stop only the three go trials remain.

Rename columns

Rename columns by providing a dictionary of old names to new names.

Table 25: Parameters for rename_columns.

Parameter

Type

Description

column_mapping

dict

The keys are the old column names and the values are the new names.

ignore_missing

bool

If false, a key error is raised if a dictionary key is not a column name.

The rename_columns command in the following example specifies that response_hand column be renamed hand_used and that the sex column be renamed image_sex. The face entry in the mapping will be ignored because ignore_missing is true. If ignore_missing is false, a KeyError exception is raised if a column specified in the mapping does not correspond to a column name in the dataframe.

Example rename_columns command.

{   
    "command": "rename_columns",
    "description": "Remove columns before splitting events.",
    "parameters": {
        "column_mapping": {
            "random_column": "new_random_column",
            "stop_signal_delay": "stop_delay",
            "response_hand": "hand_used",
            "sex": "image_sex"
        },
        "ignore_missing": true
    }
}

The results of executing the previous rename_columns command on the sample events file are:

Table 26: After the rename_columns command is executed, the sample events file is:

onset

duration

trial_type

stop_delay

response_time

response_accuracy

hand_used

image_sex

0.0776

0.5083

go

n/a

0.565

correct

right

female

5.5774

0.5083

unsuccesful_stop

0.2

0.49

correct

right

female

9.5856

0.5084

go

n/a

0.45

correct

right

female

13.5939

0.5083

succesful_stop

0.2

n/a

n/a

n/a

female

17.1021

0.5083

unsuccesful_stop

0.25

0.633

correct

left

male

21.6103

0.5083

go

n/a

0.443

correct

left

male

Reorder columns

Reorder the columns in the specified order. If ignore_missing is true, and items in the reorder list do not exist in the file, these columns are ignored. On the other hand, if ignore_missing is false, a column name not appearing in the reorder list causes a ValueError to be raised.

The keep_others parameter controls whether or not columns in the dataframe that do not appear in the reorder list are dropped (keep_others is false) or put at the end of the dataframe in the order they appear (keep_others is true).

Table 27: Parameters for reorder_columns.

Parameter

Type

Description

column_order

list

A list of columns in the order they should appear in the data.

ignore_missing

bool

If true and a column in column_order does not appear in the dataframe
a ValueError is raised, otherwise these columns are ignored.

keep_others

bool

If true, existing columns that aren’t in column_order
are moved to the end in the same relative
order that they originally appeared in the data,
otherwise these columns are dropped.

The reorder_columns command in the following example specifies that the first four columns of the dataset should be: onset, duration, trial_type, response_hand, and response_time. Since ignore_missing is true, these will be the only columns retained.

Example reorder_columns command.

{   
    "command": "reorder_columns",
    "description": "Reorder columns.",
    "parameters": {
        "column_order": ["onset", "duration", "response_time",  "trial_type"],
        "ignore_missing": true,
        "keep_others": false
    }
}

The results of executing the previous reorder_columns command on the sample events file are:

Table 28: Results of reorder_columns.

onset

duration

response_time

trial_type

0.0776

0.5083

0.565

go

5.5774

0.5083

0.49

unsuccesful_stop

9.5856

0.5084

0.45

go

13.5939

0.5083

n/a

succesful_stop

17.1021

0.5083

0.633

unsuccesful_stop

21.6103

0.5083

0.443

go

Split event

The split_event, which is one of the more complicated remodeling operations, is often used to convert event files from using trial-level encoding to event-level encoding. In trial-level encoding each row of the event file represents all the events in a single trial (usually some variation of the cue-stimulus-response-feedback-ready sequence). In event-level encoding, each row represents the marker for a single event. In this case a trial consists of a sequence of multiple events.

The split_event command requires an anchor_column, which could be an existing column or a column that must be added to the dataframe. The purpose of the anchor_column is to hold the codes for the new events.

The new_events dictionary has the new events to be created. The keys are the new event codes to be inserted into the anchor_column. The values in new_events are themselves dictionaries. Each of these dictionaries has three keys:

  • onset_source, a list specifying the items to be added to the onset of the event row being split. These items can be any combination of numerical values and column names.

  • duration a list of any combinations of numerical values and column names whose values are to be added to compute the duration.

  • copy_columns a list of column names whose values should be copied into the new events. Unlisted columns are filled with n/a.

Table 29: Parameters for split_event.

Parameter

Type

Description

anchor_event

str

The name of the column that will be used for split-event codes.

new_events

dict

Dictionary whose keys are the codes to be inserted as new events
and whose values are dictionaries with
keys onset_source, duration, and copy_columns.

remove_parent_event

bool

If true, remove parent event.

The split_event command in the following example specifies that new rows should be added to encode the response and stop signal. The anchor column is still trial_type. In a full processing example, it might make sense to rename trial_type to be event_type and to delete the response_time and the stop_signal_delay since these items have been unfolded into separate events.

An example split_event command.

{
  "command": "split_events",
  "description": "add response events to the trials.",
        "parameters": {
            "anchor_column": "trial_type",
            "new_events": {
                "response": {
                    "onset_source": ["response_time"],
                    "duration": [0],
                    "copy_columns": ["response_accuracy", "response_hand", "sex", "trial_number"]
                },
                "stop_signal": {
                    "onset_source": ["stop_signal_delay"],
                    "duration": [0.5],
                    "copy_columns": ["trial_number"]
                }
            },	
            "remove_parent_event": false
        }
    }

The results of executing this split_event command on the sample events file are:

Table 30: Results of the previous split_event command.

onset

duration

trial_type

stop_signal_delay

response_time

response_accuracy

response_hand

sex

0.0776

0.5083

go

n/a

0.565

correct

right

female

0.6426

0

response

n/a

n/a

correct

right

female

5.5774

0.5083

unsuccesful_stop

0.2

0.49

correct

right

female

5.7774

0.5

stop_signal

n/a

n/a

n/a

n/a

n/a

6.0674

0

response

n/a

n/a

correct

right

female

9.5856

0.5084

go

n/a

0.45

correct

right

female

10.0356

0

response

n/a

n/a

correct

right

female

13.5939

0.5083

succesful_stop

0.2

n/a

n/a

n/a

female

13.7939

0.5

stop_signal

n/a

n/a

n/a

n/a

n/a

17.1021

0.5083

unsuccesful_stop

0.25

0.633

correct

left

male

17.3521

0.5

stop_signal

n/a

n/a

n/a

n/a

n/a

17.7351

0

response

n/a

n/a

correct

left

male

21.6103

0.5083

go

n/a

0.443

correct

left

male

22.0533

0

response

n/a

n/a

correct

left

male

Note that the event numbers are added before the splitting and then copied as the new events are created. This strategy results in a trial number column associated with the events, an alternative to the more complicated process of adding a structure column after splitting.