Data preparation

Formatting omics data

The move-dl pipeline requires continuous omics input to be formatted as a TSV file with one column per feature and one row per feature.

If we load the microbiome OTU table from the maize rhizosphere dataset, it will look something like this:

We have columns corresponding to samples and rows corresponding to features (OTUs), so we need to transpose this table for MOVE.

Now, we can save our table as a TSV and we are ready to go. No need to do any further processing.

Formatting other continuous data

Other non-omics continuous data is formatted in a similar way.

For this tutorial, we are going to extract some continuous data from the maize metadata table. Let us load the table and take a peek:

The original metadata table contains both categorical (e.g., Maize_Line) and continuous data (e.g., Precipitation3Days). We need to separate these into different files.

In this example, we select three columns: age, Precipitation3Days, and Temperature.

Once again, we can save this table as a TSV, and we are ready to continue.

Formatting categorical data

Categorical data like binary variables (e.g., with/without treatment) or discrete categories needs to be formatted in individual files.

The metadata table contains several discrete variables that can be useful for classification, such as maize line, cultivar, and type of soil. For each one of these, we need to create a separate TSV file that will look something like:

Creating a data config file

We are missing two components to make our data compatible with move-dl. First, we need to create an additional text file with all the sample IDs (one ID per line, see example below). This file tells MOVE which samples to use, so the IDs in this file must be present in all the other input files.

Finally, we need to create a data config YAML file. The purpose of this file is to tell MOVE which files to load, where to find them, and where to save any output files.

The data config file for this tutorial would look like this:

Here we break down the fields of this file:

• defaults indicates this file is a config file. It should be left intact.
• raw_data_path points to the raw data location (i.e., the files we created in this tutorial).
• interim_data_path points to the directory where intermediary files will be deposited.
• results_path points to the folder where results will be saved.
• sample_names is the file name of the file containing all valid sample IDs. This file must have a txt extension.
• categorical_inputs is a list of file names containing categorical data. Each element of the list should have a name name and may optionally have a weight. All referenced files should have a tsv extension.
• continuous_inputs lists the continuous data files. Same format as categorical_inputs.

The data config file can have any name, but it must be saved in config/data directory. The final workspace structure should look like this::

tutorial/
│
├── maize/
│   └── data/
│       ├── maize_field.tsv       <- Type of soil data
│       ├── maize_ids.txt         <- Sample IDs
│       ├── maize_line.tsv        <- Maize line data
│       ├── maize_metadata.tsv    <- Age, temperature, precipitation data
│       ├── maize_microbiome.tsv  <- OTU table
│       └── maize_variety.tsv     <- Maize variety data
│
└── config/
    └── data/
        └── maize.yaml            <- Data configuration file

With your data formatted and ready, we can continue to run MOVE and exploring the associations between the different variables in your datasets. Have a look at our :doc:`introductory tutorial</tutorial/introduction>` for more information on this.

References