Why Dataset Discovery Matters

Public Data Reuse Begins Before Download

Public omics repositories have made it possible to reuse biological datasets generated by research groups around the world. These datasets can support secondary analysis, comparative studies, benchmarking, training materials, method development, and reference dataset construction.

However, public data reuse does not begin at the download step.

It begins with discovery.

Before sequencing files are retrieved, before metadata tables are assembled, and before quality control begins, the analyst must first decide which public studies are relevant, eligible, and usable for the intended purpose.

This makes dataset discovery a critical part of any reproducible public-data workflow.

A weak discovery process can lead to weak downstream analysis, even if the data download and processing steps are technically correct.

Finding Is Not the Same as Selecting

It is easy to find public datasets.

It is harder to select the right datasets.

A repository search may return many studies related to a topic, but not all of them are suitable for reuse. Some studies may focus on the wrong population, sample type, disease state, organism, sequencing method, or experimental design. Others may have incomplete metadata, unclear accessions, unavailable files, or insufficient documentation.

For example, a search for human gut microbiome data may return studies involving healthy adults, infants, patients with gastrointestinal disease, antibiotic-treated individuals, animal models, fecal transplant studies, and dietary interventions.

All of these may involve the gut microbiome, but they do not all answer the same research question.

Systematic dataset discovery helps separate general relevance from true eligibility.

Search Result
      ↓
Candidate Study
      ↓
Screened Study
      ↓
Eligible Study
      ↓
Included Study

Why Informal Dataset Selection Is Risky

Informal dataset selection often depends on convenience.

An analyst may choose the first dataset that appears in a search result, a study that is easy to download, a dataset mentioned in a familiar paper, or a BioProject that appears to contain many samples.

This may be acceptable for quick exploration, but it is not enough for reproducible research, client-facing work, benchmarking, or reference dataset assembly.

Informal selection creates several risks:

• important eligible studies may be missed
• unsuitable studies may be included
• exclusion decisions may not be documented
• the final dataset may be biased toward convenience
• downstream results may be difficult to interpret
• collaborators may not understand why specific studies were selected
• the workflow may be difficult to reproduce later

A structured discovery process reduces these risks by making study selection transparent and traceable.

Dataset Discovery Shapes the Final Dataset

Every decision made during discovery affects the final dataset.

The search terms used determine which studies are found. The repositories searched determine which records are visible. The eligibility criteria determine which studies remain. The screening process determines which datasets are excluded. The prioritization process determines which studies are used first.

By the time data acquisition begins, many important decisions have already been made.

Research Question
      ↓
Search Terms
      ↓
Candidate Studies
      ↓
Eligibility Criteria
      ↓
Screening Decisions
      ↓
Included Studies
      ↓
Final Dataset

This means that dataset discovery is not a minor administrative step. It is part of the scientific and analytical design of the project.

Metadata Matters as Much as Data Availability

A dataset may be public and downloadable, but still not usable.

For public omics studies, metadata are often as important as the sequence files themselves. Without adequate metadata, it may be difficult to interpret samples, define groups, identify controls, compare studies, or reproduce analysis decisions.

Important metadata may include:

• organism
• sample source
• tissue or body site
• disease status
• treatment or exposure
• age group
• sex
• geographic location
• sequencing platform
• library strategy
• study design
• accession identifiers
• file availability

A study with many sequencing files but poor metadata may be less useful than a smaller study with clear, complete, and well-structured metadata.

Systematic dataset discovery therefore evaluates both scientific relevance and practical usability.

Discovery Supports Reproducibility

Reproducibility is not only about code.

It also depends on whether the input data were selected in a transparent and repeatable way.

A reproducible dataset discovery process should make it possible to answer:

• where the studies were searched
• which search terms were used
• when the search was performed
• how many candidate studies were found
• which studies were screened
• which studies were excluded
• why each study was excluded
• which studies were included
• which accessions were selected for acquisition

These details create an audit trail.

They allow another analyst, collaborator, reviewer, or future version of the same project to understand how the dataset was assembled.

Reproducible Discovery Outputs

A systematic discovery workflow should produce structured outputs that record how studies were found, screened, excluded, prioritized, and prepared for acquisition.

#| label: create-discovery-outputs
#| eval: false

bash scripts/bash/01-create-discovery-outputs.sh

Discovery Improves Communication

Public dataset selection often involves communication with collaborators, clients, supervisors, students, or research teams.

A systematic discovery workflow makes this communication easier.

Instead of saying:

We found some datasets online.

The analyst can say:

We searched defined repositories using documented search terms, screened candidate studies using predefined eligibility criteria, excluded unsuitable studies with reasons, and assembled a prioritized list of eligible public omics studies for acquisition.

This changes dataset selection from an informal search activity into a clear, explainable decision process.

Discovery Reduces Downstream Waste

Poor dataset selection can create unnecessary downstream work.

If unsuitable studies are downloaded, the analyst may spend time validating files, troubleshooting metadata, processing samples, and generating outputs from data that should never have been included.

This can waste storage, compute time, and analyst effort.

A strong discovery process reduces this waste by identifying problems earlier.

Better Discovery
      ↓
Better Screening
      ↓
Better Inclusion Decisions
      ↓
Less Downstream Waste
      ↓
More Reliable Analysis

Discovery Is Especially Important in Omics

Omics datasets are complex because they combine biological, technical, and metadata dimensions.

Two studies may appear similar at the title level but differ in important ways, such as:

• sample type
• sequencing platform
• library preparation
• target region
• read layout
• population characteristics
• disease or exposure status
• metadata completeness
• accession structure
• repository availability

For microbiome studies, for example, one dataset may contain 16S rRNA amplicon sequencing from healthy adult stool samples, while another may contain shotgun metagenomic data from patients with disease. Both may be described as human gut microbiome studies, but they are not interchangeable.

Systematic discovery helps capture these differences before data acquisition begins.

From Discovery to Acquisition

The purpose of dataset discovery is not only to identify interesting studies.

The purpose is to prepare a structured set of eligible studies that can move into data acquisition.

In the CDI workflow, this means that systematic dataset discovery produces a documented input package for the CDI Data Acquisition System.

Systematic Dataset Discovery
      ↓
Included Studies
      ↓
Curated Accessions
      ↓
CDI-DAS Input Package
      ↓
Data Acquisition

This separation keeps the workflow clean.

Dataset discovery focuses on study selection.

Data acquisition focuses on retrieving, validating, and organizing files.

Summary

Dataset discovery matters because it determines which public omics studies enter the downstream workflow.

A systematic discovery process helps ensure that public datasets are not selected only because they are easy to find or easy to download. Instead, studies are evaluated based on relevance, eligibility, metadata quality, accession clarity, and suitability for the research question.

By making dataset selection transparent and reproducible, systematic discovery strengthens public data reuse and prepares better inputs for the CDI Data Acquisition System.

Looking Ahead

In the next chapter, we translate the need for systematic dataset discovery into clear research questions and discovery objectives.