# Aggregating datasets¶

## Motivation¶

Tomahawk generally output many millions to many hundreds of millions, or even billions, of output linkage disequilibrium (LD) associations generated from many millions of input SNVs. It is technically very challenging to visualize such large datasets. Not only because of hardware limitations such as loading all the data into memory, or directly rendering billions of data points, but also because of more practical considerations such as cramming such a vast number of data points into a finite number of pixels would result in an absolute horrendous and uninformative image.

In order to get a scope of the scale this problem presents, take for example a
small chromosome like `chr20`

with data from the 1000 Genomes Project Phase 3
(1KGP3). This data comprises of 1,733,484 diploid SNVs. Assuming we can plot the
LD data for a pair of SNVs in a single pixel, a monitor would have to have the
dimensions 400 x 400 meters to display this data*! Not only would the monitor
have to be huge, the memory requirement for plotting this image would be around
400 GB! Here we describe methods to overcome these obstacles.

* Assuming a 1920 x 1080 pixel resolution and 20" monitor as reference

## Existing solutions¶

There are several existing solutions for aggregating large datasets, such as
Datashader for Python users. But packages like this
requires us to leave the highly compressed internal binary representation of
Tomahawk in order to transform `two`

records into a form understandable by these
frameworks. We have tried several of the most popular framework for aggregating
datasets and found none that works in reasonable memory and is sufficiently
efficient when applied to our specific use-case.

## Aggregation¶

Aggregation is the process of reducing larger datasets to smaller ones for the purposes of displaying more data than can fit on the screen at once while maintaining the primary features of the original dataset. Tomahawk performs aggregation into regular grids (two-dimensional partitions) by applying summary statistics function on data collected in the given bins. At the moment, Tomahawk supports aggregation by

Function | Action |
---|---|

Summation | Sum total of the desired property |

Summation squared | Sum total of squares of the desired property |

Mean | Mean of the desired property |

Standard deviation | Standard deviation of the desired property |

Minimum | Smallest value observed of the desired property |

Maximum | Largest value oserved of the desired property |

Count | Number of times a non-zero value is observed of the desired property |

Without losing generality, imagine we start out with this 4x4 matrix of observations and we want to plot 4 pixels (2 x 2).

C1 | C2 | C3 | C4 | |
---|---|---|---|---|

R1 |
1 | 2 | 3 | 4 |

R2 |
5 | 6 | 7 | 8 |

R3 |
9 | 10 | 11 | 12 |

R4 |
13 | 14 | 15 | 16 |

Aggregation by summation

C1-2 | C3-4 | |
---|---|---|

R1-2 |
14 | 22 |

R3-4 |
46 | 54 |

Aggregation by mean

C1-2 | C3-4 | |
---|---|---|

R1-2 |
3.5 | 5.5 |

R3-4 |
11.5 | 13.5 |

Aggregation by min

C1-2 | C3-4 | |
---|---|---|

R1-2 |
1 | 3 |

R3-4 |
9 | 11 |

Aggregation by max

C1-2 | C3-4 | |
---|---|---|

R1-2 |
6 | 8 |

R3-4 |
14 | 16 |

Aggregation by count

C1-2 | C3-4 | |
---|---|---|

R1-2 |
4 | 4 |

R3-4 |
4 | 4 |