Calculate the Pearson correlation and associated p value for multiple variables

Hi all,

I am working with the huge microarray expression data set. I have the expression value of 27000 probes representing 5500 genes across 14 different data points (Variables: D1 to D14). Among these 5500 genes, few genes are represented by multiple probes (i.e., different probes for the same gene). The distribution of probe representation for 5500 genes varies from 1 to 5 (meaning few genes have 1 or 2 or 3 or 4 or 5 probes). Now, I want to compute Pairwise Pearson Correlation Coefficient and associated P-value for all the possible combinations of multiple probes of the same gene across 14 different data points (14 variables) and export the result in a 1-Dimensional format. A small portion of my input data table in CSV format is shown below

ProbeName Gene D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14
A1 A 9.1 6.6 8.2 9.3 9.0 8.8 9.9 7.5 10.8 9.0 8.3 11.6 9.3 10.9
A2 A 3.9 3.7 5.8 2.2 2.9 2.8 2.9 3.8 3.3 1.7 3.2 3.5 5.9 3.7
A3 A 4.6 4.8 6.8 2.8 4.3 3.5 4.2 5.3 4.5 3.3 4.0 4.3 6.9 4.7
A4 A 3.8 3.9 5.8 3.2 4.0 2.8 3.7 4.6 3.6 2.2 3.8 4.3 5.6 3.9
A5 A 6.3 6.6 7.7 5.9 5.9 5.6 6.2 6.4 5.8 4.9 5.4 6.1 7.7 6.9
B1 B 7.5 5.5 7.1 10.2 7.2 8.6 8.3 7.1 6.1 7.0 9.2 6.4 6.4 9.4
B2 B 4.6 4.8 5.6 4.3 4.7 4.3 4.0 5.5 4.0 3.3 3.8 5.0 5.7 4.7
B3 B 5.1 3.9 5.1 6.5 5.0 5.4 4.9 5.3 4.5 4.5 5.9 5.0 4.6 5.6
B4 B 7.6 6.1 7.5 10.9 8.0 9.2 8.5 7.1 6.3 7.4 10.0 6.9 6.9 10.2
C1 C 3.1 6.1 3.4 2.5 3.7 3.3 2.7 5.0 2.3 3.1 2.0 3.8 2.6 3.3
C2 C 3.8 7.1 4.8 4.1 4.9 4.5 3.8 5.9 4.0 4.7 4.4 5.1 2.9 4.8
C3 C 3.8 6.1 5.5 5.4 6.3 3.9 3.4 7.8 5.3 5.7 4.8 4.0 3.5 4.3
D1 D 12.2 11.7 11.4 10.5 11.5 11.4 10.7 12.0 11.3 10.5 9.9 11.7 10.5 10.2
D2 D 12.0 11.5 11.3 10.4 11.4 11.4 10.7 11.9 11.2 10.6 9.9 11.7 10.3 10.2
E1 E 2.4 3.3 7.5 3.4 5.8 3.6 1.2 3.5 0.9 2.2 3.1 4.7 7.5 4.0

The ProbeName column represents the name of the probes from A1 to E1, the Gene column represents the name of the genes from A to E, and Columns D1 to D14 (variables) represent the expression values in different data points. Rows represent the expression value of a probe representing a particular gene in 14 different data points (i.e., how much a particular gene is activated in 14 different data points with the respective probes). A1, A2, A3, A4 & A5 represent multiple probes for the same gene A, and likewise for the other genes B, C, D, and E. In this Table, I want to compute the possible pairwise Pearson correlation of multiple probes for the same gene across 14 data points (D1 to D14). For Example, the possible probe combinations for gene C to compute Pearson correlation across 14 data points are

 1. C1 (D1:3.1, D2:6.1, D3:3.4, D4:2.5, D5:3.7, D6:3.3, D7:2.7, D8:5.0,
    D9:2.3, D10:3.1, D11:2.0, D12:3.8, D13:2.6, D14:3.3) Vs C2 (D1:3.8,
    D2:7.1, D3:4.8, D4:4.1, D5:4.9, D6:4.5, D7:3.8, D8:5.9, D9:4.0,
    D10:4.7, D11:4.4, D12:5.1, D13:2.9, D14:4.8),
 2. C1 (D1:3.1, D2:6.1, D3:3.4, D4:2.5, D5:3.7, D6:3.3, D7:2.7, D8:5.0,
    D9:2.3, D10:3.1, D11:2.0, D12:3.8, D13:2.6, D14:3.3) Vs C3 (D1:3.8,
    D2:6.1, D3:5.5, D4:5.4, D5:6.3, D6:3.9, D7:3.4, D8:7.8, D9:5.3,
    D10:5.7, D11:4.8, D12:4.0, D13:3.5, D14:4.3),
 3. C2 (D1:3.8, D2:7.1, D3:4.8, D4:4.1, D5:4.9, D6:4.5, D7:3.8, D8:5.9,
    D9:4.0, D10:4.7, D11:4.4, D12:5.1, D13:2.9, D14:4.8) Vs C3 (D1:3.8,
    D2:6.1, D3:5.5, D4:5.4, D5:6.3, D6:3.9, D7:3.4, D8:7.8, D9:5.3,
    D10:5.7, D11:4.8, D12:4.0, D13:3.5, D14:4.3)

After generating the correlation matrix of the possible pairwise combinations of multiple probes for the same gene across 14 data points, I want to flatten only the upper or lower triangular correlation matrix and generate the output in CSV format as mentioned below.

ProbeName_1 ProbeName_2 Gene PearonCorrelationValue Pvalue
A1 A2 A -0.129 0.661
A1 A3 A -0.176 0.547
A1 A4 A -0.106 0.718
A1 A5 A -0.084 0.776
A2 A3 A 0.963 0.000
A2 A4 A 0.932 0.000
A2 A5 A 0.914 0.000
A3 A4 A 0.922 0.000
A3 A5 A 0.883 0.000
A4 A5 A 0.882 0.000
B1 B2 B -0.328 0.253
B1 B3 B 0.900 0.000
B1 B4 B 0.987 0.000
B2 B3 B -0.084 0.774
B2 B4 B -0.322 0.261
B3 B4 B 0.882 0.000
C1 C2 C 0.888 0.000
C1 C3 C 0.542 0.045
C2 C3 C 0.658 0.011
D1 D2 D 0.993 0.000

I do not know how to deal with this complex data with R . I humbly request the experts to help me with this problem.

Note: I do not want the correlation value of identical probe combinations i.e., A1 Vs A1 or A2 Vs A2 or A3 Vs A3 or A4 Vs A4 or A5 Vs A5. I also do not want to perform a pairwise combination of a probe of one gene with the probe of another different gene. i.e., A1 Vs B1, B2, B3, B4 or A1 Vs C1, C2, C3 or A1 Vs D1, D2, and or A1 Vs E1.

Read more here: Source link