In the TRANSFAC Matrix.dat most of the motifs are given in the form of count matrix, but a few of them are given as some kind of frequency matrix. Those aren't ordinary PWMs because rows don't sum to 1, but these sums are more or less stable across rows.

I would like to have a raw count matrix in all cases, so I trying to retrieve the count matrix from those several "fractional" cases.

At first I thought that the field BA (Basis - statistical basis of the matrix) MANDATORILY starts with a number of sequences actually used in the matrix, so I would be able to simply multiply:

$$ \frac{\text{nucleotide value}}{\text{row sum}} \times \text{number of sequences from BA} $$

Moreover, sometimes in CC field there's information like: core from: 11

Unfortunately, the results are not very close to integer value, so this cannot be explained by rounding errors.

For example, in a certain row we have (btw, the row sum is $10.73$ from rounding errors):

3.99 4.07 0.87 1.8

BA 17 compiled ... sequences; total weight of sequences: 10.74

CC core from: 11, length: 6

$$ \frac{4.07}{10.74} \times 17 = 6.4423 $$

$$ \frac{4.07}{10.74} \times 11 = 4.1685 $$

In my opinion these values $0.44$ or $0.16$ couldn't be explained by rounding errors. There's no description (link to article) enclosed to this motif. That aren't log-odds because all of them are non-negative values.

In this particular case, if I use the formula $count(x) = round( (\frac{x}{10.74}) \times 11 )$, the resulting counts will sum up to $11$, so it's fine. This method with value $11$ replaced by $17$ doesn't result in set of values summing to $17$, but $16$.

The problem is that in another example I have

BA 17 compiled binding sequences; total weight of equences: 12.86

CC core from: 1, length: 4 In this case I should rather multiply by $17$ not $1$ to get the sum of $17$. As you see, I cannot stick to either BA or CC for all such motifs.

How would you retrieve the raw count matrix in a such case and what could be the problem in my approach? (jumping between BA and CC values)

Could anyone explain how those values in matrix have been derived, is it some kind of log-odds?


So databases like TRANSFAC are of somewhat... variable quality. Those matrices maybe have been derived from a program that just doesn't output a count matrix, but instead outputs a frequency matrix directly. What I would do, and what i have seen people do in published papers, is just multiply the matrix by 100 and round down. It's a kludge, but you'd better be quality checking the matrices somehow anyway, otherwise you're going to get 90% crap.

  • $\begingroup$ Franky speaking, I rejected the idea of matrix multiplication by 100 from the very beginning, because I would get tremendous number of sequences, which would destroy the regularization process. I'm using a full regularization method (with numerically computing E, not approximation 1.5) by Sven Rahman, Tobias Muller & Martin Vingron. The tremendous jump from 17 sequences to 1074 sequences would substantially affect the regularization. $\endgroup$ – Adam Przedniczek May 1 '16 at 18:17
  • 1
    $\begingroup$ Ah, sorry, my mistake. I didn't look at your numbers carefully enough. it looks like either those are already PWMs as opposed to count matrices (i.e. the values are log(count_freq/background_freq) ) or somebody has already done something strange to them. If it's the former you should have noticed some negative values. If it's latter you can try and do something like making the BA 100 and rounding them. Also it's not clear what algorithm you're talking about regularizing in that comment. Link? $\endgroup$ – Dermot Harnett May 1 '16 at 20:19
  • 1
    $\begingroup$ Yes, unfortunately you clearly don't have the raw count matrix. The only other process which could have generated matrices like that is the process of adding pseudo counts. People often add things like (sqrt(BA)) before conversion to a PWM to avoid taking the log of zero. You could see if you can work out the original BA from that. In general though I'd avoid trying to recover bad data like that. Count matrices should be integers. $\endgroup$ – Dermot Harnett May 2 '16 at 6:39
  • 1
    $\begingroup$ That seems reasonable enough. The paper you cite is a little out of date, and I'd recommend reading some more recent ones as well. And be aware that if you use their threshold selection method you're going to get rather different results than the p-value based thresholds used in most papers. $\endgroup$ – Dermot Harnett May 2 '16 at 16:43
  • 1
    $\begingroup$ The basic method you're using is as good as any other simple model, and you won't do much better unless you want to invest a LOT more effort and do deep learning. The problem is that you can't pick any single 'most powerful' method because the threshold you pick will depend on the biological question you want answered. Also, be aware of what an approximation the PWM models are - you either have ChIP training data,which is v. messy, or PBM data, which doesn't reflect cellular conditions. $\endgroup$ – Dermot Harnett May 3 '16 at 9:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.