5
$\begingroup$

I've always been under the impression that our brains process images (relayed to it via the eyes) at a particular frequency or "framerate"; this is supported by lots of sources, such as this one.

I'm wondering how close this compares to the way our brains process sounds:

  • Do we sample "snippets"/clips of sound? If so, is there any evidence as to how long (duration-wise) these "clips" are? Do we "sample" at the millisecond level? Microsecond?; or...
  • Do we actually sample "sound frames" (much like we do with visual images)? If so, how does our brain process a "frame" that contains multiple noises occurring at the same time (say a dog is barking at the same time the television is on)? After all, a frame is all the input for an instantaneous point in time, versus a clip, which is all the input over a period of time.

Any ideas?

$\endgroup$
0
+100
$\begingroup$

It would seem that this 2014 publication adequately addresses your curiosity.

The abstract..

Does our perceptual awareness consist of a continuous stream, or a discrete sequence of perceptual cycles, possibly associated with the rhythmic structure of brain activity? This has been a long-standing question in neuroscience. We review recent psychophysical and electrophysiological studies indicating that part of our visual awareness proceeds in approximately 7–13 Hz cycles rather than continuously.

This confirms your statement regarding discrete visual processing. Continuing..

..

On the other hand, experimental attempts at applying similar tools to demonstrate the discreteness of auditory awareness have been largely unsuccessful. We argue and demonstrate experimentally that visual and auditory perception are not equally affected by temporal subsampling of their respective input streams: video sequences remain intelligible at sampling rates of two to three frames per second, whereas audio inputs lose their fine temporal structure, and thus all significance, below 20–30 samples per second.

This is saying that, (1) experimental methods for quantifying auditory processing might not be so analogous to that of visual processing, and, (2) the minimum threshold sampling rate for auditory processing is much higher than auditory sampling rates. This second statement, however, is still no indication that audition processing is continuous (as will be stated next).

..

This does not mean, however, that our auditory perception must proceed continuously. Instead, we propose that audition could still involve perceptual cycles, but the periodic sampling should happen only after the stage of auditory feature extraction. In addition, although visual perceptual cycles can follow one another at a spontaneous pace largely independent of the visual input, auditory cycles may need to sample the input stream more flexibly, by adapting to the temporal structure of the auditory inputs.


And their conclusions..

Audition does not sample at a hierarchically early level in either an ongoing way (§3b and §4a) or an entrained (flexible) way (§3c and §4a). If it does sample, then it must be at a higher level, that is, a cortical level (§4b). Many reports of attentional control of phase entrainment [74,75,115] suggest that the sampling may also be attentional, as in the visual system. Can this high-level sampling be an ongoing process as in vision, or must it be entrained by the temporal structure of auditory inputs? The lack of ongoing EEG phase influence in audition (§3b) as well as the finding that phase entrainment strongly facilitates intelligibility [65,78,86] compels us to favour the latter alternative, just like other authors have recently argued [73,76,89].

..

To sum up, if perceptual cycles exist in audition, then they must be a relatively high-level or attentional phenomenon (as in vision), and they must proceed by stimulus entrainment (contrary to vision). Based on numerous studies of rhythmic entrainment and speech processing, we believe that the cycles are most likely to be observed in the theta-frequency range (though gamma-frequency sampling cannot be categorically ruled out). But the big ‘if’ lingers. Definite evidence for auditory perceptual cycles is still lacking.


So, it would seem that the verdict is still out as to whether or not auditory processing behaves similarly to visual processing, however, current consensus is that auditory processing is more continuous, and not as discrete, and, if perceptual cycles do occur for auditory processing, they occur at very specific frequency ranges, and not during conventional hearing.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Your summary of that paper omits all of the contradictory evidence they refer to and cherrypicks a bit... It's a big jump from "oscillatory activity might explain one perceptual illusion" to "all visual processing has a frame rate." $\endgroup$ – Bryan Krause Aug 30 '17 at 18:13
  • 1
    $\begingroup$ My summary omits pretty much the entire paper.. I'm not going to refactor the entire article when the OP can just read it. I gave the abstract and conclusion. The rest it too detailed and cumbersome to try and re-express as an answer (and is something that you personally have previously criticized me about). Hence, I provide their initial declaration (abstract) and end conclusion (results). As for "all of the contradictory evidence" that you refer to, can you be specific? $\endgroup$ – user22020 Aug 30 '17 at 18:16
  • 2
    $\begingroup$ The article is making an argument that we should consider visual processing as discrete at the level of perception; that's fine. I feel like your post makes it sound like visual processing has been established to be discrete throughout, which is neither true nor what the authors of that paper are saying. $\endgroup$ – Bryan Krause Aug 30 '17 at 18:26
  • 1
    $\begingroup$ Please forgive my bluntness, but there is too much copy-paste here. "Copy-pasta" should not be the bulk of your answer. It should only be used to support your answer. $\endgroup$ – anongoodnurse Aug 30 '17 at 18:43
  • 1
    $\begingroup$ @anongoodnurse Yeah, I only thought it was you due to the close time frame between the two actions. Sometimes is it good to know though that a DV came from someone who's a heavy user, because there's much more credibility to the DV, which would incline me more to consider an edit. Thanks again. $\endgroup$ – user22020 Aug 30 '17 at 18:52
5
$\begingroup$

In your question, you ask about processing and sampling. I'd like to make a distinction between sampling, which in my view does not have a framerate, and perception which there is some evidence that there is sort of a framerate (though not as concrete as the framerate of a video display, for example). Processing is everything in between, and as such, whether it shows "framerate-like" behavior depends on the stage of processing.

Does vision really have a framerate?

The "framerate" depicted in those sources for vision is sort of a special case, talking about perception of a particular stimulus. There are no real "frames" in your vision, only temporal averaging (because phototransduction is quite slow on the timescale of neural processing in general). The article you link makes a mistake in logic in concluding that humans have a frame rate: the research they cite is about discriminating between separate flashes, which is an issue of temporal averaging rather than framerate. There is a separate issue, which is whether perception correlates with particular oscillatory frequencies; I'll get to this later in my answer.

If you flash a very bright 1 ms light, and record it with a video camera, you will often miss the light entirely (or it may appear only in part of the image, depending on the scanning pattern of the camera). With vision, though, you will always see the light, and for very brief flashes you will perceive changes in duration as changes in intensity.

The other contributor is the "magic" of perception: effectively, what you perceive is not directly the signals that come in through your eyes but rather the model your brain makes for the world based on that information. This process is the origin of all perceptual illusions; the appearance of "backward" images like with a spinning wheel is one example.

Our current model for how processing of sensory stimuli works in the brain is that there is a functional hierarchy with significant abstraction at higher levels of the hierarchy. As you get to higher order levels of processing, you will cease to see the "pixel information" and simple edge/feature detection, and begin to see more abstract representations like detection of particular visual objects or encoding of object velocity.

Okay but the question is about auditory stuff..

It's easiest to start with vision simply because, for better or worse, it is best studied historically. For auditory processing, there are a lot of similarities with vision even though the input stimuli start very different. At low levels of processing, auditory neurons are well described by spectro-temporal receptive fields. At higher levels of processing, you will instead see representations of whole features in sounds, such as phonemes.

Object segmentation is a very difficult problem in both vision and hearing (and a difficult task for computer/AI-based versions of those senses, as well). This is definitely an active field of research, but there is good evidence that some segmentation is present throughout the cortical hierarchy (for example, see here). There are no concrete answers for 'how' exactly segmentation works, just that it does, some evidence for where it might be happening, and some understanding of factors that make it harder to segment different sources. Sound localization is certainly one of the key factors, though segmentation can occur even when there are no localization cues. For example, if you have multiple speakers on the radio, or multiple musical instruments playing, there are some shared spectral characteristics of the speakers or instruments

Discrete signals in cortex

All that said, this does not mean that the brain is free from any discrete elements of processing (like "frames"), although it probably doesn't make sense to think of a frame rate but rather a type of "threading". There is recent strong evidence that activity in the neocortex can be organized into packets of activity that contain sensory or other information. So even though the auditory stream is sampled continuously, that continuous sample is most likely distilled into discrete auditory objects that reflect both the content and source of those stimuli.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Question about visual higher order. Do we have schemata for visuals? That's what I think of when you say "...more abstract representations like detection of particular visual objects or encoding of object velocity." but I'm not sure. $\endgroup$ – anongoodnurse Aug 30 '17 at 17:21
  • 2
    $\begingroup$ I would really love an explanation for the downvote...though I know how unlikely those requests are to actually produce an explanation. $\endgroup$ – Bryan Krause Aug 30 '17 at 17:58
  • $\begingroup$ @anongoodnurse, schemata looks like a psych term, haven't seen it used in (my area of) neuroscience. I'm not quite comfortable saying "yes" without understanding clearly what baggage that term might convey, but I'll say "probably" :). By abstract representations I mean to be a bit non-specific, I'm basically including anything that is not a pixel/edge representation. It could be a code for "dog" versus "cat" or simpler features like "triangular" versus "square" - those are things that all seem discrete in vision, the Necker cube being a great example. $\endgroup$ – Bryan Krause Aug 30 '17 at 18:02
  • $\begingroup$ A schema in learning theory is a concept (of information) we know in a matrix of such concepts (say it's a room among many millions of rooms). If the room has a label (e.g. "organelles"), we expend far less energy on learning new things about organelle-related subjects. If the room "organelle" is missing, we need to devote a lot of working memory in order to remember a new fact. Does that make sense? So, if I have a schema for "cat", it's easy to place new incoming info immediately where it belongs, cutting down on processing (or the use of working memory.) $\endgroup$ – anongoodnurse Aug 30 '17 at 18:17
  • 1
    $\begingroup$ Yeah, I think I understand, I'm glad I didn't jump to say yes to schemas then, just because of all the additional implications. I would say that hierarchical representations are definitely a way to reduce complexity, and they are certainly related to schemas as you describe them. Where I want to distance them, though, is that having a representation of "cat" doesn't necessarily make all things easy to classify, it depends on the task: sometimes you have to group things, sometimes differentiate them. $\endgroup$ – Bryan Krause Aug 30 '17 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.