0
$\begingroup$

Disclaimer: I have an academic background in computer vision but not biological vision.

Background: Classical computer vision is concerned with images from cameras that have a fixed exposure time for all pixels. During this exposure time, each pixel is essentially counting photons. This photon count then translates to a bitmap (usually 8 bits per pixel for monochrome cameras) which can be displayed on monitors.

However, the human/vertebrates brain does (afaik) not operate on integers or floating point values but spikes. Consequently, the brain must convert light to spikes for further processing.

Biologically inspired machine vision: Let me quickly present you the simplified working principle of two major approaches of biologically inspired machine vision. They are the inspiration for this question.

  1. Event Camera: The event camera has independent pixels that respond to changes in their logarithmic intensity $L = \log(I)$ ("brightness"). An event/spike is triggered at a certain pixel as soon as the brightness increment since the last event at the pixel reaches a threshold. Formally, let $e_k = (\mathbf{x}_k, t_k, p_k)$ be the event at pixel $\mathbf{x}_k=(x_k, y_k)$ at time $t_k$ with polarity $p_k\in\{+1, -1\}$ as the sign of the brightness change. Then an event $e_k$ is triggered if $$ L(\mathbf{x}_k, t_k) - L(\mathbf{x}_k, t_k - \Delta t_k) = p_k C $$, where $C>0$ and $\Delta t_k$ is the time elapsed since the last event at the same pixel. C is the contrast sensitivity and determines how much the brightness has to change to trigger an event/spike. This introduction is adapted from section 2.4 of this survey. For a more visual explanation see this introductory video.

Consequence of this working priciple: This camera only generates output when there is motion in the scene. Furthermore, it is not possible to reconstruct the actual intensity given spikes/events of an event camera. This is because the camera only generates output at intensity differences (it is a differential sensor).

  1. Spike Camera: This camera also has independent pixel much like the event camera. However, each pixel integrates luminance intensity over time and emits a spike whenever a threshold is reached. This means that it essentially emits spikes to a rate proportional to the light intensity. More formally, let $I(t)$ be the intensity of a pixel at time $t$. Then a spike is generated at that pixel when $$ \int_0^tI(\tau)\text{d}\tau\geq \phi $$, where $\phi$ is a threshold. More details are given in section 2.1 of this paper. Also have a look at the supplementary video for a visual impression.

Consequence of this working principle: Intensity is conveyed via rates of spikes. This rate is proportional to the measured photocurrent. Thus it is possible to reconstruct the true intensity at a pixel. Unlike the event camera, this camera also generates spikes when the scene is static.

My question: Can the spike generation mechanism of vertebrates vision be explained with these working principles? If not, is there another spike generation model that can be formulated mathematically?

Why I am asking this question: Literature regarding both cameras argue with biological inspiration but I have not found any evidence that either one of those approaches actually mimic the working principle of vertebrates' eyes. Any pointer to publications are welcome.

$\endgroup$
2
  • $\begingroup$ Are you acquainted with Carver Mead's work on bio-inspired, "neuromorphic systems" such as CCD sensors and the like? $\endgroup$ – Rodrigo de Azevedo Jun 27 '20 at 12:59
  • $\begingroup$ I am aware of his pioneering works in neuromorphic systems but have not read his publications in depth because it is out of my field a bit. $\endgroup$ – MrX Jun 27 '20 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.