Posted comment on ´ Temporal Integration Windows for Naturalistic Visual Sequences` by S L Fairhill, A. Albi and D. Melcher published in PLoS One vol 9 p e102248 (July 2014)
Fairhill, Albi and Melcher conclude from the results of their experiments that the integration of incoming sensory information over 2-3 second time periods (termed Temporal Integration Windows, TIWs) constitute the real-time natural visual experience (the subjective present). In their experiments they used movie clips as a proxy method for real-life events. They justified this use by saying that typically movie films contain multiple objects, motion and events and require visual field shifts to follow the visual stimuli. Using movie clips, they temporally shuffled the sequence of events over a range of different time scales so that any sensory re-integration would occur within and beyond the 2-3 second period. The movie clips were 12.8 second long visual clips with a frame rate of 25HZ and these were divided into base units of 5 frames (200msecs) where the 4th frame faded and the 5th frame was blank. The clips were shuffled for the designated time periods (TIWs) for the whole video or within the TIW (termed a shuffle chunk). They also looked at experimenter cuts versus ´natural` cuts (termed crosscuts). The participants of the study were asked to record their impression of the amount of effort required to watch the video (termed difficulty to follow, DtF) and score it between 1 and 9 with 9 being for the most difficult. They were also asked in 25% of trials to report on the basic narrative of the film clip so that the level of understanding could be ascertained.
The results of Experiment 1 which had 15 participants showed that DtF was influenced step-wise by the time window shuffling, but there was a large significant increase (146%) between 1600 and 3200 msec. This was consistent with Fairhill, Albi and Melcher`s hypothesis that TIWs of 2-3 seconds duration provide sensory integration important for cognition. The next big increase in TIWs was between 6400 and 12800msecs. In the case of shuffle chunk, linear plots were obtained with shuffle chunk against DtF showing that any changes were the result of low-level visual properties. In their second experiment with 28 participants, Fairhill, Albi and Melcher performed trials with TIWs of 1200, 2000, 2800, 3600 and 4400 msec and 200 and 400 msec shuffle-chunks plus a number of trials with unscrambled and fully scrambled (12800 msec) TIWs. They found that TIW and short shuffles had a significant influence on DtF, just as in Experiment 1. There was a stepwise increase with a significant big increase between 2000 and 2800msec and this was twice the increase (220%) of the next big increase which occurred between 1200 and 2000msecs. Again the linear effect of shuffle chunks and DtF could be explained by low-level visual properties.
Fairhill, Albi and Melcher concluded that integration of sensory material occurs over a period of 2-3 seconds for complex stimuli and this is consistent with our subjective visual experience. However, difficulty is experienced when the incoming information is scrambled over longer timescales.
This research is interesting because it defines with regards to time the limits of a conscious real-time experience of an event. Fairhill, Albi and Melcher define their sensory integration period as being 2-3 seconds in length and this time period relates neurochemically to the visual input, processing and memory formation for the event in question as a whole. There is conscious awareness and we can assume that there is the delay of 170msecs between unconscious input and conscious appreciation of the event (ie the subjective present). In the 2-3 seconds, visual input from the objects in the visual field fire the appropriate neuronal pathways and the binding of the features occurs creating one neuronal cell assembly. This represents one event/percept in accordance to Hebbs hypothesis for neuronal cell assembly formation that cells that fire together, wire together. The cells continue firing whilst the object remains in the visual field until the individual cells reach their refractory periods and then neuron firing mechanisms such as saccades, priority to the unattended and lateral inhibition take over forcing other features of the event to take priority. In the case of the movie clips, input and binding of features leading to the highest degree of difficulty for understanding the movie sequence when shuffling took place occurred when the clips were 2-3 seconds long implying that an event with conscious awareness is an informational unit representing input within a 2-3 second period. It is this informational unit which could constitute the memory of the event and probably relates to short term memory stores rather than sensory stores since visual sensory stores only exist for 0.5 secs.
Fairhill, Albi and Melcher`s experiment did not just indicate the importance of binding of visual features within a certain time interval to represent an event, but also indicated the importance of the serial synchronicity between the various TIWs since participants were asked about the narrative of the whole video sequence. Therefore, the movie clips must have been long enough for successful learning and recall to occur at least on a short-term basis. It is likely that only key features or common features of each period were learnt and remembered, but filling-in and processing would allow the participant to formulate a coherent narrative provided the TIW was long enough for visual learning in the first place to occur. In the case of motion, according to Fairhill, Albi and Melcher, this requires temporal integration over longer periods of time. They said that in apparent motion if two brief stimuli are separated by less than a few hundred milliseconds, then observers see smooth and continuous motion in between the two discrete stimuli. A neurochemical explanation is that in movement memory many of the event features are the same and there is continual firing of those cells representing those features until exhausted and then visual input rules such as lateral inhibition etc means that the changing features are given priority. Hence, the two discrete stimuli would appear as one feature if visual input over a certain time period is accumulated. The characteristics of the stimuli do not change only the location.
Therefore, in terms of memory research Fairhill, Albi and Melcher`s work is interesting because it defines the temporal length of an informational unit and hence, raises opportunities to further study memory content with this in mind.
Since we`re talking about the topic …….
…..can we assume that the activity in brain areas associated with the WHAT (temporal) and WHERE (parietal) pathways in some way also reflects the 2-3 second integration windows?
….would looking at brain waves whilst movie clip shuffling elucidate the difference between object and timing since work by Hsieh and collegues showed that items in working memory (number and temporal order maintenance) produced increased prefrontal theta oscillations during temporal order maintenance and alpha oscillations over posterior parietal and lateral occipital areas for item maintenance?
….what about unconscious processing and its TIW? Would objects hidden in early clips, but important in later clips or minor changes in colour features for example allow the TIW for objects not consciously processed and learnt to be investigated?
….would the use of eye tracking glasses allow the researcher to see exactly what is being looked at and determine which features of each clip are important? Different movie screen sizes could also be used.
…can it be assumed that the administration of cannabinoids causing gamma wave dysfunction leading to decreased learning due to desynchronisation of the neuronal assemblies would have a significant effect on the results of DtF? Would the administration of the NMDA antagonist, ketamine, affect the above results since it would strongly impair the ability of the participant to ignore task irrelevant and hence is likely to affect performance?
…could click trains (Fox and colleagues) be used to increase performance by influencing the delay experienced between input and conscious awareness?