This project can be split into two main parts detecting the eyes and measuring dilation. I will be going through the research put into each part along with the code I wrote to combine Infrared and Color images.
Eye Detection#
The standard way of detecting eyes in a scene would be to use a Cascade Classifier like HAAR1.
These classifiers depend on the quality and quantity data that they are trained on and are quite trivial to implement so we looked else where.
A more core image processing solution is to use templates. These work by dragging a template image across the the destination image and comparing the values2.
Templates work very well in cases whereby the templated image will show up exactly the same in the destination image. For example in a video game where certain elements are just copies of a single sprite as can be seen below with the coins.
In our case by combining the fact that a person's eyes will most likely be an Infrared Image's brightest points with basic contouring a very simple ellipse like shape could be good enough to be used as a template.
A classifier may still be required but this will be our initial solution and should work in most situations.
Dilation Measurement#
As dilation is measured in millimetres it can be very easy for external factors to hurt accuracy. That is why it is important to understand the realities of dilation measurement.
Proportions will be difficult to work with due to the variation by amount of light and age of the person. The normal pupil size in adults varies from 2 to 4 mm in diameter in bright light to 4 to 8 mm in the dark. Both pupils constrict when the eye is focused on a near object (accommodative response). The pupil is abnormal if it fails to dilate to the dark or fails to constrict to light or accommodation
For example when measuring pupil size the best metric to use is said to be the major axis of it's ellipse as it has been found to be the most consistent in practice4
Infrared imagery is a commonly used in industry to measure pupil dilation although this method suffers from the observer effect5 whereby the use of infrared light affect the dilation of pupils. This may have to be accounted for.
Our method of using both Infrared and Color cameras will help in limiting the effects caused by the aforementioned issues with high quality results possible with either solutions6
Commercial and Open Source solutions exist for this problem but seem to rely on dedicated hardware7.
Another route to go down would be to use video. In one paper the researchers concluded that video photography boasted more repeatable and accurate estimates than traditional clinical methods8
Combining Infrared and Color Images#
I honestly though it would be harder to combine the infrared and color images due to the difference in field of view and lens distortion but the code I wrote ended up working quite well.
Resulting Images#
Of course the combined image doesn't look good but that wasn't the point. Now x and y co-ordinates can be easily transferred between the two images.
Infrared#
Color#
Combined#
Footnotes#
Viola and Jones, "Rapid object detection using a boosted cascade of simple features", Computer Cool Vision and Pattern Recognition, 2001
OpenCV Official Documentation Python Template Matching
Spector RH. The Pupils. In: Walker HK, Hall WD, Hurst JW, editors. "Clinical Methods: The History, Physical, and Laboratory Examinations. 3rd edition." Boston: Butterworths; 1990. Chapter 58.
Holmqvist, Kenneth & Nyström, Marcus & Andersson, Richard & Dewhurst, Richard & Jarodzka, Halszka & van de Weijer, Joost. (2011). "Eye Tracking: A Comprehensive Guide To Methods And Measures." Holmqvist, K., Nyström, N., Andersson, R., Dewhurst, R., Jarodzka, H., & Van de Weijer, J. (Eds.) (2011). Eye tracking: a comprehensive guide to methods and measures, Oxford, UK: Oxford University Press.
Weizmann Institute Of Science (27 February 1998). "Quantum Theory Demonstrated: Observation Affects Reality". Science Daily.
McAnany JJ, Smith BM, Garland A, Kagen SL. "iPhone-based Pupillometry: A Novel Approach for Assessing the Pupillary Light Reflex." Optom Vis Sci. 2018;95(10):953–958. doi:10.1097/OPX.0000000000001289
Open Source option Pupil Labs
Twa, Michael & Bailey, Melissa & Hayes, John & Bullimore, Mark. (2004). "Estimation of pupil size by digital photography. Journal of cataract and refractive surgery." 30. 381-9. 10.1016/S0886-3350(03)00619-9.