Smart camera measures what you eat

Photography: Shutterstock


You scan your food with your smartphone and get back personalized nutritional advice. That is the future, according to nutrition researchers at Wageningen University & Research. They are teaching cameras to record not just what you eat but also how much. That allows food intake to be monitored objectively.

Monitoring food intake is a crucial element in many research projects and medical treatment programmes. The problem is that using questionnaires for this does not give objective results. How much of the sandwich did you eat, what type of butter was used, how much gravy did you pour on, how many spoonfuls of veg did you take? The answers in questionnaires can differ hugely from the actual intake because of poor estimation skills, ignorance, shame or obstinacy. This affects the reliability of the study.

Water, protein, trans fats, unsaturated fats... WUR Researcher Yannick Weesepoel's cameras have been trained to map the composition of foods, like the butter on your sandwich.

Introducing 3D cameras and hyperspectral cameras that can scan food products could make a significant difference. But how does food scanning work, what are the possibilities and the obstacles, and where could professional scanners be used?

Special scanners

The answers to those questions come from Yannick Weesepoel, project manager and researcher at Wageningen Food Safety Research. His research at Wageningen University & Research (WUR) is part of the Initial Boost programme ‘Measuring and Detecting Healthy Behaviour’, falling under the umbrella of the National Research Agenda financed by the Dutch Research Council (NWO).

‘The idea is that special scanners can resolve the problem of how to monitor food intake’

“The idea is that special scanners can resolve the problem of how to monitor food intake,” says Weesepoel. “We want to develop an objective measurement method that will ultimately be usable by people at home. You will simply scan your food with your smartphone and get back personalized nutritional advice. That’s the future!”

A wealth of information

A hyperspectral camera in combination with a 3D camera can provide a wealth of information. Hyperspectral cameras can detect many different colours as well as near infrared and infrared. 3D cameras can determine the shape and volume of the food.

Getting a smartphone to recognize green beans is quite feasible for people at home. Photo: Shutterstock

Weesepoel: “The near infrared in particular gives a lot of information about product composition, for example the fat content. If a beam of near infrared light is directed at a product, a proportion of that beam is reflected. Each component of the food product absorbs part of the radiation in its own unique manner. As a result, each component — such as water, fats, proteins or carbohydrates — has its own spectral fingerprint.”

Combining the information from these two cameras lets you form a good impression of the type of food and its composition. But at present there is no one device that combines these two functions. That is now the big challenge, according to Weesepoel.

Plate of food under the camera

Five researchers are currently working on the project. They are programming, testing and validating the cameras. “We are already thinking about the next step in the research: having test subjects try out the cameras,” says Weesepoel. “Testing will start in the lab, for example with test subjects being asked to put a plate of food under the mounted cameras. If a plate is not directly beneath the cameras, this can have a big effect on the measurements.”

The research project faces numerous challenges, says Weesepoel. The almost infinite range of quantities and combinations of possible food products is one big obstacle. “Recognizing a pile of beans or a slice of bread and butter is one thing; correctly identifying what’s in a triple-decker sandwich is quite another matter. You need a perfectly tuned combination of hyperspectral camera and 3D camera for that.”

Recognizing an elaborate sandwich is complex and requires perfectly calibrated cameras. Photo: Shutterstock

So to get reliable results, the cameras need to be trained and calibrated, and a large database created with all the necessary data. This takes a great deal of time and money, which is an obstacle. “We’re therefore looking at whether we can exchange spectral data on products with other laboratories.”

Error margin

The quality of the cameras is another issue. “Simple, affordable cameras are capable of scanning homogenous products. You can use them perfectly well to determine the amount of carbohydrates in an apple or the fat content of a slice of cheese.” Indeed, these ‘food scanners’ have been on the market for some years but they are not being used widely as yet.

However, these simple scanners are not suitable for precise measurements of the composition of more complex products, or combinations of products. “These basic cameras have quite a big error margin. At present, if you want reliable results you still need a 3D camera costing about 200 euros and a high-quality hyperspectral camera costing about 40,000 euros. This means the research project won’t immediately result in a cheap, reliable scanner you can keep at home in a kitchen drawer.”

The spectral camera performs the composition measurements. Photo: WUR

The infrared contains the information that is used to determine the composition. Photo: WUR

This ultimately results in a kind of fingerprint. Photo: WUR

Finally, the experimental setup in a laboratory is exactly the same for every measurement and uses very bright light. The situation in the average home is quite different — there is a great deal of variation in the lighting alone. Outdoor light is much more variable than the light indoors, and the amount of light varies in the course of the day. The scanner will have to be adjusted to suit the lighting if you are to keep getting reliable measurements.

Close collaboration

You have to collaborate in a big project like this, says Yannick. “WUR is putting a lot of emphasis on collaboration between the various departments and that is producing results. We’re now working closely with Wageningen Food & Biobased Research and with Human Nutrition & Health. Why reinvent the wheel if you can make use of other departments’ knowledge and expertise?”

The cameras need to be trained and calibrated before they can produce reliable results

They are also in close contact with other universities. Researchers at Radboud University Nijmegen, for example, have produced useful solutions for mathematical issues: they are validating the statistics and coming up with options for new calculation methods. “You can get good results much faster if you cooperate with the right people.”


The research project shows that the error margin for a well-trained and properly calibrated hyperspectral camera is only 5 per cent, while the 3D camera has error margins of 5 to 10 per cent. Those are good results, says Weesepoel. “The study design was on the ambitious side, as it turns out that training and calibrating such cameras is very hard work. But the design and the results are good. We expected more setbacks, such as bigger error margins for the measurements, but it is going surprisingly smoothly.”

It will take a while before this professional food scanner can be used on a large scale, says Weesepoel. “But we always keep sight of the ultimate goal: objective, personalized nutritional advice, which will let us make real progress in optimizing human health.”

Share this story

Next article

Preparing farmers for more extreme weather | Climate-resilient agriculture