Information technology has revolutionised the exchange of knowledge, but expert know-how is still best imparted in person. However, a new system proposed by scientists in Switzerland and India points to a future in which learning how to do something will be greatly enhanced by computers. And it could work using video game technology.

In a paper published in July, researchers from the Swiss Federal Institute of Technology in Lausanne and the Pune College of Engineering proposed a method of training mechanics by tracking their body movements, using Microsoft’s Kinect motion capture console interface.

The system would trace the body movements of trainee mechanics by identifying and tracking 20 separate joints in the skeleton. It would then guide the mechanic through repairs, step-by-step, overlaying instructions on top of a representation of the mechanic’s own body on a screen. It could also check whether a mechanic is using the correct tools at each stage of a repair by recognising tool shapes.

It could work like music and dance video games, they suggest, in which players must follow actions on screen and win points for getting them right. The mechanics themselves can control the system, either with voice commands such as “next” to move onto the following set of instructions, or using hand gestures.

But if this all sounds more like play than work, the scientists also propose that the system should sound an alarm if the mechanics venture out of range of the system before their work is finished. “This enables the supervisor to check whether [a] mechanic is moving out without completing the allocated work,” the paper says.