The ability to manually solve a tower building task as well as the capacity to predict the goal during the observation of the same task was investigated by the use of eye tracking in 12- and 18-month-old infants. The Mirror Neuron System is proposed to mediate a mapping of observed actions onto the individual’s own motor system. This specialized system for perceiving actions guides predictive goal-oriented eye movements and is therefore understood as the neural basis for the important social competency of action understanding. The matching of observation and executing of goal-oriented actions is thought to be dependent on the individual’s own motor experience and repertoire, implying that the emergence of such predictive gaze behavior is dependent on action development. The results from this study show that goal anticipation during action observation is dependent on performance on the manual task at a group level. In addition, a significant difference between the two groups in both their ability to predict the goal for the observed action and solve the manual task was demonstrated (only the older group were effective in both). In suggesting a link between manual ability and the ability to predict the goals of others’ actions in infants, these results add to the direct matching hypothesis and provide support to its suggested developmental course.