Learned graphical models for probabilistic planning provide a new class of movement primitives

Biologicalmovementgenerationcombinesthreeinterestingaspects:itsmodularorganizationinmovementprimitives(MPs),itscharacteristicsofstochasticoptimalityunderperturbations,anditsefficiencyintermsoflearning.Acommonapproachtomotorskilllearningistoendowtheprimitiveswithdynamicalsystems.Here,theparametersoftheprimitiveindirectlydefinetheshapeofareferencetrajectory.WeproposeanalternativeMPrepresentationbasedonprobabilisticinferenceinlearnedgraphicalmodelswithnewandinterestingpropertiesthatcomplieswithsalientfeaturesofbiologicalmovementcontrol.Insteadofendowingtheprimitiveswithdynamicalsystems,weproposetoendowMPswithanintrinsicprobabilisticplanningsystem,integratingthepowerofstochasticoptimalcontrol(SOC)methodswithinaMP.Theparameterizationoftheprimitiveisagraphicalmodelthatrepresentsthedynamicsandintrinsiccostfunctionsuchthatinferenceinthisgraphicalmodelyieldsthecontrolpolicy.Weparameterizetheintrinsiccostfunctionusingtask-relevantfeatures,suchastheimportanceofpassingthroughcertainvia-points.Thesystemdynamicsaswellasintrinsiccostfunctionparametersarelearnedinareinforcementlearning(RL)setting.Weevaluateourapproachonacomplex4-linkbalancingtask.Ourexperimentsshowthatourmovementrepresentationfacilitateslearningsignificantlyandleadstobettergeneralizationtonewtasksettingswithoutre-learning.