This work presents an architecture that generates curiosity-driven goal-directed exploration behaviours for an image sensor of a microfarming robot. A combination of deep neural networks for offline unsupervised learning of low-dimensional features from images and of online learning of shallow neural networks representing the inverse and forward kinematics of the system have been used. The artificial curiosity system assigns interest values to a set of pre-defined goals and drives the exploration towards those that are expected to maximise the learning progress. We propose the integration of an episodic memory in intrinsic motivation systems to face catastrophic forgetting issues, typically experienced when performing online updates of artificial neural networks. Our results show that adopting an episodic memory system not only prevents the computational models from quickly forgetting knowledge that has been previously acquired but also provides new avenues for modulating the balance between plasticity and stability of the models.
BaldassarreG.MirolliM. (2013). Intrinsically motivated learning in natural and artificial systems. Springer.
2.
BaltieriM.BuckleyC. L. (2018). The modularity of action and perception revisited using control theory and active inference. In IkegamiT.VirgoN.WitkowskiO.OkaM.SuzukiR.IizukaH. (Eds.), Artificial life conference proceedings (pp. 121–128). MIT Press.
3.
BaranesA.OudeyerP. Y. (2013). Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1), 49–73.
4.
BotvinickM.RitterS.WangJ. X.Kurth-NelsonZ.BlundellC.HassabisD. (2019). Reinforcement learning, fast and slow. Trends in Cognitive Sciences. Advance online publication. https://doi.org/10.1016/j.tics.2019.02.006
5.
CangelosiA.SchlesingerM. (2015). Developmental robotics: From babies to robots. MIT Press.
6.
CangelosiA.SchlesingerM. (2018). From babies to robots: The contribution of developmental robotics to developmental psychology. Child Development Perspectives, 12(3), 183–188.
7.
ChenC. C.ChuH. T. (2005). Similarity measurement between images. In 29th annual international computer software and applications conference (Vol. 2., pp. 41–42). Institute of Electrical and Electronics Engineers. https://ieeexplore.ieee.org/document/1508081
8.
ChenS.LiY.KwokN. M. (2011). Active vision in robotic systems: A survey of recent developments. International Journal of Robotics Research, 30(11), 1343–1377.
9.
ColasC.OudeyerP. Y.SigaudO.FournierP.ChetouaniM. (2019). Curious: Intrinsically motivated modular multi-goal reinforcement learning. In International conference on machine learning (pp. 1331–1340). https://arxiv.org/abs/1810.06284
10.
FeldmanA. G. (2016). Active sensing without efference copy: Referent control of perception. Journal of Neurophysiology, 116(3), 960–976.
11.
ForestierS.MollardY.OudeyerP. Y. (2017). Intrinsically motivated goal exploration processes with automatic curriculum learning. Arxivpreprint Arxiv. https://arxiv.org/abs/1708.02190
12.
FrankM.LeitnerJ.StollengaM.FoersterA.SchmidhuberJ. (2014). Curiosity driven reinforcement learning for motion planning on humanoids. Frontiers in Neurorobotics, 7, Article 25.
13.
FristonK. (2010). The free-energy principle: A unified brain theory?Nature Reviews Neuroscience, 11(2), 127–138.
14.
FristonK. (2011). What is optimal about motor control?Neuron, 72(3), 488–498.
KawatoM. (1999). Internal models for motor control and trajectory planning. Current Opinion in Neurobiology, 9(6), 718–727.
18.
LaraB.AstorgaD.Mendoza-BockE.PardoM.EscobarE.CiriaA. (2018). Embodied cognitive robotics and the learning of sensorimotor schemes. Adaptive Behavior, 26(5), 225–238.
MasciJ.MeierU.CirecsanD.SchmidhuberJ. (2011). Stacked convolutional auto-encoders for hierarchical feature extraction. In HonkelaT.DuchW.GirolamiM.KaskiS. (Eds.), Artificial neural networks and machine learning (pp. 52–59). Springer.
21.
MasseN. Y.GrantG. D.FreedmanD. J. (2018). Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. https://arxiv.org/abs/1802.01569
22.
McClellandJ. L.McNaughtonB. L.O’reillyR. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3), 419–457.
23.
MermillodM.BugaiskaA.BoninP. (2013). The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in Psychology, 4, Article 504.
24.
MirandaL.SchillaciG. (2019, June16–20). An adaptive architecture for portability of greenhouse models [Conference session]. Greensys 2019 (Acta Horticulturae), International Symposium on Advanced Technologies and Management for Innovative Greenhouses, Angers, France.
25.
OudeyerP. Y.GottliebJ.LopesM. (2016). Intrinsic motivation, curiosity, and learning: Theory and applications in educational technologies. Progress in Brain Research, 229, 257–284.
26.
OudeyerP. Y.KaplanF.HafnerV. V. (2007). Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation, 11(2), 265–286.
27.
ParisiG. I.KemkerR.PartJ. L.KananC.WermterS. (2019). Continual lifelong learning with neural networks: A review. Neural Networks, 113, 54–71.
28.
ParisiG. I.TaniJ.WeberC.WermterS. (2018). Lifelong learning of spatiotemporal representations with dual-memory recurrent self-organization. Frontiers in Neurorobotics, 12, Article 78.
29.
PathakD.AgrawalP.EfrosA. A.DarrellT. (2017). Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 16–17). Institute of Electrical and Electronics Engineers.
30.
RolfM.SteilJ. J. (2012, November29). Goal babbling: A new concept for early sensorimotor exploration [Conference session]. Humanoids 2012 Workshop on Developmental Robotics: Can Developmental Robotics Yield Human-Like Cognitive Abilities?Osaka, Japan.
31.
SalgeC.GlackinC.PolaniD. (2013). Approximation of empowerment in the continuous domain. Advances in Complex Systems, 16(3), Article 1250079.
32.
SantucciV. G.BaldassarreG.MirolliM. (2016). Grail: A goal-discovering robotic architecture for intrinsically-motivated learning. IEEE Transactions on Cognitive and Developmental Systems, 8(3), 214–231.
33.
SchillaciG.HafnerV. V.LaraB. (2016). Exploration behaviors, body representations, and simulation processes for the development of cognition in artificial agents. Frontiers in Robotics and AI, 3, Article 39.
34.
SchlesingerM. (2013). Investigating the origins of intrinsic motivation in human infants. In BaldassarreG.MirolliM. (Eds.), Intrinsically motivated learning in natural and artificial systems (pp. 367–392). Springer.
35.
SchmerlingM.SchillaciG.HafnerV. V. (2015). Goal-directed learning of hand-eye coordination in a humanoid robot. In 2015 joint IEEE international conference on development and learning and epigenetic robotics (ICDL-EpiRob) (pp. 168–175). https://doi.org/10.1109/DEVLRN.2015.7346136
ZoiaS.BlasonL.D’OttavioG.BiancottoM.BulgheroniM.CastielloU. (2013). The development of upper limb movements: From fetal to post-natal life. PLOS ONE, 8(12), 1–9.
