Recent progress in both AI and robotics have enabled the development of general purpose robot platforms that are capable of executing a wide variety of complex, temporally extended service tasks in open environments. This article introduces a novel, custom-designed multi-robot platform for research on AI, robotics, and especially human–robot interaction for service robots. Called BWIBots, the robots were designed as a part of the Building-Wide Intelligence (BWI) project at the University of Texas at Austin. The article begins with a description of, and justification for, the hardware and software design decisions underlying the BWIBots, with the aim of informing the design of such platforms in the future. It then proceeds to present an overview of various research contributions that have enabled the BWIBots to better (a) execute action sequences to complete user requests, (b) efficiently ask questions to resolve user requests, (c) understand human commands given in natural language, and (d) understand human intention from afar. The article concludes with a look forward towards future research opportunities and applications enabled by the BWIBot platform.
AggarwalJKRyooMS (2011) Human activity analysis: A review. ACM Computing Surveys (CSUR)43(3): 16.
2.
ArtziYZettlemoyerL (2011) Bootstrapping semantic parsers from conversations. In: Proceedings of the conference on empirical methods in natural language processing (EMNLP). Stroudsburg, Ediburgh, United Kingdom, 27–29 July 2011, PA: Association for Computational Linguistics, pp.421–432.
3.
BabbJLeeJ (2013) Cplus2ASP: Computing action language in answer set programming. In: Proceedings of the international conference on logic programming and nonmonotonic reasoning (LPNMR). Corunna, Spain, 15–19 September 2013, pp.122–134. Berlin, Heidelberg: Springer Berlin Heidelberg. DOI:10.1007/978-3-642-40564-8 13
4.
BaralCGelfondMRushtonN (2009) Probabilistic reasoning with answer sets. Theory and Practice of Logic Programming9(1): 57–144.
5.
BastianelliEBloisiDCapobiancoR. (2013) On-line semantic mapping. In: Proceedings of the 16th international conference on advanced robotics (ICAR). Montevideo, Uruguay, 25–29 November 2013, pp.1–6. Piscataway, New Jersey: IEEE. DOI:10.1109/ICAR.2013.6766501
6.
CaldiranOHaspalamutgilKOkA. (2009) Bridging the gap between high-level reasoning and low-level control. In: Proceedings of the international conference on logic programming and nonmonotonic reasoning (LPNMR). Potsdam, Germany. 14–18 September 2009, pp.122–134. Berlin, Heidelberg: Springer Berlin Heidelberg. DOI:10.1007/978-3-642-04238-6 29
7.
ChenKLuDChenY. (2014) The intelligent techniques in robot Kejia–The champion of RoboCup@Home 2014. In: RoboCup 2014: Robot World Cup XVIII, pp.130–141. Berlin, Heidelberg: Springer. Doi: 10.1007/978-3-319-18615-3 11.
8.
ChenXJiJJiangJ. (2010) Developing high-level cognitive functions for service robots. In: International conference on autonomous agents and multiagent systems (AAMAS). Toronto, Canada, 10–14 May 2010, pp.989–996. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).
9.
ChenXJinGYangF (2012) Extending with composite actions for robotic task planning. In: International Conference on Logical Programming (ICLP).
10.
ChenXXieJJiJ. (2012) Toward open knowledge enabling for human–robot interaction. Journal of Human–Robot Interaction1(2): 100–117.
11.
ChrungooAManimaranSRavindranB (2014) Activity recognition for natural human robot interaction. In: Social Robotics, pp.84–94. Berlin, Heidelberg: Springer. Doi: 10.1007/978-3-319-11973-1 9.
12.
CohenJ (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement20(1): 37–46.
13.
ColtinBVelosoMMVenturaR (2011) Dynamic user task scheduling for mobile robots. In: Automated Action Planning for Autonomous Mobile Robots. AAAI 2011 Workshop on Automated Action Planning for Autonomous Mobile Robots. San Francisco, California, 7 August 2011. AAAI Press. Available from: http://www.aaai.org/ocs/index.php/WS/AAAIW11/paper/view/3855
14.
CousinsS (2010) ROS on the PR2 [ROS topics]. IEEE Robotics & Automation Magazine17(3): 23–25.
15.
EiterTFaberWLeoneN. (2003) Answer set planning under action costs. Journal of Artificial Intelligence Research. 19: 25–71. DOI:10.1613/jair.1148
16.
ErdemEAkerEPatogluV (2012) Answer set programming for collaborative housekeeping robotics: Representation, reasoning, and execution. Intelligent Service Robotics. 5(4): 275–291. DOI:10.1007/s11370-012-0119-x
17.
ErdemEPatogluV (2012) Applications of action languages in cognitive robotics. In: Correct Reasoning: Essays on Logic-Based AI in Honour of Vladimir Lifschitz. pp.229–246. Berlin, Heidelberg:Springer Berlin Heidelberg, Available at: https://dx-doi-org.web.bisu.edu.cn/10.1007/978-3-642-30743-0_16
18.
ErdemEPatogluVSaribaturZG. (2013) Finding optimal plans for multiple teams of robots through a mediator: A logic-based approach. Theory and Practice of Logic Programming. 13(4–5): 831–846. DOI:10.1017/S1471068413000525
19.
FingerJ (1986) Exploiting Constraints in Design Synthesis. PhD Thesis, Stanford University, Palo Alto, CA, USA.
20.
FoxDBurgardWDellaertF. (1999) Monte Carlo localization: Efficient position estimation for mobile robots. AAAI/IAAI 1999: 343–349. In: Proceedings of the 16th national conference on artificial intelligence and the eleventh innovative applications of artificial intelligence conference innovative applications of artificial intelligence (AAAI ‘99/IAAI ’99), Orlando, Florida, USA, pp. 343–349. Menlo Park, CA, USA: American Association for Artificial Intelligence.
21.
GebserMGroteTSchaubT (2010) Coala: A compiler from action languages to ASP. In: Proceedings of the European conference on logics in artificial intelligence (JELIA). Helsinki, Finland, 13–15 September 2010, pp.360–364. Berlin, Heidelberg: Springer Berlin Heidelberg. DOI:10.1007/978-3-642-15675-5_32
22.
GebserMKaufmannBKaminskiR. (2011) Potassco: The Potsdam Answer Set Solving Collection. AI Communications24(2): 107–124.
23.
GelfondMKahlY (2014) Knowledge representation, reasoning, and the design of intelligent agents: The answer-set programming approach. New York, NY, USA: Cambridge University Press.
24.
GelfondMLifschitzV (1988) The stable model semantics for logic programming. In: Proceedings of the international logic programming conference and symposium (ICLP/SLP). Seattle, Washington, 15–19 August 1988, pp.1070–1080. Cambridge, Massachusetts: MIT Press.
25.
GelfondMLifschitzV (1991) Classical negation in logic programs and disjunctive databases. NewGeneration Computing. 9(3): 365–385.
HarnadS (1990) The symbol grounding problem. Physica D: Nonlinear Phenomena42(1): 335–346.
31.
HavurGHaspalamutgilKPalazC. (2013) A case study on the Tower of Hanoi challenge: Representation, reasoning and execution. In: Proceedings of the international conference on robotics and automation (ICRA). Karlsruhe, Germany, 6–10 May 2013, pp.4552–4559. Piscataway, New Jersey: IEEE.
32.
HelmertM (2006) The fast downward planning system. Journal of Artificial Intelligence Research26: 191–246.
33.
HoffmannJNebelB (2001) The FF planning system: Fast plan generation through heuristic search. Journal of Artificial Intelligence Research : 253–302.
34.
HusseinMETorkiMGowayyedMA. (2013) Human action recognition using a temporal hierarchy of covariance descriptors on 3D joint locations. Proceedings of the international joint conference on artificial intelligence (IJCAI).
35.
KassMWitkinATerzopoulosD (1988) Snakes: Active contour models. International Journal of Computer Vision1(4): 321–331.
36.
KhandelwalPBarrettSStoneP (2015) Leading the way: An efficient multi-robot guidance system. In: Proceedings of the 2015 international conference on autonomous agents and multiagent systems. Istanbul, Turkey, 4–8 May 2015. pp.1625–1633. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems.
37.
KhandelwalPStoneP (2014) Multi-robot human guidance using topological graphs. In: Proceedings of the AAAI spring 2014 symposium on qualitative representations for robots. Palo Alto, California, 24–26 March 2014, pp.65–72. Palo Alto, California: AAAI.
38.
KhandelwalPYangFLeonettiM. (2014) Planning in action language while learning action costs for mobile robots. In: Proceedings of the international conference on automated planning and scheduling (ICAPS). Portsmouth, New Hampshire, USA, 21–26 June 2014, pp. 472–480. Palo Alto, California: AAAI.
39.
KoenigNHowardA (2004) Design and use paradigms for gazebo, an open-source multi-robot simulator. In: Proceedings of the 2004 IEEE/RSJ international conference on intelligent robots and systems (IROS 2004), volume 3. IEEE, pp.2149–2154. Sendai, Japan, 28 September 2 October 2004, Piscataway, New Jersey, USA.
40.
KollarTKrishnamurthyJStrimelG (2013) Toward interactive grounded language acquisition. In: Robotics: Science and systems.
41.
KollarTPereraVNardiD. (2013) Learning environmental knowledge from task-based human–robot dialog. In: Proceedings of the IEEE international conference on robotics and automation (ICRA). Karlsruhe, pp.4304–4309. Karlsruhe, Germany, 6–10 May 2013, Piscataway, New Jersey, USA: IEEE.
42.
KrishnamurthyJKollarT (2013) Jointly learning to parse and perceive: Connecting natural language to the physical world. Transactions of the Association for Computational Linguistics1: 193–206.
43.
KuindersmaSRHanniganERuikenD. (2009) Dexterous mobility with the uBot-5 mobile manipulator. In: Proceedings of the international conference on advanced robotics (ICAR 2009). IEEE, pp.1–7. Munich, Germany. 22–26 June 2009, Piscataway, New Jersey, USA: IEEE.
44.
LeeJLifschitzVYangF (2013) Action language : A preliminary report. In: Proceedings of the international joint conference on artificial intelligence (IJCAI). ISBN978-1-57735-633-2, Beijing, China, 3–9 August 2013, pp.983–989. Palo Alto, California: AAAI.
45.
LeonettiMIocchiLStoneP (2016) A synthesis of automated planning and reinforcement learning for efficient, robust decision-making. Artificial Intelligence241: 103–130.
46.
LifschitzV (2008) What is answer set programming? In: Proceedings of the 23rd national conference on artificial intelligence (AAAI’08), volume 3. AAAI Press, pp.1594–1597. Chicago, Illinois, July 13–17, 2008Publisher Location: Palo Alto, California.
47.
LoweDG (2004) Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision60(2): 91–110.
48.
LuberMArrasKO (2013) Multi-hypothesis social grouping and tracking for mobile robots. In: Proceedings of Robotics: Science and systems. Berlin, Germany. 24–28 June 2013, NewmanPaulFoxDieterHsuDavid (eds.), vol.9. Available at: http://www.roboticsproceedings.org/rss09/index.html
49.
Marder-EppsteinEBergerEFooteT. (2010) The office marathon: Robust navigation in an indoor office environment. In: Proceedings of the 2010 IEEE international conference on robotics and automation (ICRA). IEEE, pp.300–307. Anchorage, Alaska, US, 03–08 May 2010, Piscataway, New Jersey, USA.
50.
MarekVTruszczynskiM (1999) Stable models and an alternative logic programming paradigm. In: The Logic Programming Paradigm: A 25-Year Perspective. Berlin, Heidelberg: Springer Verlag. pp.375–398.
51.
MatuszekCBoLZettlemoyerL. (2014) Learning from unscripted deictic gesture and language for human–robot interactions. In: Proceedings of the 28th AAAI conference on artificial intelligence. Quebéc City, Quebéc, Canada. 27–31 July 2014, pp.2556–2563. Palo Alto, California, USA: AAAI.
52.
MatuszekCFitzGeraldNZettlemoyerL. (2012) A joint model of language and perception. In: Proceedings of the 29th international conference on machine learning. Edinburgh, UK. 26 June–1 July 2012, vol.2, pp.1671–1678, New York, NY, USA: Omnipress.
53.
MatuszekCHerbstEZettlemoyerL. (2013) Learning to parse natural language commands to a robot control system. In: Experimental Robotics, pp.403–415. Heidelberg, Germany: Springer International Publishing.
54.
McCarthyJHayesP (1969) Some philosophical problems from the standpoint of artificial intelligence. In: Machine Intelligence. Edinburgh University Press. pp. 463–502. Edinburgh, UK.
55.
MudrovaLHawesN (2015) Task scheduling for mobile robots using interval algebra. In: Proceedings of the 2015 IEEE international conference on robotics and automation (ICRA). IEEE, pp. 383–388. Seattle, WA, USA, 26–30 May 2015. Piscataway, New Jersey, USA: IEEE.
56.
MunaroMMenegattiE (2014) Fast RGB-D people tracking for service robots. Autonomous Robots37(3): 227–242.
57.
NiemeläI (1999) Logic programs with stable model semantics as a constraint programming paradigm. Annals of Mathematics and Artificial Intelligence. 25(3): 241–273. Heidelberg, Germany: Springer.
58.
OreifejOLiuZ (2013) HON4D: Histogram of oriented 4D normals for activity recognition from depth sequences. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). Columbus, Ohio, 24–27 June 2014, pp. 716–723. Piscataway, New Jersey, USA: IEEE.
59.
PardeNHairAPapakostasM. (2015) Grounding the meaning of words through vision and interactive gameplay. In: Proceedings of the 24th international joint conference on artificial intelligence. Buenos Aires, Argentina. pp.1895–1901. 25–31 July 2015, Palo Alto, California, USA: AAAI.
60.
PereraIAllenJF (2013) SALL-E: Situated agent for language learning. In: Proceedings of the 27th AAAI conference on artificial intelligence. Bellevue, WA, pp.1241–1247. 14–18 July 2013Palo Alto, California, USA: AAAI.
61.
PowerTG (1999) Play and exploration in children and animals. London, UK: Psychology Press.
QuinlanSKhatibO (1993) Elastic bands: Connecting path planning and control. In: Proceedings of the IEEE international conference on robotics and automation. IEEE, pp.802–807.
ReiserUConnetteCFischerJ. (2009) Care-O-bot® 3: Creating a product vision for service robot applications by integrating design and technology. In: Proceedings of the 2009 IEEE/RSJ international conference on intelligent robots and systems. IEEE Press, pp.1992–1998. St. Louis, MO, USA, 10–15 October 2009, Palo Alto, California, USA: IEEE.
66.
RosenthalSBiswasJVelosoM (2010) An effective personal mobile robot agent through symbiotic human–robot interaction. In: Proceedings of the 9th international conference on autonomous agents and multiagent systems, volume 1. International Foundation for Autonomous Agents and Multiagent Systems, pp.915–922. Toronto, Canada, 10–14 May 2010, Richland, South Carolina, USA.
67.
RusuRBBlodowNBeetzM (2009) Fast point feature histograms (FPFH) for 3D registration. In: Proceedings of the IEEE international conference on robotics and automation (ICRA’09). IEEE, pp.3212–3217. Kobe, Japan, 12–17 May 2009, Piscataway, New Jersey, USA: IEEE.
68.
RusuRBCousinsS (2011) 3D is here: Point Cloud Library (PCL). In: Proceedings of the 2011 IEEE international conference on robotics and automation (ICRA’11). IEEE, pp.1–4. Shanghai, China, 09–13 May 2011, Piscataway, New Jersey, USA: IEEE.
69.
RyooMFuchsTJXiaL. (2015) Robot-centric activity prediction from first-person videos: What will they do to me. In: Proceedings of the 10th annual ACM/IEEE international conference on human–robot interaction (HRI). ACM, pp.295–302. Portland, Oregon, USA, 2–5 March 2015. New York, NY, USA: ACM.
70.
RyooMSMatthiesL (2013) First-person activity recognition: What are they doing to me?Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). Portland, Oregon, USA, 25–27 June 2013, pp.2730–2737, Piscataway, New Jersey, USA: IEEE. DOI: 10.1109/CVPR.2013.352
71.
SimonyanKZissermanA (2014) Very deep convolutional networks for large-scale image recognition. arXiv Preprint arXiv:1409.1556.
72.
SinapovJKhantePSvetlikM. (2016) Learning to order objects using haptic and proprioceptive exploratory behaviors. In: Proceedings of the 25th international joint conference on artificial intelligence (IJCAI). New York City, 9–15 July 2016, pp. 3462–3468, Palo Alto, California, USA: AAAI.
73.
SinapovJSchenckCStaleyK. (2014) Grounding semantic categories in behavioral interactions: Experiments with 100 objects. Robotics and Autonomous Systems62(5): 632–645.
SprangerMSteelsL (2015) Co-acquisition of syntax and semantics–An investigation of spatial language. In: Proceedings of the 24th international joint conference on artificial intelligence. Buenos Aires, Argentina, pp.1909–1915. Buenos Aires, Argentina, 25–31 July 2015, Palo Alto, California, USA: AAAI.
76.
SrinivasaSSBerensonDCakmakM. (2012) Herb 2.0: Lessons learned from developing a mobile manipulator for the home. Proceedings of the IEEE100(8): 2410–2428.
TaylorPBlackAWCaleyR (1998) The architecture of the festival speech synthesis system. Third ESCA/COCOSDA Workshop on Speech Synthesis Jenolan Caves House, Blue Mountains, NSW, Australia, 26–29 November 1998, Available at: http://www.isca-speech.org/archive_open/ssw3/ssw3_305.html
80.
TellexSKnepperRLiA. (2014) Asking for help using inverse semantics. Proceedings of Robotics: Science and systems. Berkeley, USA, 12–16 July 2014, Available at: http://www.roboticsproceedings.org/ DOI: 10.15607/RSS.2014.X.024
81.
TellexSKollarTDickersonS. (2011) Approaching the symbol-grounding problem with probabilistic graphical models. AI Magazine32(4): 64–76.
82.
ThiébauxSHoffmannJNebelB (2003) In defense of PDDL axioms. In: Proceedings of the international joint conference on artificial intelligence (IJCAI), Acapulco, Mexico, 9–15 August 2003, pp.961–966.
83.
ThomasonJSinapovJSvetlikM. (2016) Learning multi-modal grounded linguistic semantics by playing “I spy”. In: Proceedings of the 25th international joint conference on artificial intelligence (IJCAI). pp.3477–3483. New York City, 9–15 July 2016, Palo Alto, California, USA: AAAI.
84.
ThomasonJZhangSMooneyR. (2015) Learning to interpret natural language commands through human–robot dialog. In: Proceedings of the 24th international joint conference on artificial intelligence (IJCAI). pp.1923–1929. Buenos Aires, Argentina, 25–31 July 2015. Palo Alto, California, USA: AAAI.
85.
VasquezDOkalBArrasKO (2014) Inverse reinforcement learning algorithms and features for robot navigation in crowds: An experimental comparison. In: Proceedings of the 2014 IEEE/RSJ international conference on intelligent robots and systems (IROS 2014). IEEE, pp.1341–1346. Chicago, Illinois, USA, 14–18 September 2014, Piscataway, New Jersey, USA.
86.
VelosoMBiswasJColtinB. (2015) CoBots: Robust symbiotic autonomous mobile service robots. In: The Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, Texas, USA, 25–30 January 2015, Palo Alto, California, USA. AAAI Press, pp.4423–4429.
87.
WalkerWLamerePKwokP. (2004) Sphinx-4: A flexible open source framework for speech recognition. SMLI TR-2004-139, Mountain View, CA, USA: Sun Microsystems, Inc., Available from: http://dl.acm.org/citation.cfm?id=1698193
88.
WisspeintnerTVan Der ZantTIocchiL. (2009) RoboCup@Home: Scientific competition and benchmarking for domestic service robots. Interaction Studies10(3): 392–426.
89.
XiaLChenCCAggarwalJK (2011) View invariant human action recognition using histograms of 3D joints. Proceedings of the IEEE conference on computer vision and pattern recognition workshop (CVPRW). Providence, RI, USA, 16–21 June 2012, pp. 20–27Piscataway, New Jersey, USA: IEEE.
90.
XiaLGoriIAggarwalJK. (2015) Robot-centric activity recognition from first-person RGB-D videos. In: Proceedings of the IEEE winter conference on applications of computer vision. Waikoloa Beach, Hawaii, USA, 6–9 January 2015, pp.357–364. Piscataway, New Jersey, USA: IEEE.
91.
YoungSGasicMThomsonB. (2013) POMDP-based statistical spoken dialog systems: A review. Proceedings of the IEEE101(5): 1160–1179.
ZhangSSridharanMWyattJL (2015) Mixed logical inference and probabilistic planning for robots in unreliable worlds. IEEE Transactions on Robotics31(3): 699–713.
94.
ZhangSStoneP (2015) CORPP: Commonsense reasoning and probabilistic planning, as applied to dialog with a mobile robot. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, Texas, USA, 25–30 January 2015. pp.1394–1400. Palo Alto, California, USA: AAAI.
95.
ZhangSYangFKhandelwalP. (2015) Mobile robot planning using action language with an abstraction hierarchy. In: Proceedings of the 13th international conference on logic programming and non-monotonic reasoning (LPNMR). Lexington, KY, USA, 27–30 September 2015, pp.502–516. Heidelberg, Germany: Springer.
