We study an exploration method for model-free RL that generalizes the counter-based exploration bonus methods and takes into account long term exploratory value of actions rather than a single step look-ahead. We propose a model-free RL method that modifies Delayed Q-learning and utilizes the long-term exploration bonus with provable efficiency. We show that our proposed method finds a near-optimal policy in polynomial time (PAC-MDP), and also provide experimental evidence that our proposed algorithm is an efficient exploration method.

In this paper, the multiple operating room (OR) surgical case sequencing problem (SCSP) is addressed. The objective is to maximise total OR utilisation during standard opening hours. The work here is based on a case study of a large Australian public hospital with long surgical waiting lists and high levels of non-elective demand. Due to the complexity of the SCSP and the size of the instances considered herein, heuristic techniques are required to solve the problem. Constructive heuristics are presented based on both a modified block scheduling policy and an open scheduling policy. A number of real-time reactive strategies are presented that can be used to maintain schedule feasibility in the case of disruptions. Results of computational experiments show that the approach presented in this paper can be used to maintain schedule feasibility in real-time, whilst increasing OT utilisation and throughput, and reducing the waiting time of non-elective patients. The framework presented here is applicable to the real-life scheduling of OT departments, and recommendations have been provided regarding implementation of the approach.

In this paper we consider the surgical case sequencing problem (SCSP) under stochastic conditions. In addition to implementing a robust surgical schedule, we investigate the use of a number of reactive strategies that can be used to maintain schedule feasibility. We present a mixed integer nonlinear programming (MINLP) model for the reactive multiple operating room (OR) SCSP that may be suitable for direct implementation on small problem instances. A machine scheduling perspective is considered and the model is equivalent to a resource-constrained parallel-machine scheduling problem with identical machines, machine eligibility restrictions, and machine and job release dates. The explicit objective of the model is to reduce OR idle time, although other common objectives (including time to surgery and overtime) are discussed. The work here is based on a case study of a large Australian public hospital with long surgical waiting lists and high non-elective demand. Results of computational experiments show that the reactive strategies presented in this paper can be used to reduce idle time without putting excessive pressure on surgeons.

Rather than programming, training allows robots to achieve behaviors that generalize better and are capable to respond to real-world needs. However, such training requires a big amount of experimentation which is not always feasible for a physical robot. In this work, we present robot_gym, a framework to accelerate robot training through simulation in the cloud that makes use of roboticists' tools, simplifying the development and deployment processes on real robots. We unveil that, for simple tasks, simple 3DoF robots require more than 140 attempts to learn. For more complex, 6DoF robots, the number of attempts increases to more than 900 for the same task. We demonstrate that our framework, for simple tasks, accelerates the robot training time by more than 33% while maintaining similar levels of accuracy and repeatability.

Text representations using neural word embeddings have proven effective in many NLP applications. Recent researches adapt the traditional word embedding models to learn vectors of multiword expressions (concepts/entities). However, these methods are limited to textual knowledge bases (e.g., Wikipedia). In this paper, we propose a novel and simple technique for integrating the knowledge about concepts from two large scale knowledge bases of different structure (Wikipedia, and Probase) in order to learn concept representations. We adapt the efficient skip-gram model to seamlessly learn from the knowledge in Wikipedia text and Probase concept graph. We evaluate our concept embedding models on two tasks: 1) analogical reasoning, where we achieve a stateof-the-art performance of 91% on semantic analogies, 2) concept categorization, where we achieve a state-of-the-art performance on two benchmark datasets achieving categorization accuracy of 100% on one and 98% on the other. Additionally, we present a case study to evaluate our model on unsupervised argument type identification for neural semantic parsing. We demonstrate the competitive accuracy of our unsupervised method and its ability to better generalize to out of vocabulary entity mentions compared to the tedious and error prone methods which depend on gazetteers and regular expressions. In this paper, we use the terms "concept" and "entity" interchangeably. Hongxia Jin Samsung Research America 665 Clyde Avenue, Mountain View, CA 94043, USA Email: hongxia.jin@samsung.com 2 Walid Shalaby et al. Figure 1 Integrating knowledge from Wikipedia text (left) and Probase concept graph (right). Local concept-concept, concept-word, and word-word contexts are generated from both KBs and used for training the skip-gram model.

It has been argued that it is much easier to convey logical statements using rules rather than OWL (or description logic (DL)) axioms. Based on recent theoretical developments on transformations between rules and DLs, we have developed ROWLTab, a Protege plugin that allows users to enter OWL axioms by way of rules; the plugin then automatically converts these rules into OWL 2 DL axioms if possible, and prompts the user in case such a conversion is not possible without weakening the semantics of the rule. In this paper, we present ROWLTab, together with a user evaluation of its effectiveness compared to entering axioms using the standard Protege interface. Our evaluation shows that modeling with ROWLTab is much quicker than the standard interface, while at the same time, also less prone to errors for hard modeling tasks.

Embeddings of tree-like graphs in hyperbolic space were recently shown to surpass their Euclidean counterparts in performance by a large margin. Inspired by these results, we present an algorithm for learning word embeddings in hyperbolic space from free text. An objective function based on the hyperbolic distance is derived and included in the skip-gram architecture from word2vec. The hyperbolic word embeddings are then evaluated on word similarity and analogy benchmarks. The results demonstrate the potential of hyperbolic word embeddings, particularly in low dimensions, though without clear superiority over their Euclidean counterparts. We further discuss problems in the formulation of the analogy task resulting from the curvature of hyperbolic space.

Multi-hop reasoning is an effective approach for query answering (QA) over incomplete knowledge graphs (KGs). The problem can be formulated in a reinforcement learning (RL) setup, where a policy-based agent sequentially extends its inference path until it reaches a target. However, in an incomplete KG environment, the agent receives low-quality rewards corrupted by false negatives in the training data, which harms generalization at test time. Furthermore, since no golden action sequence is used for training, the agent can be misled by spurious search trajectories that incidentally lead to the correct answer. We propose two modeling advances to address both issues: (1) we reduce the impact of false negative supervision by adopting a pretrained one-hop embedding model to estimate the reward of unobserved facts; (2) we counter the sensitivity to spurious paths of on-policy RL by forcing the agent to explore a diverse set of paths using randomly generated edge masks. Our approach significantly improves over existing path-based KGQA models on several benchmark datasets and is comparable or better than embedding-based models.

In [24], the proposal for an inter-agent communication language (ACL) that gave rise to Java Agent Development Framework (JADE), whose best-known original document is [25] followed by a complementary article [26] and a much more complete text in [27]. The importance of the environment, in which the agents interact, is characterized in a very lucid manner in [28]. All active FIPA specifications are listed in Table I.

Hybrid Bayesian networks have received an increasing attention during the last years. The difference with respect to standard Bayesian networks is that they can host discrete and continuous variables simultaneously, which extends the applicability of the Bayesian network framework in general. However, this extra feature also comes at a cost: inference in these types of models is computationally more challenging and the underlying models and updating procedures may not even support closed-form solutions. In this paper we provide an overview of the main trends and principled approaches for performing inference in hybrid Bayesian networks. The methods covered in the paper are organized and discussed according to their methodological basis. We consider how the methods have been extended and adapted to also include (hybrid) dynamic Bayesian networks, and we end with an overview of established software systems supporting inference in these types of models.