Heinrich Matzinger
Georgia Institute of Technology, Atlanta, USA

We consider the problem of first order logic learning by transformer assuming given an adequate ontology. Hence, in our setting the transformer does not need to learn the vector representation of words, but is given such a vector with components corresponding to all the categorization and properties used in the first order logic. We show that under assumption of independence a disjunction of conjunctions can not be approximated closely as one liner functional followed by a ReLU in case of independent properties. We analyze how this may affect the performance of transformers.