Is Knowledge All Large Language Models Needed for Causal Reasoning?