Facilitating the interaction of human users and machines with smart devices is important to drive the successful adoption of the Internet of Things in people's homes and at their workplaces. In this paper, we present a system that helps users control their smart environment, by embedding semantic metadata in the representations of smart things. The system enables users to specify a desirable state of their smart environment and produces a machine-readable description that details which steps are necessary to reach this state, where each step corresponds to a Web request to a smart device. A client application that, for instance, runs on the user's smartphone, then implements these steps to reconfigure the user's smart environment. We report on our experiences when integrating semantic technologies with smart devices and on two use cases from the home and office automation domains that we implemented in our office space.