Wearable devices are a new form of technology that is quickly gaining popularity among mobile users. These \smart" wearable devices are equipped with a variety of high-precision sensors that enable the collection of rich contextual information related to the wearer and her/his surroundings, which in turn enables a variety of novel applications. The presence of a diverse set of zero-permission sensors on wearable devices, however, also expose an additional attack surface which, if not adequately protected, could be potentially exploited to leak private user information. The rst part of this dissertation aims to develop a comprehensive technical understanding of the privacy risks associated with inference of private user interactions with other cyber and physical systems, primarily using wrist-wearables. A detailed evaluation of novel attack frameworks validate the feasibility of inference attacks on both cyber interfaces, such as mobile keypads and computer keyboards, and on physical systems, such as combination padlocks and safes. In order to thwart these new privacy threats, e ective and usable techniques for detection and mitigation of wearable device misuse is critical and urgently needed. Consequently, the second part of this dissertation aims to protect user interactions by proposing new protection mechanisms, which take two di erent strategies. The proposed design-time protection mechanism tries to prevent inference attacks by altering the interaction interfaces, whereas the proposed run-time protection mechanism utilizes contextual information to dynamically regulate zero-permission sensor data when users are detected to be vulnerable to known inference attacks.