Wireless Body Area Network (WBAN) ensures high-quality healthcare services by endowing distant and continual monitoring of patients' health conditions. The security and privacy of the sensitive health-related data transmitted through the WBAN should be preserved to maximize its benefits. In this regard, user authentication is one of the primary mechanisms to protect health data that verifies the identities of entities involved in the communication process. Since WBAN carries crucial health data, every entity engaged in the data transfer process must be authenticated. In literature, an end-to-end user authentication mechanism covering each communicating party is absent. Besides, most of the existing user authentication mechanisms are designed assuming that the patient's mobile phone is trusted. In reality, a patient's mobile phone can be stolen or comprised by malware and thus behaves maliciously. Our work addresses these drawbacks and proposes an end-to-end user authentication and session key agreement scheme between sensor nodes and medical experts in a scenario where the patient's mobile phone is semi-trusted. We present a formal security analysis using BAN logic. Besides, we also provide an informal security analysis of the proposed scheme. Both studies indicate that our method is robust against well-known security attacks. In addition, our scheme achieves comparable computation and communication costs concerning the related existing works. The simulation shows that our method preserves satisfactory network performance.