A disconcerting ramification of water pollution caused by burgeoning populations, rapid industrialization and modernization of agriculture, has been the exponential increase in the incidence of algal growth across the globe. Harmful algal blooms (HABs) have devastated fisheries, contaminated drinking water and killed livestock, resulting in economic losses to the tune of millions of dollars. Therefore, it is important to constantly monitor water bodies and identify any algae build-up so that prompt action against its accumulation can be taken and the harmful consequences can be avoided. In this paper, we propose a computer vision system based on deep learning for algae monitoring. The proposed system is fast, accurate and cheap, and it can be installed on any robotic platforms such as USVs and UAVs for autonomous algae monitoring. The experimental results demonstrate that the proposed system can detect algae in distinct environments regardless of the underlying hardware with high accuracy and in real time.