Process graph extraction (PGE) is a recently emerged interdiscipline between natural language processing and business process management, which aims to extract process graphs expressed in texts. Previous process extractors heavily depend on manual features and ignore the potential relations between clues of different text granularities. In this paper, we formalize the PGE task into the multi-granularity text classification problem, and propose a hierarchical model to effectively model and extract multi-granularity information without manually defined procedural knowledge. Under this framework, we accordingly propose the coarse-to-fine learning mechanism, training multi-granularity tasks in coarse-to-fine order to share the high-level knowledge for the low-level tasks. To evaluate our approach, we construct two finer-grained datasets from two sentence-level corpora and conduct extensive experiments from different dimensions. The experimental results demonstrate that our approach outperforms the state-of-the-art methods with statistical significance, and the ablation studies demonstrate its effectiveness.