In contrast with scalar-weighted networks, where bipartite consensus can be achieved if and only if the underlying signed network is structurally balanced, structural balance is no longer a graph-theoretic equivalence to the bipartite consensus in the case of signed matrix-weighted networks. To re-establish the relationship between the network structure and the bipartite consensus, the non-trivial balancing set is introduced which is a set of edges whose sign negation can transform a structurally imbalanced network into a structurally balanced one and the weight matrices associated with edges in this set have a non-trivial intersection of null spaces. We show that necessary and/or sufficient conditions for bipartite consensus on matrix-weighted networks can be characterized by the uniqueness of the associated non-trivial balancing set, in the meanwhile, the contribution of the associated non-trivial intersection of null spaces to the steady-state of the matrix-weighted network is examined. Moreover, for matrix-weighted networks with a positive-negative spanning tree, necessary and sufficient condition for bipartite consensus using non-trivial balancing set is obtained. Simulation examples are provided to demonstrate the theoretical results.