Researched on weakly supervised learning of video action segmentation based on Hidden Markov Model with a polymorphic neural network to infer the emission probability of HMM. Incorporated spectral clustering of frames’ feature vectors into HMM and decoded the HMM with Viterbi algorithm to acquire optimal action label distribution in a video frame sequence.