Current vision-based roughness measurement methods are classified into two main types: index design and deep learning. Among them, the computation procedure for constructing a roughness correlation index based on image data is relatively difficult, and the imaging environment criteria are stringent and not universally applicable. The roughness measurement method based on deep learning takes a long time to train the model, which is not conducive to achieving rapid online roughness measurement. To tackle with the problems mentioned above, a visual measurement method for surface roughness of milling workpieces based on broad learning system was proposed in this paper. The process began by capturing photos of the milling workpiece using a CCD camera in a normal lighting setting. Then, the train set was augmented with additional data to lower the quantity of data required by the model. Finally, the broad learning system was utilized to achieve the classification prediction of roughness. The experimental results showed that the roughness measurement method in this paper not only had a training speed incomparable to deep learning models, but also could automatically extract features and exhibited high recognition accuracy.