Facial Beauty Prediction (FBP) is a computer vision task of quantifying the beauty of a face. Several solutions to this problem have benefitted immensely from the recent developments in deep learning. However, the majority of current methods train machine learning models to purely predict mean beauty scores, treating FBP solely as a regression task. In addition, deep learning based FBP approaches so far use transfer learning from models trained on general classification tasks such as ImageNet. We propose fine-tuning an ensemble of convolutional neural network (CNN) models originally trained on face verification tasks using a variety of loss functions such as Earth Mover's Distance (EMD) based loss. With this approach, our method can predict the entire beauty score distribution rather than just the mean, and the predicted mean scores have a higher Pearson Correlation (PC) compared to the ground truth scores. This method achieves state of the art results on the MEBeauty dataset in terms of mean absolute error, root mean squared error and PC between the predicted and the ground truth mean scores.