We propose a method of segmenting retinal layers from optical coherence tomography (OCT) images for the diagnosis. The proposed method estimates the pixel-wise labels of each retinal layer and each layer surface position using convolutional neural network (CNN). We introduce CNN to a multi-scale loss and a refinement module to improve the accuracy of pixel-wise labels and layer surface position. Through experiments using a public OCT image dataset, we demonstrate that the proposed method exhibits higher accuracy of segmenting retinal layers than the state-of-the-art methods.