In the traditional direction of arrival (DOA) estimation system, multiple radio frequency (RF) channels and antennas are used to measure the different phases of the arriving signals, which are expensive and limit the applications. In this paper, a low-cost DOA estimation system with only one transmitting and receiving channel is proposed by using the reconfigurable intelligent surface (RIS). By exploiting the target sparsity in the spatial domain, an atomic norm-based method is formulated to describe the DOA estimation problem as a sparse reconstruction problem with multiple measurements, which can also avoid the \emph{off-grid} problem in the method of using discretizing spatial angles. Moreover, to focus the power at the receiver, a modified genetic algorithm as an optimization method is proposed for beamforming schemes, i.e., optimizing the measurement matrix, using a practical RIS, which can only control several discrete angles. For the sparse reconstruction performance, the mutual coefficient between beamforming schemes is also considered in the optimization method. Simulation results show that the proposed method outperforms the existing methods in the DOA estimation performance.