This paper verifies impacts that the training data have on estimation accuracy of CNN-based solar irradiance estimation method proposed by the authors. The all datasets, training data and test data, are composed of camera images and solar irradiance data. In this paper, training data is created from three perspectives: the size of data, the bias of data based on solar irradiance and the used days of data. CNN learns the estimation model for solar irradiance using training data. Estimation accuracy is evaluated by average MAE using test data. As a result, first, increasing the size of data improves average MAE. At least, 20% of size of data composed of ten days should be used for training data to achieve high estimation accuracy and short calculation time. Second, the bias of data, average MAE improves as the bias of data gets mitigated. Unbiased data should be used for training. Last, increasing the used days of data has a significant impact on MAE. In addition, the combination of weather changed by the used days also has a significant impact on MAE. As a result, many used days' data should be used for training data to achieve high estimation accuracy. After validating the training data from three perspectives, the training data is evaluated comprehensively. CNN learns the estimation model using the training data composed of selected 19,390 samples from ten days' data. This size of training data is equivalent to 20 % of ten days' data. As a result, MAE reaches 0.0240 kW/m2 which is almost same to the estimation accuracy trained by all data of ten days. The calculation time for training reduces by 75%.