In this paper, we propose a privacy protection system that detects human faces from images captured by a wide-angle camera, which is assumed to be a surveillance camera, and encrypts the face detection area. In the proposed face detection method, a classifier is created by AdaBoost learning based on Haar-like features, and the face region is detected from the image captured by the wide-angle camera. By creating the training data based on the face images captured by the camera, faces without can be detected compromising the detection accuracy, even for surveillance cameras. We use block-scrambling encryption to protect the privacy of the detected face area. During face detection, minimizing the probability of missing a face and allowing a certain number of false positives are necessary from the privacy protection viewpoint. In the case of false positives in previous encryption methods, the color space of the background cannot be preserved, resulting in visual degradation. Therefore, in the proposed encryption method, visual degradation is suppressed by improving the processing of the color components. Through simulations, we evaluate the effectiveness of the proposed method in terms of detection accuracy and processing speed for face detection, as well as color component change and compression efficiency for encryption.