In order to tailor the magnetic nanoparticles (MNPs) properties for intended applications, it is crucial to unravelling their fundamental dynamics with respect to excitation magnetic field. In this work, we report on the development of a sensitive AC magnetometer using a resonant excitation coil for this purpose. The excitation coil fabricated from a Litz wire is connected to a capacitor network to reduce the impedance of the circuit efficiently. The high efficiency showed by the excitation coil enables investigation of MNP's dynamics in the nonlinear magnetization region. We demonstrate the sensitivity of the developed system by measuring the harmonics distribution of multi-core iron oxide nanoparticles suspended in solutions with the iron concentration down to 300 ng/ml. We experimentally show that the first harmonic component is not entirely 'transparent' to the diamagnetic background of the carrier liquid compared to the higher harmonics. We also demonstrate the complex magnetization measurement of the iron oxide nanoparticles in solution and immobilized states from 3 Hz to 18 kHz. A highly sensitive exploration of MNPs' dynamics can be expected using the developed AC magnetometer.