Non-stationarity potentially comes from many sources and they impact the analysis of a wide range of systems in various fields. There is a large set of statistical tests for checking specific departures from stationarity. This study uses Monte Carlo simulations over artificially generated time series data to assess the effectiveness of 16 statistical tests to detect the real state of a wide variety of time series (i.e., stationary or non-stationary) and to identify their source of non-stationarity, if applicable. Our results show that these tests have a low statistical power outside their scope of operation. Our results also corroborate with previous studies showing that there are effective individual statistical tests to detect stationary time series, but no effective individual tests for detecting non-stationary time series. For example, Dickey-Fuller (DF) family tests are effective in detecting stationary time series or non-stationarity time series with positive unit root, but fail to detect negative unit root as well as trend and break in the mean, variance, and autocorrelation. Stationarity and change point detection tests usually misclassify stationary time series as non-stationary. The Breusch-Pagan BG serial correlation test, the ARCH homoscedasticity test, and the structural change SC tests can help to identify the source of non-stationarity to some extent. This outcome reinforces the current practice of running several tests to determine the real state of a time series, thus highlighting the importance of the selection of complementary statistical tests to correctly identifying the source of non-stationarity.