This study presents an efficient end-to-end (E2E) learning approach for the short-term load forecasting of hierarchically structured residential consumers based on the principles of a top-down (TD) approach. This technique employs a neural network for predicting load at lower hierarchical levels based on the aggregated one at the top. A simulation is carried out with 9 (from 2013 to 2021) years of energy consumption data of 50 houses located in the United States of America. Simulation results demonstrate that the E2E model, which uses a single model for different nodes and is based on the principles of a top-down approach, shows huge potential for improving forecasting accuracy, making it a valuable tool for grid planners. Model inputs are derived from the aggregated residential category and the specific cluster targeted for forecasting. The proposed model can accurately forecast any residential consumption cluster without requiring any hyperparameter adjustments. According to the experimental analysis, the E2E model outperformed a two-stage methodology and a benchmarked Seasonal Autoregressive Integrated Moving Average (SARIMA) and Support Vector Regression (SVR) model by a mean absolute percentage error (MAPE) of 2.27%.