In modern times, the use of private clouds has become increasingly popular due to their flexibility, scalability, and cost-effectiveness. However, private clouds are defenseless to DDoS assaults, which can cause severe harm to cloud infrastructure as well as services. In particular, protocol-based DDoS attacks pose a significant threat to private clouds as they target specific network protocols and exhaust system resources. This paper presents a novel approach for detecting and preventing protocol-based DDoS assaults on private clouds. The proposed approach employs a multi-layered defense mechanism that includes network-level filtering, protocol-specific anomaly detection, and application-level filtering. The system uses machine learning algorithms to detect abnormal network behavior and employs traffic shaping techniques to control the flow of network traffic. We evaluated our approach using a private cloud environment and conducted extensive experiments to measure its effectiveness in detecting and preventing protocol-based DDoS assaults. The outcomes show that our methodology is highly effective in detecting and preventing protocol-based DDoS assaults while minimizing the impact on legitimate network traffic