With the increasing use of drones for various applications, their detection and tracking have become critical for ensuring safety and security. In this paper, we propose an algorithm for detecting and tracking drones from infrared (IR) images in challenging conditions such as noise and distortion. Our algorithm involves YOLOv7 for drone detection and utilizes the SORT algorithm for real-time tracking. To detect distortion in the drone images, we employed a vision transformer in parallel with a customized CNN. The experimental results demonstrate the effectiveness of our approach in challenging conditions and highlight the potential for future developments in drone detection and tracking using deep learning techniques. We achieve a precision of 94.2%, a recall of 92.64%, and a mean average precision (mAP) of 92.6% on the provided test data. The implementation code can be found at: https://github.com/a-bentamou/Drone-detectionand-tracking.