Sharief was 85.
He was admitted to a private hospital in Bengaluru a few days ago after he collapsed while getting into his car for Friday namaz.
"One of @INCIndia's senior-most leaders, many times MP and one of India's most successful Railway minister, Karnataka's very own son Sri CK Jaffer Shareef has passed away. A national leader who had a great connect across all communities. a truly secular leader. My condolences," Rao said in a tweet.