Howard Patterson took up swimming to recover from a bout with polio when he was 10-years-old. His mother died when he was 13-years-old, and his father was killed in a car crash only a few months later. Raised by relatives, he immersed himself in swimming as an outlet from his tough childhood. He later attended Michigan State University, where he won two NCAA backstroke titles. He also swam for the Navy swim team after he was drafted. Patterson would later swim from St. Ignace, Michigan to Mackinac Island, the only person ever to do so swimming backstroke. Patterson worked with the Michigan State Liquor Control Commission in Detroit, eventually being named district supervisor. In 2009 he was inducted into the Saginaw County Sports Hall of Fame.