Swimmers took anywhere from 80 to 180 minutes to make the more than 3-mile journey