Flavobacterium psychrophilum, the causative agent of bacterial cold-water disease, is a devastating, worldwide distributed, fish pathogen causing significant economic loss in inland fish farms. Previous epidemiological studies showed that prevalent clonal complexes (CC) differ in fish species affected with disease such as rainbow trout, coho salmon and ayu, indicating significant associations between particular F. psychrophilum genotypes and host species. Yet, whether the population structure is driven by the trade of fish and eggs or by host-specific pathogenicity is uncertain. Notably, all F. psychrophilum isolates retrieved from ayu belong to Type-3 O antigen (O-Ag) whereas only very few strains retrieved from other fish species possess this O-Ag, suggesting a role in outbreaks affecting ayu. Thus, we investigated the links between genotype and pathogenicity by conducting comparative bath infection challenges in two fish hosts, ayu and rainbow trout, for a collection of isolates representing different MLST genotypes and O-Ag. Highly virulent strains in one host species exhibited low to no virulence in the other. F. psychrophilum strains associated with ayu and possessing Type-3 O-Ag demonstrated significant variability in pathogenicity in ayu, ranging from avirulent to highly virulent. Strikingly, F. psychrophilum strains retrieved from rainbow trout and possessing the Type-3 O-Ag were virulent for rainbow trout but not for ayu, indicating that Type-3 O-Ag alone is not sufficient for pathogenicity in ayu, nor does it prevent pathogenicity in rainbow trout. This study revealed that the association between a particular CC and host species partly depends on the pathogen’s adaptation to specific host species.