Phase diagrams of the antiferromagnetic XY model on a triangular lattice with higher-order interactions.

We study the effects of higher-order antinematic interactions on the critical behavior of the antiferromagnetic (AFM) XY model on a triangular lattice using Monte Carlo simulations. The parameter q of the generalized antinematic (ANq) interaction is found to have a pronounced effect on the phase diagram topology by inducing new quasi-long-range ordered phases due to competition with the conventional AFM interaction as well as geometrical frustration. For values of q divisible by 3, the ground-state competition between the two interactions results in a frustrated canted AFM phase appearing at low temperatures wedged between the AFM and ANq phases. For q nondivisible by 3, with the increase of q one can observe the evolution of the phase diagram topology featuring two (q=2), three (q=4,5), and four (q≥7) ordered phases. In addition to the two phases previously found for q=2, the first new phase with solely AFM ordering arises for q=4 in the limit of strong AFM coupling and higher temperatures by separating from the phase with the coexisting AFM and ANq orderings. For q=7, another phase with AFM ordering but multimodal spin distribution in each sublattice appears at intermediate temperatures. All these algebraic phases also display standard and generalized chiral long-range orderings, which decouple at higher temperatures in the regime of dominant ANq (AFM) interaction for q≥4 (q≥7) preserving only the generalized (standard) chiral ordering.