
\chapter{Study of the ground state of \HgBr}
\label{sec:sciencepaper}
%\chaptermark{Optional running chapter heading}



% Place the author information here.  Please hand-code the contact
% information and notecalls; do *not* use \footnote commands.  Let the
% author contact information appear immediately below the author names
% as shown.  We would also prefer that you don't change the type-size
% settings shown here.





\section{Introduction}

Recently a number of groups have shown theoretically how fluctuating dipolar degree of freedom can appear in molecular-based  Mott insulators, and lead to a spin liquid state in presence of charge-spin coupling. Among suggested model systems are quasi-two-dimensional BEDT-TTF molecule-based Mott insulators \cite{Hotta2010,Naka2016} and optical lattices of dipolar molecules \cite{Yao2015}. In quantum paraelectrics fluctuations of electric dipoles  are observed in the vicinity of a ferroelectric transition~\cite{Rowley2014ferroelectric}. A quantum dipole liquid in an antiferroelectric on a triangular lattice was recently  reported for  BaFe$_{12}$O$_{19}$~\cite{Shen2016}, but it is a band insulator and thus  non-magnetic. A quantum dipole liquid in a Mott insulator is a new paradigm for quantum states of matter which brings together quantum paraelectrics and spin liquids.


In this work we discuss a new experimental realization of the quantum dipole liquid state in a  layered organic crystal $\kappa$-(BEDT-TTF)$_2$Hg(SCN)$_2$Br based on  molecule BEDT-TTF~\footnote{bis(ethylenedithio)tetrathiafulvalene}. It is a Mott insulator, and  in a presence of charge-spin coupling may be a new spin liquid candidate. Electronic and magnetic phenomena observed in this class of materials are defined by the properties of the molecular-based
cation layers (Fig~\ref{fig:Structure}~a).
In a Mott insulator of this origin,   electrons are localized on lattice sites of dimers (BEDT-TTF)$_2^{+1}$ with S=1/2 per site. These dimer sites form layers which can be represented by  two-dimensional anisotropic triangular lattice~(see Fig.~\ref{fig:Structure}~c).
In most of such compounds the dimers have an inversion center  and thus zero electric dipole moment.
Frustration of the lattice, competing electronic correlations and magnetic interactions can lead to charge distributed non-symmetrically  between the two molecules in a dimer~\cite{Dayal2011,Hotta2010}, producing a dipole. This can lead to a broken symmetry ground state, so called ``paired electron solid''~\cite{Dayal2011} or ``quantum dipole solid''~\cite{Hotta2010} where the dipoles order forming a ferroelectric state~(Fig.\ref{fig:Structure}~d)~\cite{Dayal2011}.
In contrast to a displacive ferroelectric,   a change in the underlying lattice is not necessary in this case. It was suggested~\cite{Hotta2010}, that dipoles can  fluctuate in a quantum dipole liquid~(Fig.\ref{fig:Structure}~e) and be an  explanation of the spin liquid state observed in a triangular lattice  \CuCN. However, the evidence for fluctuating quantum dipoles in this material remains elusive~\cite{Yakushi2015,Sedlmeier2012}.

Here we elucidate properties of the quantum dipole liquid state in a triangular lattice Mott insulator~\cite{Aldoshina1993,Drichko2014} (Fig.\ref{fig:Structure}~b) $\kappa$-(BEDT-TTF)$_2$Hg(SCN)$_2$Br (\HgBr) (T$_{MI}$= 80 K~\cite{Lyubovskii96,Ivek2017})  by a comparison to  an isostructural compound $\kappa$-(BEDT-TTF)$_2$Hg(SCN)$_2$Cl  (\HgCl) which shows signatures of a quantum dipole solid below 30~K ~\cite{Lyubovskii96,Drichko2014}, and suggest what parameter may control their ground state.

\begin{figure}[t]
\centering
\includegraphics[width=\textwidth]{Fig1_f.pdf}
\makeatletter
\let\@currsize\normalsize
\caption{(a) Schematic structure of BEDT-TTF based crystal, in the BEDT-TTF-based cation layer the molecule is highlighted in red; (b) Structure of BEDT-TTF layer in $(bc)$ plane of \HgCl\ crystal as determined from X-ray diffraction~\cite{Drichko2014}.  BEDT-TTF molecules are  bound in dimers as highlighted by circles. These dimer sites form anisotropic triangular lattice. (c) Schematic structure of BEDT-TTF layer in a dimer Mott insulator on an anisotropic triangular lattice formed by (BEDT-TTF)$^{+1}_{2}$ sites with S=1/2 (spins depicted by green arrows) and magnetic exchange between sites $J_M$ and $J'_M$.  The model is relevant to the spin liquid candidate \CuCN\ with $J_M/J'_M$=0.64~\cite{Kandpal2009}. (d) Schematic structure of BEDT-TTF layer in case of dipole solid (paired electron crystal~\cite{Dayal2011}). Within (BEDT-TTF)$^{+1}_{2}$ dimer sites charge-rich and charge-poor molecules are  denoted by red and blue colour respectively. The dimer sites thus possess a dipole moment. $J_{DC}$ is a magnetic interaction between spins (marked by green arrows) on neighboring   charge-rich molecules. According to Ref.~\cite{Dayal2011} spins of the nearest neighbor charge-rich sites will form spin-singlets.  This situation is related to \HgCl. (e) Schematic structure of BEDT-TTF layer in case of  the quantum dipole liquid.  The charge is fluctuating between the molecules in (BEDT-TTF)$^{+1}_{2}$ dimers, as denoted by blurry red and blue ovals, leading to electric dipoles fluctuation. Relevant spins also show fluctuations. This situation is related to \HgBr.}
\label{fig:Structure}
\end{figure}


\section{Results: Raman scattering data}
\subsection{Raman Spectra of Vibrational Modes}
We first use Raman molecular vibrational spectroscopy to follow  the distribution of charge on the lattice of these systems through the metal-insulator transition. The on-molecule charge is probed by measuring the frequency of the central C=C molecular bond  vibration ($\nu_2$ ) (Fig.\ref{fig:nu}~e), which  changes by $\sim$ -140~\cm\ when  the charge state changes from (BEDT-TTF)$^0$ to (BEDT-TTF)$^{+1}$~\cite{Dressel04,Yamamoto2005}. This is a result of a lengthening of the central C=C bond of the molecule when more charge occupy the highest occupied molecular orbital (HOMO). To emphasize the changes due to charge effects we also follow the temperature dependence of the  $\nu_3$ mode at 1470~\cm\ (Fig.~2) which involves C=C deformations but is known not to be  coupled to the HOMO. It stays a narrow single line through the measured temperature range for both materials. 

For \HgCl\ a single $\nu_2$ band at about 1490~\cm\ is observed in the high temperature metallic state, while in the insulating state below $T_{CO}$=30~K   $\nu_2$ is split into two bands at 1475 and 1507~\cm  (see Fig.\ref{fig:nu} a). % Other  BEDT-TTF Raman-active vibrational modes are unchanged on the  transition.
The difference in frequencies of the two modes is much higher than expected for a structural phase transition, but corresponds to charges redistributed within the (BEDT-TTF)$^{+1}_2$ dimer as $+0.4e$ on one molecule, and $+0.6e$ on the other ~\cite{Yamamoto2005,Drichko2014} which  breaks the inversion symmetry in a (BEDT-TTF)$^{+1}_2$ dimer creating an electric dipole. This result confirms that the ground state of \HgCl\ is an order of electric dipoles localized on (BEDT-TTF)$^{+1}_2$ dimer sites~\cite{Drichko2014}, so-called  dipole solid~\cite{Hotta2010}.

In contrast, for \HgBr\ a single $\nu_2$ mode observed in the whole studied temperature range suggests a symmetric dimer with both molecules carrying half a hole (BEDT-TTF)$^{+0.5}$ on average. However, the width of $\nu_2$ band shows abnormal behaviour on cooling. Line widths of phonons are determined by decay mechanism~\cite{Kim2012}, disorder and dynamics of the lattice and charge system. For example, a width of $\nu_3$ vibrational band of BEDT-TTF molecule is determined by decay processes into lower-frequency modes and decreases down to about 5~\cm\ at 10~K (see Fig.\ref{fig:nu}~d). In contrast, the width of $\nu_2$ goes through a minimum of 16~\cm\ at around 80~K, and increases again up to 20~\cm\ at 10~K~(see Fig.\ref{fig:nu}~c,d). This behaviour of $\nu_2$ band excludes possibility of structural changes or structural disorder as its origin. Another possible reason  for an increased line width which we will consider  is charge fluctuations.~\cite{Yakushi2012_Crystals,Girlando2014}.

%\subsection{Temperature dependence of vibrational modes for \HgBr}

In the spectra of of \HgBr\  $\nu_2$ mode hardens on cooling similar to \HgCl. At temperatures between 150 and 100 K we observe a small change of a slope of the temperature dependence of the position of the  $\nu_2$ mode. This change occurs still in the metallic state \cite{Ivek2017}. The change  is close to the size of an error with which we determine parameters of the bands. Here the size of triangle/rhombs with which the data are plotted depict the size of the error bars. A similar change of a hardening slope was also noted, for example, for $\kappa$-(BEDT-TTF)$_2$Cu[N(CN)$_2$]Br in Ref.~\cite{Maksimuk2001}. A reason for the small change of hardening behaviour could be the following (i) As mentioned in Ref.~\cite{Maksimuk2001},  hardening depends on the compression of the lattice, thus changes in the latter can define it. At this point no such precise temperature dependence of the crystal structure has been measured. (ii) While there is a good understanding of the dependence of the frequency of $\nu_2$ mode  on the charge distribution on the lattice in the insulating state, at this point it is not clear how this frequency would change due to interactions of this phonon with itinerant electrons. These interactions can be a reason for the change of behaviour.

We used Eq. 1 to reproduce $\nu_2$ line shapes for low temperature \HgCl\ and \HgBr\, and also some line shapes which are not present in experimental data, however illustrate the evolution of the line shape in the ``two-sites jump model''. The results of our calculations are presented in Fig.\ref{fig:nu}b. Parameters for which the calculations are done are listed in the Table 4.1.

%\subsection{Temperature dependence of vibrational modes for \HgCl}

%Both $\nu_2$ and $\nu_3$ have large intensity in A$_{1g}$ and B$_{1g}$ symmetries.
In Fig.\ref{fig:Fig}a we present temperature dependence of parameters of $\nu_3$ and charge-sensitive $\nu_2$ vibrations for \HgCl. $\nu_3$ band shows normal hardening on cooling and a decrease of the  width, with the temperature dependence coinciding with that   for \HgBr. If no coupling to electronic or magnetic excitations is present, the main process which defines relaxation of a phonon excited state is scattering on lower frequency phonons with frequency denoted here $\omega_0$. Respectively, a decrease of a line width of a phonon on cooling  is determined by a thermal population of the phonon levels $\omega_0$ involved in the scattering process: $\Gamma(T)= \Gamma_0+ \alpha(e^{\frac{\hbar \omega_0}{k_BT}}-1)^{-1}$, where $\Gamma_0$ is temperature-independent scattering rate defined by disorder, and $\alpha$ is a probability of the decay~\cite{Maksimuk2001,Kim2012}.  A decrease of the width of $\nu_3$ is described well by this formula with $\omega_0$=150~\cm, which suggest the most probable decay processes. Our results coincide with that for $\kappa$-(BEDT-TTF)$_2$Cu[N(CN)$_2$]Br in Ref.~\cite{Maksimuk2001}.

The width of the charge-sensitive $\nu_2$ in the spectra of \HgCl\ decreases below that of \HgBr\ on cooling down to approximately 40~K, and then shows a weak increase till the compound reaches the  ordering transition at 30~K, and decreases again below the transition as expected in the ordered state. 


\begin{figure}[t]
\centering
\includegraphics[width=\textwidth]{Fig2_f.pdf}
\makeatletter
\let\@currsize\normalsize
\caption{(a)-(c) Raman spectra   in the region of C=C vibrations of BEDT-TTF:  (a) Temperature dependence of the \HgCl\ spectra in the region of $\nu_2$ and $\nu_3$ modes. Note the splitting of  $\nu_2$ mode in the dipole solid (charge ordered) state at 8~K with frequencies corresponding to  BEDT-TTF$^{+0.4}$ and BEDT-TTF$^{+0.6}$.   (b) Shape of $\nu_2$ mode calculated by two-sites jump model, see Eq.~\ref{formula} (SI). The upper spectrum is of a static system ($\omega_{ex}$) with bands corresponding to BEDT-TTF$^{+0.4}$ and BEDT-TTF$^{+0.6}$ as in dipole solid state of \HgCl. Note that on the increase of exchange frequency $\omega_{ex}$ the bands widen and move close to each other. The lower two spectra at $\omega_{ex}$= 30 and 40~\cm\ reproduce the $\nu_2$ shape of \HgBr\ at 8 and 35~K correspondingly, taking into account the natural width $\Gamma$ for the relevant temperature. (c) Temperature dependence of the \HgBr\ spectra in the region of $\nu_2$ and $\nu_3$ modes. Note that the $\nu_2$ band does not split but shows some widening at lowest temperature. (d) Temperature dependence of centerfrequency (upper panel) and line width (lower panel) for $\nu_2$ (triangulares) and $\nu_3$ (diamonds) modes for \HgBr. Note that line width of $\nu_2$ for \HgBr\ goes through a minimum at around 75~K, while it decreases continuously for $\nu_3$.  (e) BEDT-TTF molecule with marked movements of atoms on $\nu_2$ and $\nu_3$ vibration. } 
\label{fig:nu}
\end{figure}


\subsection{Low-frequency Raman spectra of \HgBr\ and \HgCl\ }

In Fig. \ref{fig:nu} and \ref{fig:Fig} we present temperature dependence of the Raman scattering spectra  for \HgBr\ and \HgCl\ compounds for  A$_{1g}$ and B$_{1g}$ symmetry in the spectral region between 25 and 400~\cm. It shows that (i) Similar to A$_{1g}$ for \HgBr, the background feature due to the collective mode at about 40~\cm\ appears also in spectra of   \HgBr\ in B$_{1g}$ symmetry channel. (ii) The  low frequency phonons in \HgBr\ B$_{1g}$ spectra remain broad, which can be explained by an interaction of these phonons  with the collective mode. This is in contrast with spectra of \HgCl\ and    \HgBr\ in A$_{1g}$ polarization, where at low frequencies narrow phonon modes are superimposed on the collective mode background. (iii) Some drop of intensity in \HgCl\ spectra between 35 and 7~K is due to an opening of a gap due to an ordering transition.

Polarization dependence is an important way to characterize excitations observed in Raman scattering~\cite{Devereaux07}. For D$_{4h}$ symmetry a clear separation of A$_{1g}$ ($x^2+y^2$), B$_{1g}$ ($x^2-y^2$), and B$_{2g}$ ($xy$) polarization is expected for electronic and magnetic excitations. Spectra of some materials can be mapped on D$_{4h}$  symmetry and understood within it even if the symmetry of the unit cell is lower. A good example is the antiferromagnetically ordered organic compound $\kappa$-(BEDT-TTF)$_2$Cu[N(CN)$_2$]Cl, where in a monoclinic unit cell (BEDT-TTF)$_2^{+1}$ dimers  form  a weakly frustrated square lattice, and  two-magnon Raman excitations in that material follow the symmetry selection rules expected for D$_{4h}$.

 In the materials studied in this work,   the symmetry of the unit cell is monoclinic as well, however it was shown~\cite{Drichko2014} that the (BEDT-TTF)$_2^{+1}$ dimers in $(bc)$ plane form  slightly anisotropic triangular lattice. For D$_{3h}$  point group Raman-active   symmetries cannot be fully separated by selecting polarizations. It was shown by numerical calculations, too, that magnetic excitations on a triangular lattice loose their anisotropy between A$_{1g}$ and B$_{1g}$~\cite{Vernay2007,Perkins08}, a similar charge is expected for electronic excitations. This explains the presence of the collective mode in spectra of \HgBr\ in   A$_{1g}$, B$_{1g}$ ($xy$), and $x^2-y^2$ which would exist as  B$_{2g}$ only in  D$_{4h}$.


On the other hand,  polarization dependence of   phonons follows   the full symmetry of the lattice. Our experimental data suggest that the  coupling to the collective mode is  larger for B$_{1g}$ phonons than for  A$_{1g}$. This fact can be a realization of the expected anisotropy of the electronic excitation.

\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{Fig3_f.pdf}
\makeatletter
\let\@currsize\normalsize
\caption{(a)-(b) Temperature dependence of low-frequency Raman spectra of (a) \HgBr\ and (b) \HgCl\ in A$_{1g}$ symmetry.  Phonons are found at similar frequencies for both compounds. In the spectra of \HgBr\  a  background develops at temperatures below 100~K.  Spectra at 300 and 11~K for \HgBr\ for B$_{1g}$ symmetry are shown in the inset in (a). The collective mode is observed in B$_{1g}$ with  more prominent coupling to the phonons. (c) Temperature dependence of the collective mode in  A$_{1g}$ symmetry for \HgBr\  determined by extracting phonons from the full Raman spectrum.  The inset shows the temperature dependence of the normalized intensity of the collective mode. (d) Temperature dependence of heat capacity for \HgCl\ (red line) and \HgBr\ (black line) below 40~K.  C$_p$ shows  similar temperature dependence for these materials below 30~K. The difference in the behaviour is observed below approximately 6~K. The inset shows low temperature data with linear behaviour of heat capacity for \HgBr.  (e) Raman spectra in  B$_{1g}$ polarization at 20~K in the range between 800 and 1100~\cm\ with phonons and luminescence background extracted for \CuCN (upper panel) and \HgCl\ and \HgBr\ (lower panel). Schematic pictures of the relevant models with different charge distibution are shown.   The spectra of the dimer Mott insulator on the triangular lattice \CuCN (upper panel) demonstrate   magnetic excitations   below approx. 600~\cm. This feature is absent in the spectra of both \HgBr\ (black) and \HgCl\ (red). The increase of intensity in the spectra of \HgBr\ below 200~\cm\ is due to the collective mode fully shown in panel(a).}
\label{fig:LowF}
\end{figure}

\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{SupplementInfo1.pdf}
\makeatletter
\let\@currsize\normalsize
\caption{ (a) Temperature dependence of parameters of $\nu_2$ and $\nu_3$ vibrational modes for \HgCl. Blue line in the lower panel shows a fit of the width of $\nu_3$ by $\Gamma(T)= \Gamma_0+ \alpha(e^{\frac{\hbar \omega_0}{k_BT}}-1)^{-1}$, with $\Gamma_0$=5.5~\cm, $\alpha$=15, $\omega_0$=150~\cm. (b) Temperature dependence of $(bc)$ spectra for  \HgBr\ (upper panel) and \HgCl (lower panel). Note a collective mode which appears at low frequencies in the spectra of \HgBr, similar to $bb$ polarization. The phonons are widened by the interaction with the collective mode. (c) Comparison of temperature dependance of heat capacity of \HgBr\ and \HgCl\ with that of \CuCN.}\label{fig:Fig}
\end{figure}


\section{Analysis and Discussion}
We  estimate the effects  of charge fluctuations on the shape of the $\nu_2$ vibration using "two- site jump" model~\cite{Yakushi2012_Crystals,Girlando2014} presented by Eq. 1. In this model, we consider $+0.6e$ and $+0.4e$ charged molecules observed in the ordered state of k-Hg-Cl as two static species. They are characterized by frequencies of $\nu_2$ vibrations  $\nu_2$[BEDT-TTF$^{+0.4}$] = 1475~\cm and $\nu_2$[BEDT-TTF$^{+0.6}$]= 1507~\cm. Their natural width $\Gamma$ depends on temperature through the lifetime of the measured excited state and is expected to be the same as that of $\nu_3$. The system can jump between these two  states with a frequency of $\omega_{EX}$=$1/\tau$, where $\tau$ is a life time of each state defined by the exchange. As an exchange rate $\omega_{EX}$ between these two states  increases, the shape of the resulting spectra changes as shown in Fig.\ref{fig:nu}~c. The two original bands get wider, move together and at high enough rate $\omega_{EX}$ merge into  single band.
The calculated spectrum at $\omega_{EX}$= 0 reproduces the doublet shape of $\nu_2$  for \HgCl\ at 8.5~K (Fig.\ref{fig:nu}~a,b).
The spectra calculated for  $\omega_{EX}$ = 40 and 30~\cm\    reproduce the shape of the $\nu_2$ band in \HgBr\ spectra, where the width of  $\nu_2$ increases from  16 to   20~\cm\ on cooling below 80~K and the band gains   slight asymmetry~(Fig.\ref{fig:nu}~b,c).   These results suggest that in  \HgBr\ charges fluctuate between two molecules in a dimer with frequency $\omega_{EX}$ which slightly decreases on cooling.  In other words, electric dipoles in \HgBr\ fluctuate with this frequency,  forming a quantum dipole liquid state.

Apart from the difference in the $\nu_2$ band behaviour,  the phonon  Raman spectra of   \HgBr\ and \HgCl\ are very similar  (Fig.\ref{fig:LowF} a,b,e), since the compounds possess very similar crystal structural. On cooling, all the bands narrow down to 5-7~\cm, including  the low frequency lattice phonons, which are well distinguished in low temperature spectra in Fig.\ref{fig:LowF} ~a,b below approximately~100~\cm.
In the spectra of \HgBr\ the somewhat widened by few wavenumbers phonons, some of them demonstrating so-called Fano shape,  are superimposed on a much wider background band observed in A$_{1g}$ and  B$_{1g}$ symmetries (see A$_{1g}$ symmetry spectra in Fig.\ref{fig:LowF}, B$_{1g}$ symmetry in Fig. SI ).  This asymmetric feature with a maximum around 40~\cm\ (see  spectra with phonons extracted in Fig.\ref{fig:LowF} ~c) with the width at half maximum of about 40~\cm\ gains intensity  below 100~K, when \HgBr\ enters the insulating state, and shows weak softening at the lowest temperature.
Apparently, this  wide feature observed only in  \HgBr\ spectra    originates from a different  scattering channel than phonons. Potential other scattering channels are electronic or magnetic excitations on a triangular lattice of (BEDT-TTF)$_2^{1+}$ dimers. On a triangular lattice polarizations of electronic or magnetic excitations cannot be completely disentangled to  elucidate the origin of the excitations, in contrast to a square lattice ~\cite{Devereaux07}.

Magnetic excitations are expected in the Raman response of a Mott insulator with ordered spins or even spins developing short range correlations~\cite{Devereaux07,Drichko2015,Nakamura2014,Nakamura2016}. In Mott insulators based on (BEDT-TTF)$^{+1}_2$ dimers magnetic excitations are observed both in Raman spectra of an antiferromagnetically ordered state on a square lattice~\cite{Drichko2015} and in a spin liquid candidate on triangular lattice \CuCN~(see Fig.\ref{fig:LowF} ~e). The spectra of the latter show  a continuum  of magnetic excitations below 600~\cm, as marked in the spectra in Fig~3~e by a shaded area.  The position of the continuum is defined by the value of $J$ and geometry of the lattice~\cite{Devereaux07}. For \CuCN\ it is  in agreement with  Hubbard-model-based calculations for the magnetic response of S=1/2 on a anisotropic triangular lattice with $J_M$=250~K~\cite{Nakamura2014}.

It is clear at this point, that magnetic interactions in a dipole  solid, and possibly quantum dipole liquid would be re-normalized in comparison to a simple  (BEDT-TTF)$^{+1}_2$ dimer  Mott insulator with charge symmetrically distributed on a dimer.  Ref.~\cite{Hotta2010} suggests  a re-normalization and a decrease of $J$   in a quantum dipole liquid compared to a simple dimer Mott insulator, however without estimating $J$ values.
A simple argument suggests that in a dipole solid magnetic interactions occur between charge-rich molecules of the neighboring dimers, while in a simple Mott insulator the interactions are between dimer lattice sites, as is schematically shown in Fig.\ref{fig:LowF} e. An estimate provided by
a tight-binding approximation   as $J=\frac{4t^2}{U}$, where $t$ is a transfer integral, and $U$ is on-molecule Coulomb repulsion, yields  the value of
about $J_{DS}=80~K$ for a dipole solid, considerably smaller compared to $J_M=250~K$ ~\cite{Shimizu2003} for a simple dimer Mott insulator, where the on-dimer $U$ defines magnetic interactions. Here the Coulomb repulsion parameters, as well as transfer integrals  are estimated from the optical conductivity spectra~\cite{Drichko2007}, and the difference is
produced mainly by a variation between the values of $U$ in these two models. Lower $J$ would result in a lower ordering temperature, and a spectrum of magnetic excitation  shifting to lower frequencies.
However, the  maximum of the observed background   is around 40~\cm\ is found below the expectant $J_{DS}$ value,   which is too low in frequency below the expected $J_{DS}$ value to interpret it as purely magnetic excitations.

Another possibility is an assignment of this   mode as a collective excitation associated with dipole fluctuations. Dipole fluctuations with frequency of about $\omega_{EX}$ = 40~\cm\ are detected through the line shape analysis of charge-sensitive vibrations. If these fluctuations are a collective phenomena, we would expect a collective mode at about 40~\cm. The low-frequency mode observed in \HgBr\ thus is a good candidate for a collective response of dipole fluctuations. 
Optically detected  collective modes associated with charge fluctuations are found in the metallic state close to a charge ordering metal-insulator transition in organic conductors  \cite{Merino2003,Dressel2003} and in under-doped high temperature cuprate superconductors~\cite{Caprara2005}. In an insulating state,  the closest analogy would be a soft mode close to the transition into the ferroelectric state in   displacive ferroelectrics such as SrTiO$_3$. To the best of our knowledge a collective mode due to fluctuations of electric dipoles  in a Mott insulator    is observed by ground state spectroscopy for the   first time here. A comparatively small width of the band, as well as it increase in intensity and low-frequency shift below T$_{MI}$=80~K distinguishes it from a Boson peak observed in glasses~\cite{Malinovsky1986} and supports an interpretation in terms of a  fluctuating system of dipoles versus charge glass. An absence of glassy behavior is also supported by low frequency dielectric response of \HgBr~\cite{Ivek2017}.

Fluctuations of electric dipoles coupled to S=1/2 spins on a triangular lattice of (BEDT-TTF)$^{+1}_2$ dimers   have been suggested as a mechanism for spin-liquid behaviour~\cite{Hotta2010,Naka2016}.
 Ref.~\cite{Naka2016} discusses  the coupling between the dipole and magnetic degrees of freedom within the Kugel-Khomskii model, showing an analogy
between the fluctuating dipole liquid and orbital liquid\cite{GChen_Balents2009,Balents2010}. This model suggests  that at a certain frustration J`/J
and spin-charge coupling $K$ values,   spin order in a system is destabilized, and would produce mixed spin-charge excitations.
To understand if the collective mode observed by us in \HgBr\ originates purely in dipole fluctuations or in mixed charge-spin excitations theoretical calculations of the excitation spectrum for such a system would be of great importance.

Our heat capacity data  suggest the presence itinerant excitations in \HgBr\, but not in \HgCl.
The heat capacity $C_p$ of \HgBr\ and \HgCl\ was measured in the temperature range between 40~K and 100~$\mu$K. The temperature dependence of heat capacity for these compounds is overlapping within the error of the measurements in the temperature range above 6~K (Fig.\ref{fig:LowF} ~d), but for the feature at 30~K in the temperature dependence for the $C_p$ of \HgCl\ indicating the charge order transition.
The heat capacity $C_p=\beta T^3 + \gamma T$ of both compounds shows basically same bosonic contribution $\beta$ = 19.0 $\pm$ 2.5 mJ K$^{-4}$ mol$^{-1}$.  This is natural, since it is determined predominantly by phonons and  vibrations of BEDT-TTF molecules which are very similar for the studied compounds.
The difference between the two materials appears below about 6~K, where for \HgBr\ $C_p$ shows a linear term $\gamma = 13.8 \pm 3.1 mJ K^{-2}mol^{-1}$ (inset in Fig.\ref{fig:LowF} ~d).
Spinon excitations can produce the linear term in heat capacity~\cite{Yamashita2008,Zhou2017} and thus suggest a spin-liquid behaviour of \HgBr. For \HgCl\, $\gamma$= 0 within the precision of our measurements.

An order of electric dipoles represented by  \HgCl\ does not necessary imply  magnetic order~\cite{Hotta2010}. However, an approach of Ref.~\cite{Dayal2011} which includes antiferromagnetic interactions as a driving force for the charge order on a frustrated dimer lattice suggests a spin singlet ground state.
A single phase transition observed at 30~K can be an evidence of simultaneous electric dipole ordering and singlet formation in  \HgCl.
On the other hand, the temperature of spin ordering can be lower than that of the charge order, as is observed in one-dimensional materials, and suggested by calculations~\cite{Naka2016magnetoelectric}. Since  heat capacity is found not to be always sensitive to magnetic phase transitions in these two-dimensional materials, see for example~\cite{Nakazawa2012}, further studies such as NMR  are necessary to identify the  magnetic ground state of \HgCl. Paired electron crystal proposed as a ground state of \HgCl~\cite{Drichko2014}   can be regarded as a variation of valence bond solid~\cite{Dayal2011,Balents2010}.   In these terms, the quantum  dipole liquid in \HgBr\ can be a realization of a resonant valence bond  state~\cite{Anderson1987}.

The quantum dipole liquid was suggested as one possible explanation of the origin of the spin liquid state in  \CuCN~\cite{Hotta2010}, however our work shows that this material  does not demonstrate the fingerprints of this state. Its spectrum of magnetic excitations is well understood within a model of spin 1/2 on a triangular lattice with $J$=250~K (Fig.\ref{fig:LowF} , Ref.~\cite{Nakamura2014}). There is a recent suggestion~\cite{Powell2017} that \CuCN\ experiences a lowering of magnetic dimensionality due to destructive interference of magnetic interactions in one of the directions. A necessary test would be a comparison of magnetic excitation spectra of this model  to available experimental Raman scattering data on magnetic excitations in \CuCN\ and antiferromagnetically ordered BEDT-TTF based material. At this point it is clear that the spectrum of collective excitations in  \HgBr\ is very different from that of \CuCN.   Basing on that we can suggest, that \CuCN\ is a regular dimer Mott insulator with homogeneous distribution of charge on (BEDT-TTF)$_2^{1+}$ dimer on a relevant time scale. If the quantum dipole liquid model is relevant to \CuCN\ at all, it puts this compound quite far from a quantum phase transition into a dipole solid state.
Tuning between quantum dipole solid and quantum dipole liquid can be done by a change of $t_b/t_d$ ratio, where $t_b$ is an overlap integral between the dimers, and $t_d$ an intra-dimer one according to theoretical approach of Ref.~\cite{Hotta2010}. Indeed,   $t_d$  is found to be larger for \CuCN\  than for \HgCl~\cite{Drichko2014}.
The existing experimental data do not give a straightforward evidence of tuning from a dipole liquid to a dipole solid by  hydrostatic and chemical pressure for \HgBr\ and \HgCl. While a charge ordered state   in \HgCl\ is  suppressed by  external pressure of about 1~kbar~\cite{Lohle2016pressure}, the unit cell of \HgBr\ is somewhat larger than that of \HgCl. Calculation of electronic structure of  these materials and its change with pressure, as well as further explorations of magnetic properties are necessary for further understanding of the phase diagram.



%A phase transition between dipole solid and quantum dipole liquid state is a playground for studies of quantum criticality. The absence of   order in \HgBr\ close to the quantum dipole solids suggest that it is in the vicinity of a quantum  phase transition.  A softening of the  collective mode which we observe at 11~K is the realization of the temperature/frequency scaling which is characteristic of quantum critical regime.   While our present work is aimed on understanding of quantum dipole solid and  liquid states, the next step would be to investigate in detail a presence of quantum critical regime and critical scaling.



\section{Conclusions}

In conclusion, we demonstrated a quantum dipole liquid state in \HgBr, and found the frequency of dipole fluctuations at around 40~\cm\ both using vibrational spectroscopy and observing a collective mode associated with the fluctuations. While the frequency of the collective mode is similar to that of dipole fluctuations, a mixed charge-spin excitations are also a possible explanation.   A presence of a linear term in heat capacity down to 100~$\mu$K suggests a presence of spinon excitations in \HgBr.  Magnetic Raman scattering in \CuCN\ suggests that this material can be well understood within dimer Mott insulator model and does not show the fingerprints of quantum dipole liquid.


%
%\begin{addendum}
\begin{comment}
{\bf Acknowledgements}
The work at IQM was supported by the
U.S. Department of Energy, Office of Basic Energy
Sciences, Division of Material Sciences and Engineering
under Grant No. DE-FG02-08ER46544.
The work in Chernogolovka was supported by FASO Russia, state task \#0089-2014-0036.

 The authors declare that they have no
competing financial interests.

 Correspondence and requests for materials
should be addressed to Natalia Drichko~(email: drichko@jhu.edu).
\end{comment}
%\end{addendum}

%\bibliography{OrganicConductors}

%\newpage













%\clearpage

%\setcounter{page}{1}


%\end{methods}



%\section{REMOVE}

%\subsubsection{2. Crystal structure (move to materials section)}






%\subsubsection{4. Heat capacity (already ecplained in HC chapter}

%In Fig.\ref{} we compare our results on temperature dependence of  heat capacity for \HgBr\ and \HgCl\ with that received by us for \CuCN. This plot demonstrates the high quality of the data obtained by us, where the \CuCN\ data coincide with the literature. Within the precision of the measurements we performed, $\beta$ of \CuCN\ is similar to the published, and to that of \HgBr, which can be explained by a similar structure and thus phonon DOS of the compounds. Interestingly through, values of $\gamma$ coefficient in heat capacity of   \HgBr\ and \CuCN\ are similar, too.  \cite{Yamashita2008}.

%\bibliography{OrganicConductors}

%\clearpage


\clearpage

\makeatletter
\let\@currsize\normalsize
\makeatother
\ssp

\begin{longtable}{cccccc}
\caption{List of the parameters used in the formula Eq. 1 SI for the calculations of the line shapes presented in Fig.\ref{fig:nu}b of the paper.}\\
\hline
$\omega_{EX}$, \cm & $\omega$, \cm & $\Delta$, \cm & $\Gamma$, \cm & $a_p$ & $a_R$\\ 
\hline
\endfirsthead
\multicolumn{2}{@{}l}{\textbf{Table \thetable} \ldots continued} \\
\hline
$\omega_{EX}$, \cm & $\omega$, \cm & $\Delta$, \cm & $\Gamma$, \cm & $a_p$ & $a_R$\\ 
\hline
\endhead
0  & 1490 & 15.5 & 6 & 2.05 & 3.56\\
1  & 1490  & 15.5 & 6 & 2.05 & 3.56\\
5  & 1490  & 15.5 & 6 & 2.05 & 3.56\\
15 & 1490  & 15.5  & 6 & 2.05 & 3.56\\
30 & 1490  & 15.5 & 6 & 2.05 & 3.56\\
40 & 1490  & 15.5  & 7 & 2.05 & 3.56\\
\hline
\end{longtable}


\dsp
\clearpage












