### 3.1 Resonant frequency—light shift

In a first step we analyze the angular dependence of the magnetic resonance center frequencies as plotted in Fig. 3. We added to the figure also the mean value of the measured center frequencies found for the two different circular polarizations.

In general, the magnetic resonances are strongly shifted by the ac-Stark shift due to the strong off-resonant pumping. The angle between the laser beam direction, represented by its *k*-vector, and the magnetic field \(\vec{B}_{0}\) influences to the atom-light interaction by the transition dipole moment [6]

$$ \vec{D}\cdot \vec{E} = \frac{E_{0}}{2\sqrt{2}} e^{i ( \vec{k} \vec{r}-2\pi \nu _{L} t )} \bigl( [ \cos \alpha \pm 1 ] D_{+} + [ \cos \alpha \mp 1 ] D_{-} +2 \sin \alpha D_{z} \bigr). $$

(13)

Therein \(E_{0}\) is the amplitude of the electric field, *k⃗* the k-vector of the laser, *r⃗* the atoms position, \(\nu _{L}\) the frequency of the laser beam, and the \(D_{i}\) are components of the dipole operator. In words, above equation states that with modifying the angle *α* the weight of pumping to different excited states is strongly influenced. Namely, \(D_{+}\), \(D_{-}\), and \(D_{z}\) components couple to excited states with increased, decreased, as well as unchanged magnetic quantum number \(m_{F}^{\prime }= m_{F} + \{ +1,-1,0 \} \), respectively.

The strongest characteristic results from the vector light shift that can be interpreted as virtual magnetic field added in direction of the lights angular momentum [24]. It is strongest close to angles of zero and ±*π* and reduces to zero close to perpendicular orientation. Also as expected, it changes its sign for a change in the laser’s helicity. A more detailed discussion of the light shift can be found in Ref. [6]. We use the findings therein to calculate the expected light-shifted transition frequencies between Zeeman levels with highest and lowest magnetic quantum numbers to their neighboring states. These correspond to the respective blue and orange solid lines plotted in Fig. 3 and can be identified with the transitions probed when pumping to the dark states \(m_{F} = \pm 4\), where the plus and minus sign are to be used for \(\sigma _{+}\) and \(\sigma _{-}\) light, respectively. For the curves we used a magnetic field strength of \(B_{0} = 49.664~\mu \mbox{T}\), detunings of \(\delta _{F^{\prime }= 4} = -8\text{ GHz}\) and \(\delta _{F^{\prime }= 3} = -9.2\text{ GHz}\) from the respective optical transitions \(F=4 \rightarrow F ^{\prime }= 4\) and \(F=4 \rightarrow F^{\prime }= 3\), a linewidth of the optical transitions \(\varGamma _{\mathrm{opt}} \approx 4\text{ GHz}\), and optical driving amplitudes in frequency units \(\varOmega _{L} = 3.15\text{ MHz}\) and 3.45 MHz for \(\sigma _{+}\) and \(\sigma _{-}\) polarized beam, respectively. The last values corresponds to the on-resonance Rabi frequencies [28] introduced by the pumping beams.

We achieve a good correspondence between our model and the experimental results. It is best close to angles of zero and ±*π*, where additionally to the strong light shift also a very good pumping to the dark states is achieved. Close to angles of \(\pm \pi /2\), not only the observed light shift is reduced but also the population is distributed between several ground state levels. This effect reduces the magnetic resonance amplitude and, additionally, enables probing more ground state transitions both reducing the agreement between theory and experiment.

Nevertheless the model allows the reconstruction of the laser intensities \(I_{L}\) from

$$ \varOmega _{L} = \frac{1}{h} \sqrt{\frac{I_{L}}{4cn\epsilon _{0}}} \bigl\langle J=1/2 \bigl\Vert \vec{D} \bigr\Vert J^{\prime }= 1/2 \bigr\rangle . $$

(14)

Here the vacuum permittivity \(\epsilon _{0}\), speed of light *c*, reduced transition dipole moment \(\langle J=1/2 \Vert \vec{D} \Vert J^{\prime }= 1/2 \rangle \) [29], as well as the refractive index *n* enter to the equation. We found the two intensities to be 64 and 76 W/m^{2}. Since their ratio does not correspond to the scaling factor *a*, we expect some influence of slightly varying beam profiles or non-perfect circular polarization. If we assume the laser power distributed equally to a circular beam with a diameter of 4 mm we find a total value of about 1 mW which fits well to separate measurements.

### 3.2 The dc-photocurrent

A second basic characteristic is found for the dc-photocurrent as introduced in Eq. (1). It is plotted as a function of the heading angle in Fig. 4.

As demonstrated in the figure, we observe a dependence that roughly follows a \(\vert \cos \alpha \vert \) or \(\vert \cos \alpha \vert ^{2}\) function. Furthermore, the orientation of the \(B_{1}\) coils has no influence to the photocurrent away from the magnetic resonance, as expected. Although we used an electronic balancing, we still observe a slightly higher dc-photocurrent for the \(\sigma _{-}\) beams compared to the \(\sigma _{+}\) helicity of about 2% in maximum that is best demonstrated from the difference signal.

The explanation of the shape of the measurement curves is found in the interaction of the dipole moment *D⃗* with the laser’s electric field *E⃗* as given in Eq. (13). We included this effect by modified relaxation and excitation rates depending on the effective circularly polarized laser intensity (6). The measured photocurrent is increased when the pumping of the atoms to dark states is more efficient because they cannot absorb anymore light. A large dark state population is related to large differences in the ratio of light pumping to larger and smaller magnetic quantum numbers. They are respectively proportional to \(D_{+} = D_{x} + i D_{y}\) and \(D_{-} = D_{x} - i D_{y}\).

Our model (Eq. (8)) predicts a change from a cos^{2}*α* dependence to one proportional to \(\vert \cos \alpha \vert \) when the effective pumping amplitude \(\varOmega _{L}\) is increased compared to the relaxation of the polarization given by a rate \(\gamma _{r}\). That allows for a more accurate fit of the photocurrent as a function of the orientation angle *α* as demonstrated by the solid lines in Fig. 4. For this curve, we estimate the ratio of relaxation to optical pumping rate to be \(p_{1} \approx 0.3\) and the amplitudes \(I_{0}=0.49~\mbox{mA}\). Keeping in mind the electronic amplification of the \(\sigma _{+}\) channel, the variable \(p_{1}\) is multiplied by \(1/1.6\) for this channel making it necessary also to adjust the amplitude \(I_{0}\) when fitting the curve measured for this helicity to 0.56 mA. Additionally, in consistency with the experiment, our model predicts a smaller photocurrent with decreasing the optical pumping amplitude \(\propto \varOmega _{p}\) that usually is not included in Bloch equations.

### 3.3 Magnetic resonance amplitude

In contrast to the very similar angular dependencies of the dc-photocurrent for the two different \(B_{1}\) coils, a clear discrepancy is found in the normalized amplitude of the magnetic resonance signal \(i_{0} = I_{0}/I_{dc}\), as demonstrated in Fig. 5.

This mentioned discrepancy is well explained by the additional modification of the effective \(B_{1}\) field amplitude when modifying its direction compared to \(\vec{B}_{0}\). We included this in our calculation by the factor *c* that is constantly 1, when using the *y*-coil, and \(\vert \cos \alpha \vert \), when the *x*-coil is used. With Eq. (5) we fit the normalized amplitudes presented in Fig. 5 and found very good agreement between experiment and theory with a factor \(p_{2} = 0.12\). Note, for \(\sigma _{+}\) we additionally had to adjust the value of \(p_{2}\) by a factor of 1.6. The dependence we observed for the normalized resonance amplitude driven by the x-coil is strongly influenced by the cos function given for the effective \(B_{1}\)-field amplitude *Ω*. Namely, the resonance height is maximal at \(\alpha = 0\) and *π* and tends to zero in vicinity of \(\alpha = \pm \pi /2\), where effectively no \(B_{1}\) field remains.

In contrast, when using the *y*-coil, our theory predicts an increase in resonance amplitude towards one for angles close to \(\alpha = \pm \pi /2\). Except in close vicinity of these angles this general tendency is also found in the experiment. From our model, it is clear that this increase is connected to a reduction of the optical pumping to the dark state: A given strength of the \(B_{1}\)-field corresponds to a certain rate of population shifting back from the dark state to absorbing states. In other words, the \(B_{1}\) field introduces a Rabi oscillation whose frequency depends on the field amplitude. Thus for larger powers, the shifting of population to the absorbing state is faster. If this rate is smaller than the possible population transfer that can be achieved by the pumping laser, the measured photocurrent in resonance does not reach down to zero. Therefore the amplitude \(I_{0} / I_{dc}\) is smaller than one. When rotating towards \(\pm \pi /2\) and using the *y*-coil, the effective \(B_{1}\) field amplitude stays constant. Because at the same time the optical pumping to the dark state gets less effective, the photocurrent amplitude is increased towards one. That means that all the atoms that are optically pumped contribute to the resonance with the \(B_{1}\) field. Still, our model does not accurately reproduce the extracted values in close vicinity of \(\alpha = \pm \pi /2\), where the experimentally observed amplitudes drop to zero. This deviation probably results from not considering the linearly polarized pumping at these angles, that not only leads to optical alignment but also takes the role of the \(B_{1}\) field in redistributing population.

### 3.4 Resonance width

A fitting of the experimentally observed resonance widths with the same parameters \(p_{1}\) and \(p_{2}\) turned out to be unsuccessful. Thus we found it necessary to introduce an additional fitting parameter \(p_{3}\) to Eq. (11). Theoretical curves with parameters \(p_{3} = 3.5\) and \(\varGamma _{\varphi }= 350\text{ Hz}\) are plotted together with the experimental data points in Fig. 6.

Our adjusted model again fits nicely to the experimental results in the case of the *x*-coil as demonstrated by the solid lines’ correspondence to the data presented as diamonds in Fig. 6. We observe a reduction of the power broadening introduced by the rf-field. Therefore, we assume that close to angles of \(\alpha = \pm \pi /2\) the minimal possible magnetic resonance width is achieved for this certain temperature and laser power.

In contrast, applying a constant effective magnetic rf-field by the *y*-coil, leads to a strong increase of the resonance width in the experiment. We account this to a large redistribution of population between Zeeman states resulting in an overlay of several ground state transitions and the linear optical pumping remaining at these angles. Since our model is restricted to two-levels and we neglected the linear pumping, we fail to catch the magnitude of this increase in magnetic resonance width. Still, we note that the qualitative behavior is accurately reproduced by the theoretical model.

As already mentioned above, we account the factor \(p_{3}\) to a power broadening due to the strong laser power. To justify this assumption we can estimate the expected laser power broadening similar to Eq. (12) by (compare e.g. Ref. [30])

$$ \Delta \nu _{\mathrm{laser}} = \frac{\varOmega _{L}^{2}}{2\varGamma _{\varphi }\varGamma _{\mathrm{opt}}} = 3.5, $$

(15)

where we used the values for \(\varOmega _{L}\) of the \(\sigma _{+}\) beam and \(\varGamma _{\mathrm{opt}}\) as noted above. Since this value is in agreement with the fitting parameter, we identify the strong power of the detuned laser as one important source of broadening of the magnetic resonance.

### 3.5 Shot-noise-limited sensitivity

Finally, as a key parameter of the proposed magnetometer we extracted the shot-noise limited sensitivity of the LSD-Mz signal as explained in Sect. 2.1. The respective experimental values as a function of the angle *α* for the two rf-coil-orientations are presented in Fig. 7. Our experiment shows a minimal shot-noise limited resolution for the used parameters of \(20~\mbox{fT/}\sqrt{\text{Hz}}\) already close to the optimal conditions [21]. This optimal resolution is found in the case of parallel alignment of the laser direction to the magnetic field. This results from the strong dependence of the sensors performance on the pumping to the dark state and achievement of a reasonable light shift. Both of these requirements are connected to the atom-light interaction and thus are reduced towards perpendicular orientation. There, both the magnetic resonance signal as well as the light shift are reduced resulting in a loss of sensitivity.

We observe a strong dependence of the amplitude and width of the magnetic resonance signals on the angle that additionally is slightly different for the two used orientations of the \(B_{1}\) coils. Still, because the width of the magnetic resonance as well as the light shift are both dominated by the laser intensity, the shot noise limited resolution is quite stable for angles of ±20^{∘} around optimal orientation.

As shown above, our model allows for a description of the orientation dependence of all the key parameters of the magnetic resonance. Thus it enables us to estimate the shot noise limited sensitivity as a function of *α* in the case of perfect balancing of the channels. To do so, we calculate the steepness of a single resonance curve as the derivative of (1)

$$ \frac{s}{2} = \biggl\vert \frac{dI}{d\nu } \biggr\vert = \frac{8 I_{0} \Delta \nu ^{2} (\nu -\nu _{0})}{ ( 4 (\nu - \nu _{0} )^{2} + \Delta \nu ^{2} )^{2}} $$

(16)

and evaluate it at \(\nu = \nu _{0} + \nu _{LS}\). There the maximal steepness of the LSD-Mz signal is achieved and corresponds to twice the value of a single magnetic resonance \(s(\nu _{0} + \nu _{LS}) = 2 \vert \frac{dI}{d \nu } \vert (\nu _{0} + \nu _{LS})\). Substituting into the equation for the shot-noise-limited resolution (2) results in

$$ B_{sn} = \sqrt{\frac{e}{I_{dc}}}\frac{1}{8\gamma i_{0} } \frac{ ( 4\nu _{LS}^{2} + \Delta \nu ^{2} )^{2}}{\Delta \nu ^{2} \nu _{LS}}, $$

(17)

with the parameters \(I_{dc}\), \(i_{0}\), and Δ*ν* defined by the respective equations (8), (10), and (11). The theoretical estimated sensitivity is added to Fig. 7. In the calculation we used the experimental parameters of the starting angle and for \(\sigma _{+}\) light and assumed perfect channel balancing. In general a slightly better sensitivity is expected from our calculation but the qualitative shape fits very well to the experiment. Note, the two additional peaks around angles of \(\pm \pi /2\) result from the overlay of the two resonance curves for the \(\sigma _{\pm }\) beams meaning \(\omega _{LS} =0\), as can be seen in the plotted resonance frequencies of Fig. 3. This overlay results in a difference signal constantly equal to zero and thus \(s\rightarrow 0\). Still, due to the lack of contrast at perpendicular configuration, this effect is not visible in the experimental data. Finally, we note that although a perfect balancing has only little influence to the sensitivity, it makes the sensor more robust against heading error, concerning the reconstructed absolute magnetic field, see green data points of Fig. 3.