Which output should be used will matter mostly on the quality of the DACs (Digital-to-Analog-Converters) and the analog signal path in the preamplifier or processor that the Rotel CD player is being connected to. Rotel CD players utilize high quality Burr-Brown or Wolfson DACs coupled with a minimalist circuit design for the analog stage, carefully selecting quality capacitors and rectifiers and providing a solid power supply to provide the best sound quality possible from our CD players, and in many cases the DACs in the CD player will clearly exceed the sound quality of the onboard DACs from many processors.  In these cases, the analog outputs of the CD player would be the best logical choice. However, in some cases you may find in a high quality processor or outboard DAC that the DACs used may sound better than the CD players internal DACs, which would mean using the coaxial digital output.

If possible, connect both to your system and compare the two outputs to determine which sounds best in your system.