That's actually a really good question, and uncovers lots of interesting and subtle things...
First, it might
be possible to correctly trigger on the two sine waves, but that depends on your 'scope. I'll get on to that in a second. But, before we do, we have a problem:
You have two sine waves, but not locked to each other. They also might not be exactly
the same frequency. And you're trying to measure the phase difference. Unless they genuinely are the same frequency, the phase shift will be constantly changing. If you do contrive to display both on your 'scope, stably triggered, then you'll appear to have two sine waves that are in phase, but different (similar) frequencies. Clearly impossible!
Next, using X-Y mode to measure phase isn't the best way to do it. In fact, I rarely recommend X-Y mode at all. Yes, I know that every text book shows Lissajous curves, but apart from being very pretty and vaguely sci-fi, there is usually a better way to do it. I only use X-Y mode for setting up IFs. Oh, and at the risk of stating the obvious, the timebase is disabled in X-Y mode, so no triggering possible.
So, regarding the triggering, triggering on two separate signals isn't always possible - it depends on the topology of the scope. First, you have to be in Alt mode - where it draws channel 1 on one sweep, and then channel 2 on the other. The other option is "Chop", where the switch jumps between the two traces at 100kHz or more.
Having got Alt set up, you need to change the triggering to Alt (as opposed to Ch1 or Ch2). This means that the trigger signal is "sniffed" after the beam switch. Now, not all 'scope offer this option, and some that do aren't very good at working in this mode. On some Philips 'scopes, this mode is called "Comp" IIRC.
That clearly refers to analogue 'scopes. Whether your DSO offers the option depends entirely on luck. After all, there is no beam switching in a DSO, so no obvious place to pick up the trigger signal. As I said above, it's a really good question - when I get a chance, I'll look to see if it's possible on the Rigol...
For info, here is a better way to measure phase difference:
You know the variable timebase speed control? The one that has to be in the "Cal" position so that the numbers around the timebase switch (e.g., 1us/div) are accurate and correct. As soon as you adjust the VAR control, those numbers are no longer true. As a result, you've probably wondered why they provide the VAR option at all...
Here's where it's useful. Feed in the first sine wave, trigger from that channel only, and adjust the timebase speed so that one complete cycle occupies 8 divisions. You'll need to turn the switch clockwise until one complete cycle shows as more than 8, and then turn the VAR control anti-clockwise until you've hit 8.
Why 8 divisions? Because now, you've got 45 degrees per division. Voila
Then you enable the other sinewave, and by looking at where the zero-crossings are, relative to each other, you can measure the phase difference. Remember to turn VAR back to the cal position afterwards...
As mentioned above, your two sine waves are not phase-locked to each other; they are free-running. So, you'll see one sine wave is stationary, and the other will "run through" like a hum bar. Not much you can do about that. If you set up the scope to trigger alternately on each, as described above, then they will appear to be in phase, but at different frequencies - clearly impossible!
That definitely works on an analogue 'scope, and should also work on a DSO. There should be the option to fine-adjust the timebase speed, and happily, it should actually be calibrated (though not in the convenient 1-2-5 sequence).
Sorry for the long post, but it's one of those "simple question, complicated answer" questions. Hope it makes sense - for understanding purposes, it might help to dig out the analogue scope for a bit.