I have the same issue with ShadowPlay. Nothing, nada, zip, doesn't matter which key I press or assign to it. Sadly, I *think* I've found the answer...ShadowPlay requires a newer Nvidia GPU driver than I'm running in order to work, and (right, let's jinx it) as my FSX currently looks OK I feel disinclined to fiddle with my Nvidia drivers; if it ain't broke an' all that. I'm still running in DX9, BTW.
If it's any help, my way around the issue has been to use Bandicam...much easier to set up than Fraps, has a DX9 option, and if set to low priority doesn't seem to cause stuttering (much) during playback and recording. It also gives me 10 minutes' recording in the unregistered version vs. Fraps' 30 seconds.
This is what I did with it (first attempt ever):
https://youtu.be/CDybVsU_BzsHope that helps.
- Sam