I unfortunately cannot help specifically with regards to DXTory, but I can at least attempt to provide an alternative: have you tried using Open Broadcaster Software (OBS) before?
I'm not sure entirely what the purpose of you using DXTory is, but you can use OBS to not only stream, but also record locally. The settings are highly customizable as well. Additionally, if the purpose of having multiple sound sources is so that you can record yourself on a mic and output the sound of a game/program at the same time, OBS can facilitate that.
Regardless of that suggestion, we can still look into why you might be having some issues
Let's delve a bit more into your recording situation: what exactly are you trying to record? What is the resolution of the game that you are trying to record, and what are the various settings of the game that you are trying to record? Given that your CPU is a AMD FX 6300, your CPU benchmarks respectably well
and you claim it runs well in FRAPS. There might be another underlying issue here.