I may not be able to answer these questions satisfactorily for the non-Christian, because understanding God means having faith even while knowing that I will never understand God fully. I am however, going to attempt to draw on some scripture and its insight, in order to answer some of these questions in my own mind, and in so doing, hopefully help some others out with their questions.
The first scripture that comes to my mind, always, in times of turmoil is John 16:33 in which Jesus says "I have told you these things, so that in me you may have peace. In this world you will have trouble. But take heart! I have overcome the world." This lets me know that things will not always be easy in this life; trouble will inevitably come to me. It gives me peace to know that my Savior has overcome this world, and that by trusting in him, I will be victorious along side him one day.
Simply knowing that Jesus warned me there would be trouble, doesn't always help me with immediate problems, though. Sometimes I need help just making it through a struggle. Jesus says in Matthew 5:4 that those who mourn will be comforted. There are times when my prayers aren't answered as I wanted and I just need to be comforted. How would I know how to comfort others, as I am called to do in Romans 12:15, if I've never know what it is like to mourn? Jesus felt feelings of loss and sadness too, as he did when Lazarus died and he wept for his friend. If God allowed Jesus to experience this pain, it must have been for a reason.
I know one thing for sure and that is that God does not punish his children because they didn't pray hard enough. Or yet, because they did something to deserve cancer, or homelessness, or poverty, or depression, or any number of things that cause us pain in our lives. If that were the case, then we would all be living in poverty and dying of cancer because there is no one who deserves God's blessings more than the next person. We are told in Romans 3:21-26 that everyone has missed the mark (sinned). We all deserve the bad things, not the good.
So, why pray at all if God is just going to allow whatever is going to happen, happen? I pray because I am called to. In Romans 12:11-13 I am told, to be joyful in hope, patient in affliction, faithful in prayer. This is one way I demonstrate my obedience, and faith, to God. I know I am undeserving of his grace, yet I'm hopeful that he will have mercy on me.
Alas, some things that I pray to be relieved of will not be removed from my life this side of heaven. This is where the faith comes in. I have to trust that God sees around the corners that I am blind to, since his ways are higher than my ways (Isaiah 55:7-9).
I can't say I wasn't expecting trouble, because Jesus did warn of it. I must accept it as a lesson in humility, and benevolence, because I want to be able to comfort those who need my comfort when the time comes, just as Jesus taught me. I consider it pure joy. I may not be happy about it, but I consider it pure joy.
Consider it pure joy, my brothers, whenever you face trials of many kinds, because you know that the testing of your faith develops perseverance. Perseverance must finish its work so that you may be mature and complete, not lacking anything. If any of you lacks wisdom, he should ask God, who gives generously to all without finding fault, and it will be given to him.
11/14/14: I add this prayer for a friend today.
Dear Lord, I don't pretend to understand your ways. Nor do I have a frame of reference to dig from my own past that can offer words of comfort for [this family] today. I pray, God, that you will send someone along beside them who has walked in their shoes. In Jesus' name I pray, believing that you will hear my prayer this day.