This is an excellent question and one that I have been asked many times.
It is often assumed that today we are saved by accepting Jesus as our Savior and that believers in Old Testament times were saved by their obedience to the Law of Moses. But that is not at all true. The argument of the Apostle Paul throughout Romans is that the Law was never meant as a means to save anyone, and he makes it abundantly clear that "...no one will be declared righteous in his sight by observing the law..." Romans 3:20 (NIV)
So, people in Old Testament times died placing their faith in things hoped for and things not yet seen.
As for where they went after death, Jesus made it abundantly clear that prior to His death on the cross which opened the way to heaven, no one who had died before that time had ever gone there.
No one has ascended into heaven except he who descended from heaven, the Son of Man. John 3:13 (ESV)
So, if they didn't end up in heaven, where did they go? Jesus revealed the answer to this question in a story that he told which is recounted in Luke's Gospel account. As you read the verses below, take special note of the descriptions given about the place where people went after death.
“There was a rich man who was clothed in purple and fine linen and who feasted sumptuously every day. And at his gate was laid a poor man named Lazarus, covered with sores, who desired to be fed with what fell from the rich man’s table. Moreover, even the dogs came and licked his sores. The poor man died and was carried by the angels to Abraham’s side. The rich man also died and was buried, and in Hades, being in torment, he lifted up his eyes and saw Abraham far off and Lazarus at his side. And he called out, ‘Father Abraham, have mercy on me, and send Lazarus to dip the end of his finger in water and cool my tongue, for I am in anguish in this flame.’ But Abraham said, ‘Child, remember that you in your lifetime received your good things, and Lazarus in like manner bad things; but now he is comforted here, and you are in anguish. And besides all this, between us and you a great chasm has been fixed, in order that those who would pass from here to you may not be able, and none may cross from there to us.’ And he said, ‘Then I beg you, father, to send him to my father’s house—for I have five brothers—so that he may warn them, lest they also come into this place of torment.’ But Abraham said, ‘They have Moses and the Prophets; let them hear them.’ And he said, ‘No, father Abraham, but if someone goes to them from the dead, they will repent.’ He said to him, ‘If they do not hear Moses and the Prophets, neither will they be convinced if someone should rise from the dead.’” Luke 16:19-31 (ESV)
It's important to note that this story is not a parable. This is a true account concerning real people and real places. (Notice that Jesus used a proper name in these verses, which was never done when merely relating a parable.)
When Lazarus (no relation to the man of the same name whom Jesus raised from the dead) died he was taken to a place which Jesus referred to as "Abraham's side" (the Greek word is literally translated bosom.) This is certainly not heaven, but a kind of holding place of comfort where those who died in faith awaited entrance into heaven.
The rich man in the story also died and Jesus said that man was sent to a "place of torment" called Hades (which literally means the grave or hell). The incredible thing is that these places were within some kind of visual range from one another. It was even possible for the inhabitants of each place to converse, even though moving from one place to the other was impossible.
I believe this story gives us a unique glimpse into the place where those who died prior to the cross of Christ awaited the opening of heaven. Now that Jesus has paid the full price for our sin, all who have placed their faith in His finished work on the cross are immediately ushered into the presence of God upon death. The words that Jesus spoke to the penitent thief on the cross are now ones we can all confidently embrace: “Truly, I say to you, today you will be with me in Paradise.” Luke 23:43 (ESV)