You're getting sparrows because you're putting out sparrow feed. The mixed seed that is sold is primarily milo and other low cost fillers. Sparrows love it. I use a sunflower feeder and go through ~$66 of oiler sunflower seed a year (3 - 50 lb bags from the feed store). I don't get any sparrows, but I do have Cardinals, Nuthatches, Chickadees, Tufted Titmouses, House Finches, Blue Jays, Yellow Shafted Flickers, six kinds of woodpeckers and occasional mourning doves. Although there are numerous sparrows and pigeons around, none come to the feeder.
I keep the squirrels away by using a piece of 5" stove pipe at the base of the feeder. Its pretty funny to see them shimmy up the 3/4" galvanized pipe, only to slide back down when they try to grab the baffle.
I have a separate thistle feeder that will have 6-8 Goldfinches at a time during the summer.
We have a lot fewer birds this year, compared to last year. The experts say that West Nile Virus made a real dent in the songbird population.