Too much makeup can definitely irritate skin and make it more oily, but you say you only wear a small amount of powder, so I'd say the makeup probably isn't doing much harm. It could be, though, so maybe try not wearing it for a couple days and see what happens. I have oily skin too and I don't wear makeup because I'm afraid of it making it worse, but I've found that just making sure to wash your face a lot helps. If your skin's getting really oily, try washing more than twice a day. I've heard that sometimes you aren't supposed to do this because it might dry your skin out, but so far that hasn't happened to me! Just letting the oil stay can make it worse, I'm pretty sure. Also maybe just rinse your face with water a little throughout the day.
& on WebMD I found this:
"Avoid oil-based cosmetics. How can you tell? Follow this simple guideline: Creamy foundation or blush generally can clog pores. Bare mineral cosmetics, which are light and powdery, are less likely to do so."