I had a similar experience the first couple of years that I did my lights -- contacted papers, sent videos to local news, and only got fairly light visitation and no media interest (but those that did, stayed for a 20 minutes before leaving). I kind of got frustrated that it was so hard to get visibility (and that was back when computerized lighting was fairly unusual).
Then one year a local TV station contacted me and came out to do interviews and live segments on the 5, 6, and 10 o'clock news. Before the 5 pm news was even over cars started showing up. That year I had way too many cars, gridlock at time in my culdesac, and even though my neighbors were positive I knew that things were getting out of hand for my own taste. I also had to shorten the show significantly to turn over cars. I had finally gotten what I thought I wanted and didn't like the resulting hassles...
After taking the last two seasons off, I'm back live this year and traffic so far is fairly light. I got contacted by the local paper who was interested in doing a story and I actually declined, saying that I wasn't actively seeking publicity and prefered to have the display for friends, neighbors, and those who stumble upon it by driving by or from word of mouth.
I'd prefer a smaller number of people to have a really nice experience watching the full show than a larger crowd watching a shorter show and inconveniencing my very tolerant neighbors.
Depending on where you live and what your streets/neighbors are like, one of the ironies of display popularity is that it can ultimately make you limit or even shutdown your display...
I think Tim is right though, it is natural to see a continued year by year increase of traffic as word of mouth spreads and families include visiting your display into their own holiday traditions, etc.
Great job on your display!