I had a really similar reaction! "Story of Your Life" mainly got a "Huh" reaction from me the first time I read it; on a reread, it had slightly more emotional impact, but not that much. Although I like his stories for the thinkiness and the cool ideas, I don't think I love them, and I had problems with the depiction of disability in "Hell Is the Absence of God" and the story on lookism also just did not quite sit right with me.