I did notice the overlap of senses in the stories. A couple examples I caught:
On page 3, he compares the taste of the food to the high notes, modulations, chords and dissonances of music.
When his wife is discussing cannibalism with the tour guide, the guide says that dish was "a terrible harmony."
Watching his wife eat, the narrator imagines the sensation of his wife tearing into his flesh and chewing.
These references are an acknowledgment, I think, that life can't be easily divided into the individual experiences of the senses. Its a cute idea to say that this story is about taste or hearing, but we interact with the world in a much more multifaceted way.
I agree with Blair that the second person narration doesn't hold up. The only other story that I have read in second person was by Junot Diaz (who I am obligated to now mention in every post). The best aspect of Diaz's writing is his personal, informal voice and it allowed for a rhythm similar to that of someone talking with you. Calvino could have replaced the "you"'s with "he"'s just fine. Diaz's story was also much shorter, whereas in the Calvino story the gimmick got old after awhile.
The two missing senses might be the most difficult to imagine in this context. 99 percent of stories are "sight" stories and focus on relaying visual images. I'd imagine touch being a story in the vein of "Johnny Got his Gun" (or "One" by Metallica). The protagonist would have to be blind, mute, unable to smell and deaf -- existing only as a consciousness and kept alive through the sensation of physical contact. Something about touch is different than the other senses and I think its because the loss of touch is paralysis. Its the most basic way we interact with the world and losing it means either paralysis or death, but its also overshadowed by all the other senses.
Or maybe not. Maybe a story about touch could the simplest of all five.