What does Scout mean by the "starched wall of a pink penitentiary" in To Kill a Mockingbird?

By chapter 14 of To Kill a Mockingbird by Harper Lee, Aunt Alexandra has arrived at the Finches' home, and for Scout more than anyone else this is a disaster. Atticus is glad that she is here, since he is going to be busy with the trial and wants her there for the children; and she is his sister, after all. Calpurnia is not thrilled, but she does not complain and the two women manage to work together. Jem is a few years older than Scout, so he is able to kind of ignore his aunt--and of course he is a boy, so the two of them do not have too many clashes.

For Scout, however, Aunt Alexandria is a major road block and speed bump on the metaphorical road of her life. Everything that Scout is just happens to be everything Aunt Alexandra thinks a young lady should not be. This includes all kinds of things, but one of them is a real point of contention between the two females. Scout is a tomboy and she wears overalls whenever she has a choice. They suit her life as well her temperament, and wearing anything else is an almost monumental sacrifice for Scout. 

Aunt Alexandra, on the other hand, thinks she is doing everyone a favor by trying to change Scout from a tomboy into a model young lady. The easiest thing for her to change is what Scout wears, so that is one of Aunt Alexandria's primary targets. In this chapter, Scout overhears a conversation between her father and Aunt Alexandra about the subject of Scout's distinctively unladylike behaviors, though they never mention Scout's name. While Scout is not certain they are talking about her, we (the readers) are. 

Scout comments that she feels

the starched walls of a pink cotton penitentiary closing in on [her].

What this means, of course, is that she can already feel the restrictions on her freedom to be a tomboy that Aunt Alexandra is about to impose on her. The starch in this comment refers to prim and "girly" clothes which have been pressed (ironed) and starched (made stiff), and of course Scout would see this kind of clothing unappealing both because it does not suit her but also because it is uncomfortable. The "pink cotton" is a specific reference to frilly, girly, distinctly non-tomboyish clothing which, again, Scout abhors. 

The rest of the phrase is a metaphor in which Scout is comparing the potential of having to wear such things to a prison, and she already feels the restrictive walls of Aunt Alexandra's prescribed "proper young lady" wardrobe closing in on her. In short, she feels as if she is a prisoner, about to be smothered (or pressed to death) by the clothes she fears her aunt is about to make her wear. 

What does the sentence "I felt the starched walls of a pink cotton penitentiary closing in on me" mean in To Kill a Mockingbird?

In order to fully appreciate this quote, it's first important to understand some of the context which is lost in our modern world of wash-and-wear clothing.

When Scout was growing up, it would have been common for most clothing to be ironed. This would have been a hot and tiring process, but it was considered respectable to wear clothing that was as wrinkle-free as possible. In order to make clothes even stiffer (and therefore even more resistant to wrinkles), starch was often added during the ironing process. People who wanted to conform to societal expectations of "proper" dress were therefore willing to sacrifice a great deal of comfort.

Aunt Alexandra is one of these people. Viewing herself as the matriarch of the Finch family, she wants young Scout to conform to the expected ideals of "ladylike" behavior—and this means that Scout also needs to dress the part. Scout, meanwhile, values the ability to move and play in comfort and therefore wears overalls.

This difference of opinion in "proper" behavior creates conflict between Scout and Aunt Alexandra. When she hears her aunt and Atticus talking in the next room, Scout assumes (and with good reason) that they are once again talking about Scout's need to conform to the expected ideals of ladylike behavior. Scout comments that she feels "the starched walls of a pink cotton penitentiary closing in" on her.

"Pink cotton" represents all of the ways Aunt Alexandra wants Scout to conform and demonstrate that she is a girl from a "fine" family with privileged social standing. These expectations feel "stiff" to Scout, who prefers adventure and scoffs at conformity. Aunt Alexandra's expectations feel like a "penitentiary," or prison, to Scout, restricting her freedoms and stifling her own independence. The "pink" and "starched" clothing represents the ideals which Aunt Alexandra holds for Scout and which Scout herself rejects.