Social insects are characterized by extraordinary social behaviors and thrive in colonies made up of phenotypically distinct castes. However, because each caste is governed by the same genetic architecture, caste phenotypes may not be able to evolve independently and reach their caste-specific optimum. We studied the genetic architecture underlying phenotypic variation in the social, eastern yellow jacket wasp, Vespula maculifrons, to understand if genetic constraints inhibit phenotypic evolution in social insects. To explore this question, we collected ten V. maculifrons colonies and measured several important morphological traits in workers and prereproductive queens (i.e., gynes). We genotyped individuals using highly variable microsatellite markers and assigned each to a patriline. We then determined if traits differed as a function of genotype, and whether traits showed correlations between gynes and workers. Our data revealed patriline effects on four morphological traits located on the wings and legs. Further, principal component analysis generated new variables, Prin1 and Prin2, which also had significant patriline effects, indicating genetic influences on size and shape in this taxon. Linear regression analysis showed a strong, positive correlation between the caste phenotypes from each patriline, suggesting that genotype had similar effects on the size of workers and gynes. Lastly, scaling relationships were similar between castes, except for body length, gaster length, and Prin2, suggesting an important, genetically derived body “slenderness” characteristic that is caste specific. Overall, our data provide evidence for a genetic basis of trait variation among castes that may constrain caste evolution in social insects.