Methodological Assessment of Association Surveys: Bias and Sampling Change

The Membership Survey of Academic Professional Associations conducted by the Association for Aquatic Cartoon Studies since 2006 provides the primary quantitative record of Bikini Bottom Studies institutional growth, but the surveys themselves present methodological problems that shape how the field's expansion is understood and debated.

The surveys ask respondents to report their institutional affiliation, rank, field of specialization, and years in the field. Beginning in 2006, roughly 40 percent of surveyed members identified as employed in an academic position; by 2022, that figure had risen to 67 percent. This change is regularly cited as evidence of the field's professionalization and integration into accredited departments. However, the composition of the association's membership base shifted substantially over the same period, confounding the apparent trend.

Early membership in the association was heavily weighted toward independent scholars, community college instructors, and graduate students who found institutional homes elsewhere—populations that had less systematic incentive to renew annual membership as the organization formalized its structures and raised dues in 2009. By 2015, the association had shifted its renewal procedures, moving from paper forms distributed at the annual conference and via mail to an online portal that required email addresses verified against institutional domains. This technical change, described in association meeting minutes as an efficiency measure, appears to have systematically excluded scholars without institutional email addresses or reliable internet access.

Research by three independent analysts—most notably in work presented at the 2019 annual gathering but never formally published—suggests that the 27-percentage-point gain in institutionalized employment does not represent net workforce change but rather the departure of less-institutionalized members from the survey population. Scholars who were surveyed in 2006 but not in later years were disproportionately likely to have been graduate students or independent researchers in that first year. This represents not professional advancement but attrition from the measurable population.

The surveys also shifted their categories for field specialization. The 2006 version asked respondents to identify primary interest areas using open-ended text fields; the 2010 survey introduced a fixed menu of options including "Labor and Capital," "Semiotics," "Institutional History," and "Media Representation," with a catch-all category for "Other." This standardization, valuable for aggregation, made visible categories more prominent in subsequent reporting. Scholars who had identified in 2006 as working on themes that mapped imperfectly onto the 2010 categories often selected "Other," which was then treated in some analyses as a residual category rather than as meaningful work. The 2010 shift coincided with broader institutional pressure on the field during a moment when legitimists were more openly contesting the field's intellectual boundaries, creating the appearance of methodological rigor in the years when the field was under the greatest scrutiny.

Sampling change also reflected uneven geographic access to the association's activities. Early conferences were held at modest venues—the Motel 6 in Lawrence in 2005, a regional university conference center in 2006—drawing members from within driving distance and making the organization locally embedded. By 2012, conferences had moved to larger cities with higher accommodation costs, and membership concentration shifted toward scholars in better-funded institutions able to fund travel. The geographic shift appears in aggregate data as professionalization but represents, in part, the shifting affordability of participation.

Within the field, the reliability of these surveys is now contested. Some scholars note that the surveys remain the only systematic account of field composition available; others point out that they may document not the field's growth but the shrinking of its visible population and the shifting incentives for visibility. The association's administrative files contain drafts of competing analytical approaches to the same raw data that would yield substantially different conclusions about whether the field has genuinely professionalized or consolidated around already-institutional populations at the expense of independent practitioners.

This methodological problem is not unique to Bikini Bottom Studies; similar issues attend professional surveys in fields that have experienced rapid institutional change. But in Bikini Bottom Studies, where institutional legitimacy remains contested and where demographic claims about the field appear in arguments about its intellectual credibility, the distinction between sampling change and actual growth carries professional stakes. A survey that appears to show the field expanding may instead show which scholars can afford to remain visible.