I tried creating a listbox using headlessui. I simply copied the code from HeadlessUI Listbox and just increased the people count and added max-h-40 and overflow-y-auto to ListboxOptions element.

import {
  Listbox,
  ListboxButton,
  ListboxOption,
  ListboxOptions,
} from '@headlessui/react';
import { CheckIcon, ChevronDownIcon } from '@heroicons/react/20/solid';
import clsx from 'clsx';
import { useState } from 'react';

const people = [
  { id: 1, name: 'Tom Cook' },
  { id: 2, name: 'Wade Cooper' },
  { id: 3, name: 'Tanya Fox' },
  { id: 4, name: 'Arlene Mccoy' },
  { id: 5, name: 'Devon Webb' },
  { id: 11, name: 'Tom Cook' },
  { id: 12, name: 'Wade Cooper' },
  { id: 13, name: 'Tanya Fox' },
  { id: 14, name: 'Arlene Mccoy' },
  { id: 15, name: 'Devon Webb' },
];

export function App() {
  const [selected, setSelected] = useState(people[1]);

  return (
    <div className='mx-auto h-screen w-52 pt-20'>
      <Listbox value={selected} onChange={setSelected}>
        <ListboxButton
          className={clsx(
            'relative block w-full rounded-lg bg-white/5 py-1.5 pr-8 pl-3 text-left text-sm/6 text-white',
            'focus:outline-none data-[focus]:outline-2 data-[focus]:-outline-offset-2 data-[focus]:outline-white/25'
          )}
        >
          {selected.name}
          <ChevronDownIcon
            className='group pointer-events-none absolute top-2.5 right-2.5 size-4 fill-white/60'
            aria-hidden='true'
          />
        </ListboxButton>
        <ListboxOptions
          anchor='bottom'
          transition
          className={clsx(
            'w-[var(--button-width)] max-h-40 overflow-y-auto rounded-xl border border-white/5 bg-white/5 p-1 [--anchor-gap:var(--spacing-1)] focus:outline-none',
            'transition duration-100 ease-in data-[leave]:data-[closed]:opacity-0'
          )}
        >
          {people.map(person => (
            <ListboxOption
              key={person.name}
              value={person}
              className='group flex cursor-default items-center gap-2 rounded-lg py-1.5 px-3 select-none data-[focus]:bg-white/10'
            >
              <CheckIcon className='invisible size-4 fill-white group-data-[selected]:visible' />
              <div className='text-sm/6 text-white'>{person.name}</div>
            </ListboxOption>
          ))}
        </ListboxOptions>
      </Listbox>
    </div>
  );
}

The max height is not being set in the listboxoptions. Output in image below
result

An alternative method that worked was instead of max-h-40, I used h-40. The result was as expected. However, this creates another issue: when there are fewer elements, there is extra space. I decreased the people count and now there is extra space

const people = [
  { id: 1, name: 'Tom Cook' },
  { id: 2, name: 'Wade Cooper' },
];

<ListboxOptions
          anchor='bottom'
          transition
          className={clsx(
            'w-[var(--button-width)] h-40 overflow-y-auto rounded-xl border border-white/5 bg-white/5 p-1 [--anchor-gap:var(--spacing-1)] focus:outline-none',
            'transition duration-100 ease-in data-[leave]:data-[closed]:opacity-0'
          )}
        >

Please find the output of the above case below
result

what I expect to happen is if there are fewer options, it takes only enough space as needed and if there is overflow, there must be a max height and overflowing data must be scrollable.