I'm personally missing this development? Please clarify.Well, you're missing about 1800 years of Indian commentarial development.
Who said anything about abiding in "form sphere jhana?" The attainment of shamatha is not equivalent to the attainment of the first meditative stabilization, which is why it's described as preliminary or as access to the full realization of the first meditative stabilization. And we agree, one cannot engage in vipassana while abiding in the form sphere jhana. And yes, I also agree that the form sphere jhana is not essential to the path of seeing.Firstly, for Buddhaghosa and all subsequent Theravāda commentators who follow his system, one cannot engage in vipassanā while abiding in form sphere jhāna, because the object-basis of a form sphere jhāna is not a paramattha dhamma. Therefore, a yogi must emerge from form sphere jhāna, then engage in vipassanā prior to entering the first noble path of stream-entry. Thus, form sphere jhāna is an optional pathway, and not essential for the attainment of the path of seeing.
What relevance does this have to my quote above?And a number of Sarvāstivāda, Sautrāntika, and Yogācāra treatises also maintain that a śrāvaka can attain the path of seeing via prepatory stage samādhi (anāgamya-samādhi, which is the equivalent of access samādhi), without first developing form dhyāna (although, contrary to Buddhaghosa, et al, they all maintain that vipaśyanā can be engaged while abiding in dhyāna, if one has developed śamatha to that level).
I am realizing, thanks to comments such as yours and others, that there is a diversity of practice traditions in Theravada just as there is in Tibetan Buddhism. I'm assuming, however that these different practice traditions in Theravada all eventually lead to the same end?Secondly, contemporary Theravāda is primarily an ordination lineage these days, with a diversity of different practice traditions. Not everyone accepts Buddhaghosa or even the entirety of the Abhidhammapiṭaka as being authoritative. And teachers from different practice traditions don't always agree with each other. This is somewhat akin to Tibetan Buddhism, where there is the common Mūlasarvāstivāda ordination lineage, and a number of different commentarial and practice traditions (Nyingma, Kagyu, Sakya, Gelug, etc.) which don't always agree.
Metta,
Manjusri