A mixed retail and affordable housing development proposed by Adam Potter, the founder of the Friends of Bay Street, for the office district in Sag Harbor Village hit a speedbump recently, as news got out that the developer he was working with, Conifer, had dropped out of the project. The Smith & Henzy Advisory Group, a real estate development firm brought in by Conifer and specializing in funding large-scale housing developments through tax credits, also departed.