Dspy is an exciting idea but is still in it's infancy

 Like all followers of the main AI bloggers, I saw Simon Willisons endorsement of DSPy. It reminds me of MLFlow. Early on in my pivot to AI, I recall many a langchain + Mlflow tutorial. It is premature. Evals. Evals. Have i ever actually created a good evals system. Nope.  A few things needed within DSPy. First of all, it has to be able to better handle complete garbage output from the student models (And the teacher models, for that matter -- not to mention the judge reflection models). What i mean is suppose I am creating an AI pipeline that does something. The example doesnt matter but lets say it is categorizing emails in bulk. I should be able to use Dspy to come up with a prompt that works through an optimizer like MIProV2. And if i pick a model that at DSPy's first attempt at a prompt for the task returns something, The optimizer should not only be expecting it to not be the correct category but structurally, not at all something that is what the system should be able to handle. When evaluating this library,  I guess the problem i see is that it seems useful in very unusual, specific circumstances. It is not robust enough t needs easy ways to have the optimizer first solve an easy problem before solving the harder problem. it needs easy ways to start with smart models and move to dumb models, and vice versa.  The dspy signatures dont solve problems. They are for simple tasks. 

The coolest thing about this stream of consiousness article was that when writing it had the links filled out . Kind of neat.