LLM and Project Management – The Intelligence in AI is limited!

The best known LMM (Large Language Model) is ChatGPT. The developers acknowledge ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as during the LLM training there is no single source of truth (the app is biased towards what it has been taught). The ‘training source’ for ChatGPT is the internet which has 1000s of incorrect posts on various aspects of project management – GIGO applies.  

To overcome this problem a number of developments based on groomed training materials are in development, some proprietary in-house, and some to be publicly available. Two of these are:

Marvin, a collaboration between Projecting Success and Praxis, Mosaic is supporting this development – watch this space in 2024.

PMI’s Project Infinity, based on the vast library of publications and papers held by PMI.  This tool is still in Beta, but as a PMI member I’ve had access to test the model.

Project Infinity Test (1)

The ‘intelligence’ in LLM applications is the ability to understand and reply to language in a coherent way.  You ask a question, the app understands what you have asked, searches its learning, and formulates an answer you can understand. What seems to be missing is the ability to qualify the information in its systems, particularly when specific facts are required.

My test for Project Infinity was based on asking “What is the critical path?”.  The correct answer to this question based on the PMI Practice Standard for Scheduling (Third Edition) and ISO Standards is something similar to: 

Critical Path: sequence of activities that determine the earliest possible completion date for the project or phase.  See: Defining the Critical Path 

As anyone with even a limited knowledge of CPM scheduling knows, ‘Zero Float[1]’, or float in general cannot be used to define the critical path for the following reasons:

1. Date constraints can cause changes in the float value, usually to create negative float, see: Negative Float and the Critical Path
   
2. Multiple calendars can change float values along the critical path, see: Calendars and the Critical

A summary of my ‘chat’ with Project Infinity is:

Q1 – what is the critical path?

       ‘The critical path is the longest sequence of activities in a project plan which must be completed on time in order for the project to be completed on schedule. It is determined by identifying the activities that have zero slack or float time,….’

Q2 – Why are you claiming the critical path has zero float when it may have negative float?

        ‘Negative float can occur when an activity’s late finish date is earlier than its early finish date, which means that the activity is taking longer than planned and is delaying the project. However, the critical path always has zero float because any delay in the critical path activities will cause a delay in the overall project timeline. …….  In summary, while negative float can occur in non-critical path activities, the critical path always has zero float because any delay in these activities will cause a delay in the overall project timeline.’

Q3 – so what happens when the finish milestone and the project overall have a constrained finish?

        ‘When the finish milestone and the project overall have a constrained finish, total float may be used to provide an early indication of potential project completion slippage.’
[Note: I’m assuming ‘Total Float’ can be negative PI did not say so.]

Q4 – Why does your third answer contradict the first answer?

        ‘My third answer does not contradict the first answer. ……’

The full exchange can be seen at Project Infinity Test 1 – Critical Path. 

Conclusions

This simple test shows at the current level of development Project Infinity has a lot or learning to do:

• It ignored key PMI documents, in particular the PMI Practice Standard for Scheduling (Third Edition)
   
• It failed to recognize a direct contradiction in its answers.
  

Therefore, while LLM tools can help save time bringing together information from diverse sources, their training to date is limited to absorbing information from documents, the next stage of development involving qualifying and grading the data may be a way off. So if you do not know the right answer to a question, you cannot rely on an AI tool using LLM to provide you with a way out.  

To make matters worse, accountability in AI is a complex issue. We know AI systems can misstep in numerous ways, which raises questions about who is responsible? This is a complex legal issue and in the absence of someone else who is demonstrably at fault, you are likely to carry the can!

For more on AI in project management see:  https://mosaicprojects.com.au/PMKI-SCH-033.php#AI

[1] The concept of the critical path having zero total float arose in the 1960s when computer programs were relatively simple and most schedules were manually drawn and calculated. With a single ‘Day Number’ calendar and no constraints the longest path in a network had zero float. The introduction of computer programs in the 1980s that allowed multiple calendars and constraints invalidated this definition.