The exact formula depends on your programming language. For example, it would be different in PHP and Excel. So without that detail (and a lot more) all I can give is the pseudocode:
if (difference > 5 hours) difference = difference - 0.5 hours
Can you help me with creating a timesheet where staff enter the start and finish time have the next field calculate the total time and if its more than 5 hours automaically taking off 30minutes..?