Thats exactly what I did. I think the 2.4 base slack kernel has USB mouse issues.
I'm going to compile 2.6 at some stage over the next few days, then I'll see what happens.