When you install VirtualBox it builds a driver to match the running kernel; the driver isn't actually part of the kernel, but merely compatible with it. It's some time since I've used PCLinuxOS, but there will be a configuration file somewhere in /etc which is responsible for loading modules at boot time. This may be /etc/modules, containing a simple list of modules to be loaded, or it may be something like /etc/init.d/boot.local, which is a script which can have some modprobe statements added to it.
I think all you'll need to do is locate the relevant config file and delete the references to the unwanted drivers.
