If you want to make a driver that uses hardware acceleration, then no, that’s not possible. For some graphic cards, there are open source drivers. Have a look at Mesa and the DRM project. But there is nothing that works independant of the hardware.
Of course you can write a software implementation that sits on top of your VBE3 driver. Or just use Mesa.