Currently, the BPEL transport in OSB is not supporting FMW 11g. However, I just found a way, although it still is proven in theory based on my knowledge, to enable transaction propagation between OSB and FMW 11g SCA composites. The basic idea is that you have to communicate between OSB and FMW 11g SCA composites using the SDO - EJB binding in 11g. The t3 protocol used as the communication protocol between the ejb client and the SCA (soa_infra) engine should take care of the transaction context propagation.
This should do the trick until OSB gets native support for FMW 11g interoperability. One disclaimer; I still have to prove my theory by running a test, but I am quite sure that it will work so therefore I shared it already here. I also still have to elaborate on which MEPs could be supported with this solution, so any thoughts are welcome.