Looks like creating the software raid volume also fails when you have another existing array. I have another software array (RAID 5) in my machine. Error on the console was ls: cannot access /sys/block/*/holders/*/dev: No such file or directory after it seemed to get info about the wrong RAID array (the existing RAID 5 one). After disconnecting those disk it worked fine.