It's a great Idea. I did a lot of b2b work and a majority of successful ones I saw were people that after they made the intial investment could put more profit back into the business for leads etc and have it run autonomously and eventually make more money then your day job.
Stop it.... At the core this business is not that difficult. Esp if you concentrate on a simple area like health or term life. Easily understood products, advance contracts etc. The only reason people dont make money in this field is they suck at their craft (due to poor training or mentors) or they dont work.
If you want to make six figures you cant really work 4 hours a day but if you want to make 40,000 you can certainly do it.
Nothing is impossible, I believe that many people have to start part-time because they cannot afford to quit the source of income that they are currently surviving on until their career takes off. Just make sure if you have to do this part-time that you are disciplined enough to put in hours each week.