The people you invite don't have to click Yes or No -- they can just ignore your invitation if they don't have time to join you. The whole point of hangouts is that they're semi-spontaneous "Let's chat" invitations, not obligations. Keep that in mind when you're inviting people to hang out -- folks with day jobs, for example, might get annoyed if you send repeated hangout pitches during work hours. If someone you've invited to hang out tries to start their own hangout while yours is in session, they'll receive a notice that your hangout is already happening so they can consider stopping by yours, instead.
Google states on one of its Google+ help pages that "No one owns a hangout," meaning that anybody in the hangout can invite anybody else they'd like to join the hangout. So you may start a hangout with just a few select people, but one of those people could click the Invite button and send an invitation out to all their circles, or even set it as Public, and that invitation will show up in people's streams. The invitation post may include your profile picture (except to people you have blocked) to show that you're one of the participants. So hangouts aren't a way to have a secret< meeting -- just like you wouldn't hold a secret meeting on your front porch.
Camera, microphone, and speakers working? Check. Invitations sent? Check. Now that you've done all the prep work, go ahead and click the "Hang out" button on the hangout setup page. When you do, you'll see the following graphic.
Aww. That robot looks so sad with all those balloons and no one to share them with. But no worries -- your invitees probably just haven't seen the link to your hangout yet. Remember the dorm room/front porch analogy -- folks just need some time to wander by and join in. And hangouts aren't like Skype calls (voice or video calls you make via the Internet), where your chat request "rings" each person individually. Give it some time, and people will start showing up, most likely.
As people start showing up, small versions of their video feeds show up at the bottom of the hangout window, just above the control buttons. (If somebody is going audio-only for this hangout, then you see their Google+ profile picture instead of a video feed.) Google+ tries to figure out who's currently talking and puts that person's video (or picture) in the upper part of the window and makes the image much larger than the tiny ones below. Google+ doesn't do a perfect job of this, but it's pretty good job at switching between speakers. You have to try a hangout or two to get the feel for it, but it's really pretty natural after you settle in.
Down in the lower-right corner of the window are some handy controls. The Mute Video button temporarily turns off your webcam so the other people in the hangout can hear you but not see you (useful, for example, if your toddler niece wanders into the room to show you her new diaper-removing trick). When your video is muted, folks will see your profile picture instead of your video feed. The Mute Mic button turns off your microphone (in case you need to sneeze or answer the phone, say). Click either button a second time to undo its effect, or click the Unmute button that appears at the top of the hangout window. The Settings button takes you to the settings screen you saw on Inviting People to Hang Out.
The buttons in the lower-left corner of the window are more interesting. Click Invite to bring up the standard circle- or people-picking box so you can type in names of people or circles to invite them to hang out. The Chat button opens up a panel on the left side of the hangout window, where you can type things instead of saying them out loud. This is helpful for pasting in links to websites (or YouTube videos, as explained next), for those hanging out with video but not audio (such as folks joining from work), and when someone wants to contribute a quick thought without interrupting the speaker (especially useful in formal hangouts, like team meetings). Click the Chat button again to collapse the panel.
The YouTube button is mostly just for fun, but it can be a practical tool if there's a video you want to share with the group. When you click the button, the center of the hangout window changes to let you search for YouTube videos. Type a search term in the text field, hit Enter (Return on a Mac), and then click the video you want everyone to see.