Mail is a pain in the ass, in general. God help you if you've got "normal" users.
Anyway, from a hostname standpoint, you'll want to make sure your system hostname is NOT just plain ol' "example.com" if you want to handle mail for "example.com". Go with "bob.example.com", or "mailsrv1.example.com", or "server.example.com", or whatever, but never "example.com". You'll want to set up an A record from that hostname to its IP address, and then set up reverse DNS from that IP address to that hostname.
The hostname does not have to have the word "mail" in it. However, it would be a good idea to create another A record, mail.example.com, pointing at the same IP address. This will be what you'll point MX records and mail clients at.
You then have three things you have to worry about:
1) Incoming mail via SMTP. This is handled by Postfix. On each domain you want to accept mail for, you'd set up
a domain and some users. Then, you'd add an MX record to each of those domains: blank hostname (i.e. just the domain), type MX, priority 10, target mail.example.com (which is an A record pointing at your IP). The target doesn't have to be in the same domain; it merely has to exist and not be a CNAME.
Make sure that example.com (or whatever the domain(s) are) does NOT appear in the mydestination setting for Postfix. Otherwise, your mail will go somewhere you aren't expecting it to.
If this is working well, messages to
bob@example.com should be plopped into /home/vmail/example.com/bob/Maildir. Are they?
2) IMAP connections from users. These are handled by Dovecot. I don't use Dovecot. If this is working well, you should be able to connect via IMAP, authenticate as
bob@example.com, and see whatever messages are in /home/vmail/example.com/bob/Maildir.
3) Outgoing message submission from users. These are handled by Postfix, typically using some sort of SMTP authentication, and often on port 587 ("submission") instead of the usual 25. It doesn't sound like you've gotten this far yet, but you should be able to send an outgoing message, authenticating as
bob@example.com, to some other domain. More importantly, you should NOT be able to send an outgoing message without authenticating. This is also something I don't tend to work with very often.
Posting a better problem description, logs, and/or config files might be helpful, as well.
_________________
Code:
/* TODO: need to add signature to posts */