I found this incredible documentary on YouTube with tips on writing dialogue, including interviews with screenwriters and authors. I thought I would post some of the top tips from it for other writers out there who want to improve their dialogue. If you'd like to watch the full show, click here.
When you're writing dialogue, you have to choose with intent what must be said, and what can be shown through background details, facial expression, body language, gestures, etc. A general rule of thumb is once you think you've narrowed your dialogue down, "cut 20%". You don't want something too dialogue heavy with too much explaining, there are other ways to give information to your viewers. I'm writing a comic, and this applies tenfold - the background, expressions, camera angle, and all other visual information are essential. Basically, you want to give the most amount of ideas in the least amount of words.
You've heard the tip before about going and recording conversations in a coffee shop, seeking conversations "in the wild". This isn't exact, because people talk in circles; they're not direct and to the point. You have to cut a lot out... However, there is a lesson to this: people rarely say what they mean. There is always subtext. What they say and what they want are often different from each other, sometimes in a purposeful, manipulative way, and other times because the character doesn't want to be vulnerable and say what they really think.
People also often misunderstand each other in real life. For example, if your partner says, "I'm sick of you working your job." You might just hear "I'm sick of you" and come up with your response before hearing the end of their sentence. Dialogue isn't real if all characters understand exactly what the other said. Another example is a wife has kicked her husband to the couch after learning that he's been having an affair. After a few weeks, she says him "That sofa can't be too comfortable." He thinks she's inviting him back up to bed, and she intends it as, you need to find somewhere else to stay. People project their own desires and fears into conversations.
You know those moments where someone says something to you, and you become tongue tied or are too polite to say what's in your mind, or even worse, you think of the perfect response too late? THAT is what makes good dialogue - the fact that you wouldn't say it in real life.
Another method is to create your character in your ideal self. The character says and does everything you wish you would. My thoughts: Perhaps another character carries many of your insecurities, and their words and actions reflect more of your flaws. A different character is maybe then based off of your childhood best friend. Using real life people and personalities can help develop the character and what their best reaction would be.
There are layers to story writing: Text - the dialogue, the words that are said Subtext - the intention, what people want (this can be shown from behaviour, and often someone's behaviour and desires, or their speech and desires, are opposing). Context - the story itself, the situation the characters are in, including the conflict they're facing
Every character is different. If you ask 10 people to say the same thing, they'll do it differently. For example, one of the writers in the show goes to a coffee shop and observes the different baristas. Each of them wear the same uniform and have the same "lines" or dialogue with the customers, but they add their own spin on it. One person always has a positive attitude, whereas another person always makes conversations about himself. Think about the general line for the situation, and consider how that particular character would say it in relation to others.
Consider vocabulary: Every character will have a different way they say common words like yes or no, curse words, pet words (very, really, actually, but, whatever), and filler words (uhm, ah, right, okay, like, y'know/you know). Make a list of common phrases and assign them to different characters according to what suits them best.
After your first draft, look over each line and ask: Why is the character saying this line? Often you'll find it's because you want the character to say it. You need to figure out what the character wants, because what they say is how they try to get there. Rewrite it until it drives the character closer to their goal. Each scene is a power grab - everyone wanting to take control of the situation, in whichever way that means to them.
Read it aloud! Whether you do it on your own (try different voices like actors would), or with a friend, it will help you figure out what doesn't sound right.
Avoid scenes where one character is asking the other a bunch of questions - it gets tiresome.
Consider dynamics between people. In any given scene, the characters should be opposites, contrasting each other in what they say and how they say things. One should have more formal speech and the other more slang or use of contractions. One character might be longer winded, and the other speak in very short sentences.
One final tip: You won't be focused on all of these tips in the first draft. In the first draft, you're often getting the story into place. The next draft, maybe you're more focused on word choice, and a different draft could be focused on character and voice.
All of this information is from the Film Courage Documentary, "10 Tips on Writing Better Dialogue." July 3, 2018, https://www.youtube.com/watch?v=qfQpJxRP8ew&t=477s.