They are only similar to a certain degree. Both are irrational reactions to fear and can make you nervous and sometimes paralyze you with fear to the extent you can not function as you normally would.
However, your condition changes from panic to a phobia when you start to avoid the situation that causes you the fear. This is how almost all people with a debilitating phobia of public speaking started their phobia. One or several perceived embarrassments in front of one or more people caused them to start avoiding the situation. Notice the word perceived. Even if the audience did not notice, the subconscious picks up what you tell it. The same goes for any other phobia. The fear causes the person to start avoidance. Whether it is bridges, people, traffic or the workplace just to name a few.
Some people may face the same situation as the anxiety prone individual faces and only feel slight discomfort if any, and just shake it off, so to speak. However the anxiety prone individual starts to ponder the situation and process the thoughts of discomfort and they start planning how to avoid that discomfort in the future. They may go on for days thinking about the situation. This process only heightens the anxiety level.
Some professionals now believe that some physical attributes of illnesses such as IBS are forbidden by the fear of the consequence of the illness. For example a person may have had a close call in not being able to get to the restroom in time or maybe they did not make it to the restroom in time. In the anxiety freedom individual, not being near a restroom can quickly become a phobia. The phobia in this situation is the fear of the embarrassment of soiling their clothes and being humiliated in public.
When the fear leads to avoidance, then the fear has become a phobia.
Some people may have a simple phobia of one thing. They manage to avoid that one thing until one day they are forced to deal with it, such as a forced speech by the boss or forced business trip on an airplane. In the extreme anxiety person, the phobia can be something as simple as standing in line at the grocery store. Dealing with the fear and the phobia is important so that you do not live your life in fear of things that are normal to everyday life.
In fact, avoidance of one thing can lead to avoidance of another. For example a phobia of crowds, traffic and cars can lead to agoraphobia, where you are too fearful to wander outside the safety of your living environment.
This is one of the main reasons it's so important to face your fears, one by one. I'm not advocating that you become a lion tamer to conquer your fear of lions. I'm just suggesting that you face the fears that keep you from being the person that you dream of being. You can do it, but only you can do it. I speak from experience. Face the fear and panic and in the process, you will never know the phobia.