It basically boils down to two issues.
Immigration, as you mentioned, is one of the issues. As one of the strongest economies in Britain (and being English-speaking and a reputation for being more acceptive of immigrants) Britain saw a huge influx of EU migrants in the last decade. Many parts of Britain are feeling the impact of overcrowded social services and a severe housing shortage. The ramp up in EU immigration followed a decade of very large non-EU migration under the previous Labour government. There are also cultural aspects to the immigration issue. The government and economists generally claim that immigrants are a net economic benefit to British society but most of these studies have not factored in the burden placed on housing and social programs, which have not kept up with the population growth particularly in the south of England. While the British government can take steps to reduce non EU immigration, under current EU laws they cannot prevent any EU citizen from moving to the UK, so there's a perception of uncontrollable immigration for the foreseeable future.
Sovereignty is the other issue. Britain entered the EU when it was still the Common Market and strictly a trade entity. Over the years the Common Market has transformed, largely through stealth, into the present day EU, which is now an entity that no one in Britain voted for in the Common Market referendum in the 1970s (although if Remain side wins the referendum it will be seen as an indirect approval of the EU). As the EU has grown, more power has been transferred from national governments to a centralized EU government in Brussels, which is not directly elected by the British population and is a very opaque, byzantine machine with minimal accountability yet with tremendous powers. The trend of the EU is towards greater and greater centralization and eventual federalization. Many people in Britain are not happy about this trend. They are also unhappy that each stage of the EU's growth happened more or less on the sly. Tony Blair promised a vote before he signed a key treaty, but he went ahead and signed it without calling for one. Gordon Brown signed another major treaty handing over powers without the direct consent of the British population. In many other EU countries, referendums were held, but the results were ignored if the voters rejected the proposed treaties (or submitted to another referendum, as my Irish friend, who is generally pro EU, even admitted that in the Irish case, the government literally promised to keep having referendums until they got the result they wanted).
If you can imagine the countries of North America in a similar set up like the EU, with free population movement across all the countries from Canada to Mexico to the various islands, with a supranational court headquartered in, say, Havana or Mexico City that takes precedence over the US Supreme Court, a supranational government and political body of whose representatives the vast majority are not from your country and more often than not outvote your own representatives, a common currency for all these nations, a substantial portion of your own laws have to be rewritten to incorporate mandates from the political body despite that you didn't vote for it, well, that gives you a decent idea of what the EU is like. That's why it is always hypocritical for Americans, if you consider a nation's sovereignty to be paramount as many Americans do, to say Britain should stay in the EU if it prefers to leave, as the Americans (or Canadians) would never enter such a similar entity.
The real concerns for Brexit outside the UK or the EU is primarily fears that it would lead to severe economic repercussions not just for the UK or globally, but I've always disinclined to believe it. The economists predicting doom and gloom are generally the same economics who completely missed the 2008 economic collapse. Or the IMF, which accepted austerity for Greece with predictions that the Greek economy would return to regular growth by 2013 (laughable, as you can see).
If the Remain side wins, it will largely be because of these economic fears, and it is understandable why so many people are unwilling to risk the status quo where things aren't overly bad for the unknown. The long term future of the EU, however, I find interesting. It's currently a veneer of respectability that barely hides bubbling dissatisfaction across many parts of Europe.