Citrus trees need a warm climate to survive, which is why you don't see many orange or lemon trees in Boston. In fact, there are only a few places in the U.S. warm enough — over 70% of the country's citrus fruit comes from the state of Florida.
The USDA actually divides the country into "hardiness zones", a scale that defines how cold each areas' climates can get. Most citrus plants requires a zone between 9 and 11 on the scale — if an area gets any colder than 20° F, the plants won't make it.
Boston is a little lower and colder on the scale. We're between a 6 and a 7 — a climate more suited to hardier plants like apple trees, elderberry bushes, and pawpaw.
Despite the cold, there's a place in Boylston (near Worcester) where you can see oranges, lemons, and limes growing in the dead of winter: Tower Hill Orangerie.