As per WFTDA, it's a minimum of 3, a maximum of 7. However, you may see the maxium change for large competitions.
For intra-league competition, I think 6 is enough, but I'd prefer 7, even 8.
They have various duties.
1 Head ref - confers with team captains and managers about any questionable calls, concerns, answers, etc. they can also overturn calls made by other refs at their discretion, call fouls, penalties, etc. they're focus is on all the players on the rink.
2 Jammer refs - they follow the jammers, and keep their scores. they also watch for lead or no-lead jammer status, and whether they foul or are fouled.
1-4 pack refs - they watch the pack for fouling, 20 ft. infractions, equipment failure, pulling people for major fouls, etc.
I would say you need a pretty decent skater. After experiencing the tournament this past weekend, and the speed and control of the players, it only makes sense the refs are on par with their abilities.