Cisco Switching/Routing :: ESW520 - VLAN Configuration

I have a headquarters office that has recently bought a new voice system.
 
We have a site to site connection from the head office to 3 remote offices. All have ASA5505 firewall.
 
I have created 2 interface on the ASA5505: 1: inside with vlan1 and switch port port12: voice with vlan100 and switch port port2
 
Port1 on the asa goes to port 23 on the switch for vlan data
Port2 on the asa goes to port 24 on the switch for vlan voice
 Port23 is member of vlan1 data

I added vlan100 to port 24 but by default vlan1 is member and i can't remove it. Its greyed out
 
All ports on the switch are member of vlan1 and vlan 100 because the port on the switch goes to the phone and from the phone there is a port that goes to the PC. Phones are getting addresses, and PC as well.
 
I am having a kind of loop because there is 2 exit for vlan1 (port 23 and 24) and that's slowing down my system and sometime i loose the connectivity to my servers.
 
When i do a show arp on my asa, i do see that some IP are beeing learned on the wrong interface. some PC addresses 192... are on the voice and some voice IP 10.10... are on the inside.
 
I am pretty sure that the problem comes from my switch configuration.