With the fast progress of MIMO technology and its growing applications in networks, it is important to develop techniques to enable more efficient MIMO network communications. However, in a dynamic ad hoc network with changing channel conditions and topology, it is very challenging to coordinate channel evaluations and data transmissions as well as fully exploit the advantages of MIMO technology. Having observed that most of the literature works lack realistic model of the interaction between physical layer and the upper layers, we are trying to model the realistic physical layer characteristics and constraints to assist the design of algorithms and protocols that enhance the performance and efficiency of MIMO ad hoc networks. This dissertation focuses on modeling and algorithm design of MIMO ad hoc networks and addresses several important issues spanning over several network layers. First, we formulate a concrete physical model for MIMO ad hoc networks, and propose cross-layer algorithms which take advantage of physical layer channel information to opportunistically schedule cooperative spatial multiplexed transmissions between nodes to maximize the network throughput. Second, we propose a series of scheduling algorithms to support opportunistic and cooperative MIMO transmission in different scenarios, including adaptive scheduling in heterogeneous transmission environment, seamless use of cooperative relay to cope with harsh channel condition and distributed interference management to better leverage MIMO advantages. Third, we formulate the MIMO-enabled multi-source multi-destination multi-hop routing problem into a multi-commodity flow problem and developed a polynomial time approximation solution that maximizes the scaling factor for the concurrent flows as well as a distributed algorithm to minimize the congestion in the network links. Finally, we study the deployment of MIMO nodes as relays to assist weak links in wireless networks, with the aim of reducing the number of relay nodes and providing performance provisioning. We provide a polynomial-time approximation scheme algorithm, as well as centralized and distributed algorithms to effectively determine the MIMO relay nodes positions over the network and flexibly select various transmission strategies to further leverage the advantages brought by MIMO.