CIS5930-07 Parallel Computing: Announcements and review questions

Review questions:

1. Do you know the following terminology? SISD, SIMD, MIMD, SMP, UMA, shared memory machine, distributed memory machine, NUMA, cc-NUMA, Distributed shared memory, SPMD, message passing, shared memory programming, task/channel model, multicomputer, multiprocessors. (We have discussed some of these topics in class yet, but you should read Chapter 1.)

2. Give an example to demonstrate why lack of cache-coherence can lead to incorrect results.

3. Suggest some methods of ensuring cache coherence efficiently.

4. Can you simulate, through software, a distributed memory machine on a shared memory machine, or a shared memory machine on a distributed memory machine?

5. Can you efficiently simulate, through software, a distributed memory machine on a shared memory machine, or a shared memory machine on a distributed memory machine?

Review questions:

1. Do you understand the issues of: partitioning, domain decomposition, functional decomposition, communication, local vs global communication patterns, structured vs unstructured communication patterns, static vs dynamic communication patterns, synchronous vs asynchronous communication patterns, agglomeration, mapping, use of graph partitioning in mapping, divide and conquer paradigm.

2. For the first prefix scheme we discussed in class, can you show that the time complexity is O(log²N)?

3. Can you come up with an example where you may be willing to sacrifice load balance requirement to improve the total computation time?

4. We considered parallel prefix in a message passing paradigm. How would you implement it in a shared memory paradigm? What will the time complexity be? What assumptions on memory access did you use to derive the time complexity (for example, can multiple processors read the same memory location simultaneously? Can they write simultaneously?)?

5. Design a parallel algorithm for matrix-vector multiplication and matrix-matrix multiplication.

Check HW 1

Check the partial list of papers, from which you will present later in in the semester.

Review questions:

1. Do you understand the issues of: the various factors to consider in performance (such as execution time, memory requirement, software development cost, etc), Amdahl's law and limitations, limitations of extrapolating from observations, asymptotic analysis, modeling execution time, the communication model we consider, and improvements to account for contention, efficiency, speed-up, scalability analysis, iso-efficiency function, the different network topologies.

2. Can you perform a scalability analysis for the parallel prefix algorithms we discussed in class?

3. Can you create a torus using only "short" wires (that is, wires of constant length)?

4. Can you create a hypercube of dimension greater than three in the 3-dimensional world in which we live? Can you create it such that wires do not cross? Can you embed an arbitrary graph in 3-D space so that wires do not cross?

Review questions:

1. Do you know the following: modularity issues in sequential and parallel software, the three composition techniques, their advantages and disadvantages, different matrix distribution schemes (block versus cyclic, striped versus checkerboard, one dimensional versus two dimensional)?

2. Given a problem, can you suggest a suitable parallel algorithm and data distribution, and discuss the trade-offs involved with different composition techniques? For example, give the total memory required, discuss factors that may change the total execution time, and analyze the communication and computation costs.

3. For an example of the above, try to analyze the following two problems: (i) addition of N numbers, and (ii) matrix-vector multiplication, where the matrix is distributed in a checker-board manner, while you can assume any suitable distribution for the vector, with each processor having N/P elements of the vector.

Pacheco's tutorial

Gropp's tutorial

Review questions:

1. Do you know the following? The 6 basic MPI calls, need for tags and communicators, buffering issues and deadlock, how to prevent deadlocks, immediate sends and receives, duplicating and splitting communicators, collective communication, topologies, derived data types (vectors and structures).

2. Given a desired topology (for example, a hypercube), can you give suitable arguments to create a Cartesian mesh that is identical to the desired topology?

3. Can you give an example to demonstrate how send/recv can cause deadlocks, or given an example, can you determine if deadlock can occur, under what conditions, and how it can be prevented using facilities provided by MPI?

4. How can you implement reduction using only sends and receives?

5. Given a sequential algorithm, you should be able to write an efficient MPI program for it. For example, try to do this for matrix-vector multiplication.

Review questions:

1. Do you know the following? concept of threads, OpenMP execution model, compiling and OpenMP program on the SGI origin 2000, compiler directives for creating a parallel region and work-sharing a for loop, data scope attributes clauses (private, last private, and first private), how private variables are created, reduction, library calls to set the number of threads, and get the thread number.

2. Can you give examples (other than those discussed in class) to demonstrate errors that can occur in a program when multiple threads execute a piece of code?

3. Given a piece of sequential code, you should be able to parallelize it with OpenMP directives. For example, try parallelizing matrix-matrix multiplication.

4. What do you think is the most likely reason for the loop variable in a work-shared construct to be private, by default?

5. What do you think is the most likely reason for the restrictions OpenMP places on the type of for loops?