Programming Project 2 Distributed File Systems

COP 5611, Operating Systems, Spring 2003

Department of Computer Science, Florida State University

¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾

Points: 100

Due: Week 15, Thursday, April 17, 2003

Maximum Team Size: 1 ( In other words, this is an individual project)

Purpose: To know how to implement distributed file systems and experience technical and design issues in distributed file systems.

Background: A distributed file system is an important resource management component in distributed operating systems. The primary goal of a distributed file system is to provide network and location transparency so that users do not have to be aware of the location of files to access them.

A typical distributed file system is organized as a client-server architecture, where a number of file servers store and manage local files and provide services to clients. To provide location transparency, a name server is often used to map files names to different file servers.

Server and Client Processes: To simulate a distributed file system, in this project there will be three types of processes. There is a name server which provides the file server IP and its location for a particular file. There are several file servers which manage local files and provide services to clients upon their requests. Finally there are a number of client programs which provide a shell-like interface for a user to copy/delete/create/list files and directories in the distributed file system.

Because file systems are typically part of an operating system, to simplify the problem, in this project we will implement a virtual distributed system at the user level on top of the operating system through TCP/IP protocols. More specifically, under a directory of your choice (which acts as the root directory of the virtual distributed file system), there will be a number of locally mounted subdirectories, each of which is managed by a file server. The following is an example, where the root of the distributed file system is located at /home/faculty/liux/cop5611-DFS, and there are five subdirectories. From this diagram alone, we cannot distinguish local subdirectories on the main file server and the ones that are locally mounted. This is specified in the name server (see below for an example).

· Name Server

The name server maintains a prefix table that shows the file servers in the system. For each file server, it specifies the logical path in the distributed file system, its IP, and port, and the corresponding directory on the file server. For the above example, the prefix table may look like this

Logical Prefix	Server IP	Server Port	Remote directory
/	128.186.120.34	1281	/home/faculty/liux/cop5611-DFS
/linprog1	128.186.120.53	1282	/tmp/liux/cop5611-DFS
/program1	128.186.120.33	1280	/tmp/liux/cop5611-DFS
/program3	128.186.120.55	1285	/tmp/liux/cop5611-DFS

This table indicates that there are four file servers in this system: The “/” directory is implemented on “program1” at port 1281 and there are three additional servers. Two of the directories (namely “linprog2” and “program2”) are local directories on the main file server.

The name server responds to the following client request.

- File Server Lookup Request: Given the logical path provided by the client, the name server returns the server IP and its port to the client.

· File Servers

Each file server responds to the following client requests.

- File Create Request: The server creates the file specified by the client. If the file exists, it will be overwritten. Then it returns a message to the client whether the operation is successful or not.

- File Timestamp Request: It returns the most recent time that the file specified by the client has been modified.

- File Header Request: It returns the information about the file such as its size.

- File Open/Read/Write/Seek/Close Requests: We assume that these requests will happen in order. For the File Open Request, the server opens the file and returns the file descriptor to the client. Then the client sends subsequently a number of File Read/Write Requests, which the server reads from the file and sends the data back or writes the received data to the open file. After the file is open, a user can also change the position of the next Read/Write operation using a Seek Request. The Close Request will close the open file. For each request, the server sends an acknowledgement, indicating whether the operation is successful or not.

- Directory Create Request: The server creates the directory specified by the client and returns a message indicating whether the operation is successful or not.

- File Delete Request: The server deletes the specified file and returns a message indicating whether the operation is successful or not.

- Directory Delete Request: The server deletes the specified directory if it is empty and returns a message indicating whether the operation is successful or not.

- Directory List Request: The server returns a list of files and subdirectories in the directory specified by the client. Note that recursive listing is not provided at the server level.

· Clients

In this project, the clients provide interfaces so that users can access the distributed file system you have implemented. Each client provides a shell-like interface and it accepts the following user commands.

A user can specify a file or a directory using either absolute path or relative path (relative to the current working directory, which is “/” initially).

- Copy file1 file2: Copy the content of file1 to file2; if file2 exists, it will be overwritten.

- Mkdir dir: Create a directory.

- List dir: List the files and subdirectories under the directory given by dir; if no argument is given, it will list the files and subdirectories under the current working directory.

- Pwd: Show the current working directory.

- Chdir dir: Change the current working directory to dir; if no argument is given, the current working directory will be “/”.

- Delete file/dir: Delete the file or the directory specified by the parameter.

- Exit: Close all the communication sockets and exit from this program.

The commands in the client programs are case-insensitive.

Assignment: Implement the three programs based on TCP/IP.

After the programming is done, test your programs using (at least) the following combinations. The distributed file system you test should include at least four file servers.

(1) All of the programs run on one of the “program” machines (that is, all the programs run on the same machine).

(2) The name server runs on one of the “linprog” machines; half of the file servers run on at least two different “program” machines and the rest of the file servers run on at least two different “linprog” machines.

(3) The name server runs on one of the “program” machines; half of the file servers run on at least two different “program” machines and the rest of the file servers run on at least two different “linprog” machines.

For each combination, you need to run two clients (one on a “linprog” machine and one on a “program” machine). You need to run (at least) a few commands to show the following cases at the minimum.

- Copy a file on one file server to a file on a different file server. Note that there must be at least one copied file whose size is at least 1 mega bytes.

- Delete a file and a directory on a server.

- List the files and subdirectories at a particular directory.

- Create a directory.

- Demonstrate at least three cases where the error handling is involved and errors are handled properly.

Submission:

• Report – You need to turn in a report, describing your implementation of each of the three processes. You need to specify 1) the communication protocols between processes, 2) data structures used, and 3) major steps in each of the programs. You need to analyze the performance of your system and its network and location transparency.
• Source code – You need to attach all the source programs you developed for this programming assignment. Note that you cannot submit other’s work including programs found on the web except those given to you by the instructor.
• Test cases – You need to include your results for the above test cases. Note that the results you submitted must be obtained from the programs generated from your own source code. Any discrepancy will be regarded as cheating.

Hardcopy is required for submission. After submitting your project report, you need to schedule an appointment with the instructor for a demonstration for grading purpose.

Grading:

• Report – 25 points
• Failure to address one required component results in a penalty of 10 points.
• Correct implementation – 45 points.
• Name Server – 10 points.
• Clients – 10 points.
• File Servers – 15 points.
• Efficiency, error handling, and other issues – 10 points.
• Robustness – Each segmentation fault/abnormal termination results in a penalty of 10 points.
• Test cases – 30 points
• Failure to include one required case results in a penalty of 5 points.
• Each case with incorrect outcomes results in a penalty of 5 points.
• Client caching – 10 points
• Recursive listing – 5 points

Extra Credit (15 %): Please state clearly in your report if you have implemented the following extra credit options.

Client Caching To achieve high performance and reduce network traffic, caching is widely used in distributed file systems. Here you need to implement two caching methods on the clients: caches on disk and caches in main memory.

Specifically, when a remote file is accessed the first time, a copy will be created locally and subsequent accesses will use the local copy if it is valid, validity of which is determined by comparing the timestamp of the copy on the server and that of the local copy. You need to demonstrate the new functionality with additional test cases.

Recursive listing: Your client program accepts a new command, called ListR, which lists the files and subdirectories recursively (i.e., list the files and subdirectories under subdirectories recursively). Note the recursion must be implemented at the client side.

Additional Information

Distributed file systems are discussed in Chapter 9 in the textbook and the case studies in Sect. 9.5 may be helpful. Also Chapter 10 in the Tanenbaum and van Steen book (“Distributed Systems: Principles and Paradiagms”) provides case studies of a few distributed file systems.

To keep on schedule and avoid the last minute panic, you need to start the project way in advance.

As a reminder, you should test/run your programs only on “programs” and “linprog” as the programs may involve creating threads/processes. Violations may result in losing some of the privileges you have.