An FTP Server for the Palm Pilot

By: Baruch Toledano, Nigel DregoStudent IDs: 87754615, 22552969

University of California, IrvineECE 145 - Senior Design Project

Professor Pai Chou

Introduction

Are you interested in the Palm Pilot? Have you ever attempted to transfer filesfrom your personal computer to the Palm and back to your computer without using thedarn Hot Sync cable?

We have the solution!!! An FTP server on your Palm Pilot is the solution for you.This server is implemented on your Palm and can be downloaded and used to grab yourfiles from any computer.

Background and Related Work

In order to implement any application on the Palm, a developer must be familiarwith the Palm Application Programming Interface (API). The Code Warrior IDE (http://www.palmos.com/dev/tech/tools/ ) is the common developing tool. This IDEprovides a common interface using the C/C++ languages with additional libraries tosupport a variety of applications, including networking which is essential to the FTPServer implementation.

To help developers, the Code Warrior IDE also comes with a tutorial whichintroduces programmers to programming paradigms( http://www.palmos.com/dev/tech/docs/palmos/CompanionTOC.html ) for the PalmPilot. This document aids in applying efficient programming strategies for memory anddisplay usage. Furthermore, it describes how to use the built-in Palm Libraries fornetworking, serial and infrared access, and internet and messaging. We heavily dependedon this document for understanding the usage of provided functions.

To build an FTP server, one must be familiar with the purpose of FTP, which is ahelpful tool to transfer data or programs from one environment to another. The FileTransfer Protocol is defined by RFC959 ( http://www.cis.ohio-state.edu/htbin/rfc/rfc959.html ) RFC959 defined its purpose asfollows: “The objectives of FTP are 1) to promote sharing of files (computer programsand/or data), 2) to encourage indirect or implicit (via programs) use of remote computers,3) to shield a user from variations in file storage systems among hosts, and 4) to transferdata reliably and efficiently. As one can see, we relied upon this document during thewhole implementation process.

Finally, in order to have some idea of how to structure an FTP Server, webrowsed through some open source FTP server code implemented on the Unix platforms.( http://betaftpd.sourceforge.net/ ) This allowed us to plan out our own structure withspecifics related the Palm Platform. Furthermore, in learning the networking proceduresfor the Palm, we carefully studied and used some code provided by Palm Inc. as samplecode. This code was a sample of various network-oriented client, as opposed to servers,which we set out to implement. ( http://e3.uci.edu/01w/15278/src/NetSample/ )

Project Specification

The goal of this project is to develop an FTP Server that implements a subset ofthe RFC 959 protocol.

Specification

The RFC 959 FTP protocol is a very well defined protocol. An FTP process isdefined by a control connection over which Control Commands are sent between a clientand server Control Process. Control Commands follow ASCII Telnet protocols and aredefined by a 3 or 4 character command word followed by a space followed by optionalarguments and terminated by a carriage return line feed sequence (CRLF). Additionally,commands are broken into six groups: 1. Login 2. Logout 3. Transfer Parameter 4. FileAction 5. Informational 6. Miscellaneous

In addition, an FTP Process must establish a data connection between client andserver in order to achieve any transfer of data. A data connection can be established bythe client or the server, depending on the mode of transfer. This connection isbidirectional and supports both input and output of data.

We plan on supporting the following subset of commands defined by the RFC959 document (note that this a greater subset than the minimum requiredimplementation):

Login Logout File Action TransferParameter

Info. Misc.

USER QUIT PORT STOR PWD NOOPPASS MODE (only Stream) RETRCWD TYPE (ASCII and

Image)LIST

STRU (only File) NLSTDELEABOR

Table 1: Supported FTP Commands

Responses to various commands differ depending on the nature of the command.For example, a PWD command will cause the server process to return the currentworking directory via the control connection. However, a RETR command will cause theserver process to output a file over an established bidirectional data connection.

Platform Specific IssuesWe plan on implementing this FTP Server for the Palm Platform (Palm III - Palm

VII). This means that the only assumption we can make regarding memory is that theuser may have the least possible amount of memory installed on the Palm Pilot (128KB).However, since the System (with installed Networking libraries) uses half of this memory

by itself, we will require that the Palm have more than 128KB of installed memory. Thissaid, we also desire an efficient server that will use the least possible amount of memoryin achieving a working implementation.

In order to achieve an FTP Server that can communicate with the outside world,the Palm Pilot must be able to remotely connect to a network service. This will requirethe use of a Palm Modem in the absence of built-in wireless connectivity.

User Interface

The Palm FTP Server will run as an installed and registered Palm OS Applicationfollowing all guidelines set out by the Palm OS SDK Companion for development of aPalm OS application.

To simplify matters, the user-interface will consist of a command-line (Unixstyle) interface. Initially, FTP Server users will enter commands to start and stop theserver. Server messages will also be displayed on this interface. In the future, users maybe able to type commands which affect configuration options associated with the server.As such, there will be a minimal number of GUI-style elements like buttons and menus inorder to promote efficiency and user-responsiveness.

Project Design

High-Level Design

The design of the FTP server was broken down to several modules. Each modulesupports a different functionality of the design and some support several functions. Ablock diagram of the overall design is below:

Figure 1: High-Level Block Diagram

NetworkingApplications

MemoPad.c

Data/Control

MemoPad.ccmds.ccmds.h

Standard I/O

AppStdIO.cAppStdIO.h

The networking aspect of the Server will be primarily taken care of by theNetworking library provided by the Palm OS SDK. This means that we will simply be auser of networking functions as opposed to developing our own Networking strategy. TheNetLib provided by the Palm OS closely follows the Berkeley Sockets implementation,which we are familiar with.

Data and Control will be implemented as a two-level state machine. One levelwill handle the user control commands and the second level will handle data transfers.

The Standard I/O Module implements the Command-Line user interface and isborrowed (with permission) from Palm’s NetSample code. We find it important to reusecode where it is not critical to the implementation of the server.

Server Architecture

The FTP server is implemented as shown in the following state diagram:

Figure 2: Server State Diagram

MemoPad.c contains the state machine shown above in Runftpd( ) . The statemachine has four basic states:

State OPEN_SOCKET.State LISTEN_CONTROL.State LISTEN_CLIENT.State TRANSFERRING_DATA.

Each one of the states corresponds to a sequence of operations.

State OPEN_SOCKET: In this state we establish the network connections needed forthe server. We open a socket, binding it and listening to a particular port (21, defined byRFC959) for incoming connections. We are using a sequence of supported routinesfound in NetLib within the Palm OS SDK and test for errors along the way. The basicformat of these functions and their arguments is shown here.

socketRef = NetLibSocketOpen (refNum, netSocketAddrINET,netSocketTypeStream, netSocketProtoIPTCP, appNetTimeOut,&error) ;

refNum = Library reference number.NetSocketAddrINET = Specifies that the address is an Internet Address

rather than a process.NetSocketTypeStream = Specifies a TCP Stream connection.NetSocketProtocolPTCP = Specifies that this connection will use the TCP

Transport Layer Protocol.AppNetTimeOut = A timeout on the particular operation. It is

dependent upon the application.error = General error check.

By the end of this state we establish a listening port on which incoming connections mayconnect to the server.

State LISTEN_CONTROL: In this state we accept an incoming connection to theserver. This state is very similar to the LISTEN_CLIENT state in that commands areprocessed in exactly the same manner. However, only Login commands areaccepted/processed in this state in order to authenticate the incoming connection. Thefollowing sequence defines a typical authentication sequence:

Incoming Connection Accepted |V

Welcome Message sent over Control Connection |V

“USER ftp” command received/processed |V

“PASS password” command received/processed |V

Authentication complete

State LISTEN_CLIENT: During this state we perform requests from the client. We goabout parsing the incoming command and responding accordingly. Since commandsequences are fairly short and filenames cannot be longer than 32 characters on the PalmPilot, we limit the incoming receive buffer of the control connection to 128 ASCIIcharacters (bytes). This provides memory efficiency while still providing all necessaryfunctionality.

The basic structure of supported commands is shown here:

3/4 Character Command <SP> <optional arguments> <CRLF>

An example of a command is:

USER <SP> <username> <CRLF>

A reply to a command consists of a 3 digit code followed by a space <SP>followed by one line of text and terminated with the Telnet Carriage Return Line Feedpair. The 3 digit code must follow certain rules as defined by the FTP Protocol.

The implementation of all supported commands is found in cmds.c and follow theRFC959 protocol. A non-familiar command generates an error message to the client.

The server remains in this state while there is no data transfer going on and whilethe user is connected to the server. A data transfer command will move the server fromthis state into the next one, in which handling of all data transfer is done.

State TRANSFERRING_DATA: The server is brought into this state by only fourpossible commands: LIST, NLST, STOR, RETR. All of these functions involve sometransfer of data over an established data connection.

Based on whether the user is wishes to transfer file (raw data) or database(formatted for FTP output as a Palm .prc resource file) data, this state will appropriatelyhandle the manner in which data should be transferred.

A data connection must have been previously established in order to support thetransfer. If such a connection has not been established, the server must attempt toestablish by one by connecting to the client’s default data port (20).

For ASCII type transfers we use an ASCII filter where we parse the file and inserta carriage return character before a new line character. We use a 256 character buffer tostore filtered segments of the file. This is not necessary for Image type transfers as datamust be transferred as-is.

For database transfer we set up a whole state machine. This machine establishes adata connection in the same manner a normal file transfer would, but uses a list of staticvariables in order to maintain current pointers to data being formatted as .prc files. Everydatabase has associated with it a header that describes it. Resource and Record databasesare handled in similar manners, except that Record databases have additional data for

Application specific information. The .prc file is formatted such that for both types ofdatabases, the header is written before the actual data is written.

In this state we need to monitor both the control and data connections. To achievethis we use the NetLibSelect() function which, through the use of socket descriptors, cannotify us of data readiness on any socket we choose to monitor.

Client Connection State

Since the FTP Protocol depends on client/server interaction, it is necessary forboth the client and server to keep track of the current state of interaction. The followingmodel is a general description of this interaction:

------------- |/---------\| || User || -------- ||Interface|<--->| User | |\----^----/| -------- ---------- | | | |/------\| FTP Commands |/----V----\| ||Server|<---------------->| User || || PI || FTP Replies || PI || |\--^---/| |\----^----/| | | | | | | -------- |/--V---\| Data |/----V----\| -------- | File |<--->|Server|<---------------->| User |<--->| File | |System| || DTP || Connection || DTP || |System| -------- |\------/| |\---------/| -------- ---------- -------------

Server-FTP USER-FTP

Figure 3: FTP Use Model

In order to keep track of the client’s state on the server side, we use the followingdata structures:

typedef struct dbSendStateStruct {int state;int resDB; //0 = record db, 1 = resource dbint recNum; //Current record numberint ptrLocked; //Is the pointer locked? Everytime you read raw data

//from a record or resource, a pointer must be//locked.

int releaseHandle; //In case something happens and we need to release a//RESOURCE handle

Handle closeHandle; //Just in case something happens and we need to //close stuffBytePtr srcP; //Pointer to beginning of what we're currently//sendingBytePtr curSrcP; //Pointer to current location of what we're sendinglong remainingBytes; //Number of bytes remaining in the current buffer

//we're sending.DmOpenRef dbP; //Pointer to open database.

} dbSendState;

typedef struct connStruct {NetSocketRef ctrlSocketRef;NetSocketRef dataSocketRef;int state;int cwd; //Current working directory, 0 = Root, 1 =Palm_Home,

//2 = FTP_Homeint transferType; //0 = ASCII mode transfers, 1 = IMAGE mode transfers

//(will be .prc formatted for Palm_Home databases)FileHand fileHandle;char fileName[32];LocalID dbID;int dbCardNo;long remainingBytes; //Apparently ints are 16 bits on this platform, so

//make this a long.dbSendState dbSS; //Keep track of what state of sending a database

//we're in.} conn;

These structures not only keep track of the current client state, but also keep trackof various data-related parameters. This provides accountability and ease-of-use inmonitoring the client and correctly responding to commands.

The associated states that correspond with these structures are defined as follows(and are pretty self-explanatory):

/* Connection State defines */#define NEW_CONNECTION 0#define WAIT_FOR_PASS 1#define WAIT_FOR_CMD 2#define DATA_SOCKET_LISTENING 3#define DATA_SOCKET_OPEN 4#define SENDING_FILE_DATA 5#define SENDING_DB_DATA 6#define RECEIVING_DATA 7#define CONNECTION_CLOSED 8

/* Database Send State defines */#define DB_NEW 0#define DB_WRITE_HEADER 1#define DB_WRITE_HEADER_RES_INFO 2#define DB_WRITE_HEADER_REC_INFO 3#define DB_WRITE_APPINFO 4#define DB_WRITE_SORTINFO 5#define DB_FILLER_WORD 6#define DB_WRITE_RESOURCES 7#define DB_WRITE_RECORDS 8#define DB_DONE 9#define DB_CLEANUP 10

FTP Directory Structure

The general structure of directories from the root, as we implemented on the Palmare Palm_Home directory and FTP_Home directory. Palm_Home will list all databasesaccessible by the Database find functions and will be transferred as .prc formatted files.There is no upload support to this directory to protect user data. FTP_Home is acollection of files created by the server on upload. These files are created on upload usingthe File Streaming API provided by the Palm OS SDK. In a similar manner, when aclient wishes to download these files, the File Streaming functions are once again used totransfer the file.

These two directories share some common ground. When one looks at a directorylisting, he or she can observe the names of the files/databases, the date of creation and thesize of the file.

Error Handling

In all states we carry a general error routine which checks for errors along theway. Typical errors could be associated with establishing the data and controlconnection, transfer of data and others. The error numbers are associated with theRFC959 protocol and the type of error that has occurred.

In rare cases, when an error forces the server into an unknown or hazardous state,the data and/or control connections may be shutdown by the server process. The serverwill then “reset” itself by attempting to listen for a new incoming control connection.

Development Plan

Environment

We decided to use CodeWarrior and the C Language since all of the networkingfunctions are in C. The Berkeley Sockets API, which is followed by the Palm OS NetLibis also in C, so this provided an easy starting point. Furthermore, the Palm OS Tutorialuses C. This allowed us to easily setup the basics of the application and how interactswith the Palm OS.

CodeWarrior can be downloaded from the Palm OS Developer Website (http://www.palmos.com/dev/tech/tools/cw/ ). In addition to CodeWarrior, one must alsohave a Palm Emulator (POSE) and ROM images which allow debugging through theCodeWarrior IDE. The ROM Images are also found on the Developer Website at Palmbut one must be registered as a developer (which can be done free-of-charge).

Code and Version Control

The actual implementation was a combined effort. For example, Nigelimplemented the GUI interface while Baruch began implementation of handling FTPCommands.

At the beginning of the project code was written separately by both Nigel andBaruch and merged when we were able to meet. While we thought of trying to use aCVS server for source control, we didn’t really have access to any public CVS servers.After we couldn’t easily post our code on the web, this changed. Professor Chou gaveNigel access to his CVS server. This meant that we could both work on the same file atthe same time. However, since only Nigel had access to the server, Baruch would stillhave to send his code to Nigel for upload to the CVS server.

The following documents who wrote particular sections of code:

Nigel BaruchUser-Interface (using AppStdIO fromNetSample)

Sample program to test NetLib networkingfunctions and display of text onscreen( DoTest() )

Initial Server Socket Operations (makinga socket listen to the client port)

FTP Authentication Commands

Server State Machine Server State MachineFTP Directory Listing Commands FTP Transfer Parameter CommandsCommand Parsing Command ParsingDatabase Transfer File Transfer

Cross-Development

Since neither of us has a Palm Pilot, we used the emulator to do most of ourdevelopment and debugging. We used the Windows Palm OS Emulator to do ouremulation. As we use Networking functions, it was important to map all networkingfunctions to the Host TCP/IP Stack.

After we had completed much of the codebase and had a fairly stable and workingFTP Server implementation, we borrowed a Palm Pilot and Palm Modem from ProfessorChou to do more extensive testing on a native platform.

The only other platform we used was the Tea ( http://www.devweb.pt/tea )scripting language for Java. We used this in our initial test application to test the NetLibfunctions. As Baruch wrote the sample application for the Palm, Nigel wrote a Tea scriptto connect to the Palm Pilot and transfer simple lines of text for display on the Palmscreen.

Testing

Testing the application was an on-going process. In order to validate our progressand correct functionality, we tested along the way using a simple (any) ftp client andobserved the behavior of the Server on the Emulator. Furthermore, we also verifiedcorrect FTP behavior and response on the Client side using a client (CuteFTP,http://www.globalscape.com ) which displayed both FTP requests and responses. Inorder to verify breadth and stability of operation, more than one FTP client was used.

We downloaded the application .prc into the emulator to test it. Nigel also testedwith some friends all over the country who connected to the FTP server. The last phaseof testing and debugging was done over a real palm. Thanks to Prof. Chou we borroweda Palm Pilot and a modem and connected to a PPP server to test the FTP Server on a realPalm Pilot.

A successful result was dependent upon the test phase and objective. Early on, wetested the correct operation of various non-data transfer commands, the establishment ofconnections (Data and Control) and the ability to communicate using those connections.In further phases, it was sufficient to test with the emulator while observing correctresponses and directory listings. We also tested different types of files, ASCII, ImageFiles, individually to observe the data integrity when we transfer to and from the Palm.Later on, we also tested Database transfers in the same manner as file transfers. Lastly,we did all this on a real Palm Pilot to verify that the Server indeed ran on a real platformas well.

The schedule for testing was as follows:

Week #1-2 Test the GUI, Emulator and Developing environmentWeek #3 Display text on the Palm Pilot using Networking functionsWeek #4 Test for interaction with an FTP ClientWeek #5 Preliminary Authentication CommandsWeek #6 Transfer Parameter CommandsWeek #7 ASCII Parse/Transfer, DELE, ABOR and STOR commandsWeek #8 RETR command, Image and Database transferWeek #9 Testing on a Palm connected to a Server with a modem and client.

Check limit on number of transfersWeek #10 Final testing to ensure proper demo operation for Presentation

Project Evaluation

In order to evaluate the FTP Server we look at the results and basic functionality.

Qualitative Evaluation

The specification became more clear as we progressed in the design. Like everygood divide and conquer approach, we went about implementing the server with thislimited time. The overall difficulty were in many aspects of the design, such asnetworking connection, usage of the Code Warrior environment and in particular thetransfer of the databases. These objectives were all met to a sufficient degree. Once webuilt a general state machine and understood the RFC959 protocol we followed it andexpanded the design to its details.

Future developments could include testing and proper implementation of thePASV command to allow clients behind firewalls and such to properly connect to theserver and interact with it. In addition, we would love to see this Server running on awireless platform, like the Palm VII. However, the Networking functions for this areslightly different and would require appropriate modification of the application. Lastly, amore general directory structure as well as the ability to handle multiple users wouldmake the server more marketable.

The RFC959 Protocol is inherently robust if implemented correctly, due to itsabundant error handling. However, due to the memory limitations of the Palm Pilot, suchrobustness could not always be achieved. For example, there are limitations of size ofclient command requests, filenames, and passwords. Therefore, it is unknown what mighthappen when these limitations are exceeded. Further testing and more time would allowus to determine the consequences. In addition, the TCP/IP Protocol as implemented bythe Palm OS NetLib introduces certain limitations like the inability to send packetsbigger than 32KB. Appropriate checks were needed to overcome this limitation.

Quantitative Evaluation

The overall performance was very satisfactory. The transfer of files anddatabases worked correctly a majority of the time. While data transfers work perfectly,errors occur after downloading more than 10-12 files in succession. This is likely causedby a stack overflow due to a memory leak. Further debugging and testing of the code,which is not allowed by the available time, is needed to correct this flaw.

Currently, there is debugging work being done on file transfers as some clients donot receive the final few KiloBytes of memory. The FTP Server itself reports that it sentout the whole file, but it is quite probable that the data connection may be closed toquickly following the end of file and clients may not have enough time to receive thefinal parts of the file. Work will continue to correct this problem.

By the time we were performing physical testing we were able to transfer all typesof data in this protocol. The size of the server to download is considerably small at amemory footprint of 28KB. Dynamic memory usage is not more than 10KB but a largepart is determined by the number of files on the system which affects the size of thedirectory listing.

The transfer size is limited to the amount of memory installed on the Palm, andhopefully will be expanded over time. Currently, there appears to be no limits on numberof files in the system other than overall memory usage on the Palm Pilot.

Another area of improvement for the future is the user-responsiveness. While wehave achieved user-responsiveness, meaning that the Palm user can quit the program ortype other commands while the server is running, it is slow at best. We would like to seethis be real-time responsive and allow the user to configure the server as he would like.For an initial implementation, it is good enough to have just a working server, but futureversions should allow some user configuration.

Concluding Remarks

The Palm FTP Server was an extremely interesting and engrossing project. Toallow something as small in size and memory as a Palm Pilot to run a professional webserver based on a standardized protocol is always a challenge. Nevertheless, we were ableto accomplish it to some degree.

We would love to see a completely RFC959 compliant version in the future, withvirtually no limitations other than memory of the Palm. Furthermore, extending theserver to a wireless platform would make for an even more portable and power serverapplication.

In addition, the server state machine is cumbersome in its current application.Cleaning up this code would make it more efficient, allow for better implementation of aserver that would support multiple clients, and respond to Palm users better.

Credits

Baruch Toledano - Code contributions shown above. Writeup Sections: Introduction,Background and Related Work, Project Specification and some of Project Design.

Nigel Drego - Code contributions shown above. Writeup Sections: some of ProjectDesign, Development Plan, Evaluation and Concluding Remarks, with somemodifications to Baruch’s sections.

AppStdIO Code was used from the NetSample application written by Ron Marianetti ofPalm Inc.

Ideas for FTP Server architecture taken from BetaFTPD Source Code, written bysgunderson@bigfoot.com

Bibliography

1. RFC 959. J. Postel and J. Reynolds of the Network Working Group, October1985, <http://www.cis.ohio-state.edu/htbin/rfc/rfc959.html>.

2. Palm OS SDK Companion. Palm Inc.,<http://www.palmos.com/dev/tech/docs/palmos/CompanionTOC.html>.