Friday 30 March 2007

MySource 4.0 WebDAV ideas

I've been thinking about how to implement some new features and improvements into the MySource 4.0 WebDAV server. All these changes would go in after the alpha release and user feedback, but I thought I'd try and list them in one place so I don't forget about them.

Adding new files and folders
The problem with adding (and deleting) files and folders is actually editing. That sounds a bit weird, so let me explain.

When you edit files, the application or file system decides on the process it will use to save the file. Most applications don't make things easy by using a simple LOCK then PUT (a new version) then UNLOCK. Most applications I test with move the current file and create temp files and directories before finally renaming a temp file to replace the existing file we are editing. This means that I get a lot of DELETE, MOVE, PUT and MKCOL requests coming through. If I actually processed all of these requests in the CMS, assets would be moving all over the place, and often being replaced by new assets with the same name while they end up in the trash.

To solve this problem, I don't action any of these request when asked. Instead, I keep a record of where files and folders should be. This allows me to pretend that files have been moved, created or deleted without having to actually do it. Then when a temp file is moved to the location of an existing file, I update the file's contents in the CMS.

That is a lot of background to basically say that I can never trust a PUT request. There is no way of knowing if it is a real request to create a new file or if it is just an application trying to save an existing file. Forget sessions and cookies to store information about what is going on. They are not supported in WebDAV.

So here is my idea. When someone drags a file into a folder, I pretend that it exists. I don't currently show that file in the folder, but I could. If I do show the file, I can also allow it to be edited. For users of WebDAV, it would appear that the file has been created in the CMS, but it really only exists in the WebDAV database. To commit the file to the CMS, a user could go to some location in MySource 4.0 and indicate that a pending WebDAV file should be turned into a real File asset.

Deleting existing files and folders
For the same reason as adding files and folders is a problem, deleting is a problem as well. There is no way to tell if the user deleted the file or if the application deleted it while saving.

In the same way that added files can be shown, deleted files can be hidden. In fact, I already do hide them because it causes OS X's Finder to go crazy if it thinks a file should be deleted but it is still there (fair enough I guess). Again, a user could go to a location in MySource 4.0 and indicate that files marked as deleted should be moved to the trash within the CMS.

Authentication and permissions
Forcing a user to authenticate before connecting to the WebDAV server is dead easy. However, you do need to re-authenticate every time a request is made because sessions (cookies) are not supported. That isn't a problem, but it is something to remember when this functionality is added.

Similarly, checking permissions on assets before displaying them is not going to be hard. Where I do see things getting a little interesting is the display of projects and project folders. They don't have any specific permissions applied to them, so who do we show them too? We could say that any user can see the projects in which their account exists, or we could say that they see projects where they can read at least one item in the project. The first method is quick, the second is slow. Unsurprisingly, I'm leaning towards the first option.

Temporary items on OS X
I noticed, completely by accident, that if I told Word on OS X that the Temporary Items folder didn't exist, it would save the file using less requests. The figures are something like 30 requests versus 22. Instead of moving temporary files around, it would just create backup copies in the same location as the existing file.

I do have support for the Temporary Items folder in the WebDAV server, but I'm wondering if taking out that support might actually improve performance on OS X. Word on Windows doesn't suffer from the same problem as it choses to save files differently (of course).

I'd like to do some testing to see if this could be a possible improvement to the WebDAV server.

Tuesday 27 March 2007

MySource 4.0 WebDAV complete

Well, complete enough for the alpha release anyway. There is no authentication or permission checking in there, and you cant add and remove files and folders. I didn't leave these features out because they were hard, but because I don't want to finish planing this server without some feedback from users first. I have some ideas about how I want those features to work, so I'm interested to see if they go down well.

I'll write another post with my ideas in it later. This one is all about the fact that the server finally works. I didn't have to change any of the functionality to get it working on Windows, but it has been quite a wait to test it.

The problems I had were interesting.

First, I was using Lighttpd as the web server. Everything worked fine on OS X, but when saving a Word document on Windows, Lighttpd would receive a very specific GET request from Word and drop it. That is, it would return a 400 (bad request) error without ever asking by script to process the request. This caused Word to always open the file as read-only. There was nothing I could do about this error. I couldn't get Word to change the request and I couldn't get Lighttpd to accept it.

The server admins then set me up with Apache running the php5-cgi module (our Apache already runs mod-php4, so we couldn't just use mod-php5). Everything was working great with my existing saved connections until I tried creating a new connection for testing. The very first OPTIONS request that gets sent to see what version of the WebDAV protocol is being used was being handled by Apache without asking my script to process the request. Apparently, there was a bug in Apache where it handled the OPTIONS request for all CGI scripts. The version of Apache on our dev box could not be updated for a few weeks, so we had to find another way.

The solution turned out to be FastCGI on Apache. Every request made it through to my script and saving/browsing was working on both OS X and Windows XP.

It's been a long road, but the functionality this server provides will be a fantastic addition to MySource 4.0. I'm looking forward to trying it out on some real data when the alpha is released.

Wednesday 21 March 2007

Easier custom standards with PHP_CodeSniffer

I committed a fairly large change to PHP_CodeSniffer today that allows you to create custom coding standards without having to write your own sniffs.

The MySource 4.0 team needed a new code sniff to check that we were including files correctly, but the sniff was very specific to the product, so I could not include it in the Squiz coding standard. I needed the team to use a new MySource4 coding standard to check their code, but I wanted it to be the Squiz coding standard with a few more sniffs.

The only way to do this previously was to create empty sniffs files for every Squiz sniff that I wanted to include. This meant a lot of directories and a lot of maintenance as I would have to update the MySource4 standard every time I added or removed sniffs from the Squiz standard.

So I decided to give coding standards their own management class and allow them to override a method in there to tell PHP_CodeSniffer which sniffs, directories of sniffs, or whole coding standards they would like to include in addition to their own. This not only allowed me to create the MySource4 standard easily, but also allowed me to remove some empty sniff files from both the Squiz and PEAR coding standards.

I haven't update the end user docs for PHP_CodeSniffer yet, but I will get around to it before the next release. All existing coding standards will need this management class, so you'll need to be aware of this if you do have your own custom standard.

Saturday 10 March 2007

WedDAV test server

While trying to work out why I couldn't save Microsoft Word documents from OS X, it occurred to me that a good way to get it working was to look at what other WebDAV servers do and try to copy them. I used this technique with Ethereal and a Windows Sever 2003 box to get my prototype server working earlier in the year. The only problem this time was that Microsoft Word on OS X couldn't save files on a Windows Server 2003 WebDAV share running under IIS, and that's a big problem for me.

I'm not sure why saving doesn't work, but I tried it on two different Macs using two different versions of Microsoft Word and both failed. On a Windows XP machine, saving works fine. I needed to find another WebDAV server to use for testing, or configure one myself. My limited access to server hardware meant that I was going to have to go with the first option.

Enter the WebDAV Testing Server. This is a free public WebDAV server that runs Apache with mod_dav to provide WebDAV functionality. It's primary goal is to provide a test server for developers writing WebDAV clients, but it also happens to come in handy when writing your own server.

Luckily for me, this test server can read and write Microsoft Word documents on OS X just fine, so I used Ethereal to compare the requests and responses and I can now (finally) save Microsoft Word docs from OS X into MySource 4.0. As an added bonus, my problems with saving from Photoshop have gone away as well. One of the changes made today must have caused it to start working, so things are looking up.

Saturday 3 March 2007

MySource 4.0 WebDAV server started

I've been waiting about a month for MySource 4.0 development to reach a point where it made sense to begin porting my stand-alone WebDAV server to MySource 4.0. This week, it finally caught-up.

Currently, my WebDav server can display the projects in your system as folders. Three project folders are listed under each; Documents, Images and Movies. The server currently only supports the viewing of Folder and File assets under these folders, but the mime type of each File asset is determined, so they appear as different file types in the OS.

I've had to put the ID of the asset in the path for now (which means it also appears in the folder/file name) because we don't have URLs assigned to project folders yet. For example, a folder name would be something like [23] Holiday Photos. Once these URLs go in, I'll be able to use the web path of the asset instead of the ID. For example, a folder name would be something like holiday_photos. I'm thinking of making this an option though, because some people may prefer seeing the name rather than the web path, even if the name does have the asset ID on the front. It's also one less query in most cases, so that is a bonus too. It's just a lot harder to remember a URL with /[23]%20Holiday%20Photos/ in it than /holiday_photos/ if you want to navigate to that folder directly.

The next thing to tackle for WebDAV is the editing of files, which includes working out a permanent solution for locking. Supporting the creation of new files and folders would be the next logical step, before finally moving the server into MySource 4.0 and converting it to a System with Actions and Channels.