How to save during real-time collaboration

I want multiple users to edit same document. Problem I’m facing is when a new user joins, he might see an outdated document. How do I make sure that new users get most recent changes?

Some solutions I thought of:

Save on every change. I don’t like this solution because it will slow things down on UI and put load on db.
When new user joins, trigger save on all other clients. After other clients saved, load document. With this there can be inconsistency still.

Any other suggestions would be helpful.

UPDATE: After looking into suggested solution, Google Realtime API, I found out that:

Users of your app must have Google Drive and give you access to their drive. This could at best present awkward UI flow or prevent users who don’t have Google Drive from using realtime feature.
All the sharing settings that are done on your side, have to be replicated for Google document.

UPDATE 2: To acoomplish the goal, I went with Google’s Firebase

Google Drive

If you’re trying to make your own version of google docs, I suggest you take a look at the Google Realtime API. Google recently released this with the intent of allowing other developers to use the same tools they did to allow for realtime collaboration. This would allow you to save time on your development and get a working product sooner.

You could easily take the data that is in the document and push it into your database at regular intervals, or have the database itself be a ‘participant’ of the exchange, simply listening to and logging all the changes. It also allows for a user to define their own data structures which are then usable in the realtime API, so you are free to extend it as you see fit.

Non-Google Drive

So according to your research, Google Drive isn’t an option. That’s fine, but it’s going to be tougher and possibly not work as well, depending on how much you put into it.

Here’s a general strategy I would use to accomplish this problem:

Have the server be the communication multiplexer. Each person talks to the server, and the server sends out that information to everyone else. This way the server always has the most up to date view of the document.
Find a third party algorithm/module for conflict resolution. Conflict resolution is tough, and is something that still isn’t perfect. Doing this alone could easily increase the scope of the project to be far too large. If you can’t use a third party algorithm, I would suggest that you only allow one user to edit an area of a time, so that the user must obtain a lock before editing an area, or you risk destroying another users work, which will get very old, very fast.
When a new user joins, give them the most recent document and automatically start streaming the commands to them. The server has the most recent view and thus can dish it out automatically.
Backup to the database at certain intervals. Decide how often you want to back up (every 5 minutes or maybe every 50 changes.) This allows you to maintain the backup you desire.

Problems: This isn’t a perfect solution, so here are some issues you might face.

Throughput of the server could bottleneck performance
Too many people reading/writing could overload the server
People may become out of sync if a message is lost, so you may want to make sure you synchronize at regular points. This means sending out the whole message again, which can be costly, but otherwise people might not have the same document and not know it.

I’d recommend 1 persistent copy of the document on the server. When a client connects to the server you issue an UPDATE command(s) to that client with all the changes.

Update WorkFlow

User causes triggering change -> Client sends UPDATE to Server -> Server sends UPDATE to Clients

Viable triggers

User clicks Save
User completes a specific task
- Finishes editing a cell
- Finishes editing a sentence/paragraph/line
User clicks Undo
User presses Return key
User types a key (save on every change)

Update Implementation

I would suggest being able to re-create the document with a series of UPDATE commands so that the server stores each UPDATE and when a new client connects the client can be sent the series of updates and it itself can re-create the document to display to the user. Also, you could alternatively have a SAVE command that is separate and have UPDATE be temporary changes that can be used for UNDO requests and have SAVE actually store it to be re-opened if the server is closed or all clients disconnect.

1) Have a look at Knockout.js

It follows an MVVM pattern and will automatically push out notifications to the View based upon changes to the Model. For instance, look into their observable array to provide a little more information on how they do that.

2) Mix that in with SignalR and you should now have the ability to send out notifications to other users working on the document. From their site:

SignalR also provides a very simple, high-level API for doing server to client RPC (call JavaScript functions in your clients’ browsers from server-side .NET code) in your ASP.NET application, as well as adding useful hooks for connection management, e.g. connect/disconnect events, grouping connections, authorization.

So you’ll need to have some hooks at your model level within Knockout.js to make some SignalR calls whenever a change occurs. The other clients will receive the notice from SignalR and then trigger a corresponding change in their copy of the Model, which will push back up to their View.

It’s an interesting combination of the two frameworks, and you should be able to search and gather more information to handle the particulars.

For example, this codeproject example specifically addresses Co Working UIs and Continuous Clients which seems to be exactly what you’re trying to do.

New age web applications may need to offer new age user experiences – and should handle co-working and continuous client scenarios properly. This involves ensuring that the user interface is syncing properly itself across devices and across users to ensure the state of the application and user interface is maintained “as is”.

This blog post looks to be an entry point into a series of blog posts discussing the use of the two packages and contrasts that with a traditional ASP.NET approach. May provide some points for consideration while you’re designing your site.

This blog post appears to be a little bit more basic and provides the groundwork for combining the two packages.

Disclosure: I’m not affiliated with any of the above links, nor have I really dug into their content to see how sound or correct it is.

The solution is Operational Transformation (OT). If you haven’t heard of it, OT is a class of algorithms that do multi-site realtime concurrency. OT is like realtime git. It works with any amount of lag (from zero to an extended holiday). It lets users make live, concurrent edits with low bandwidth. OT gives you eventual consistency between multiple users without retries, without errors and without any data being overwritten.

But implementing OT is a difficult task and time consuming. So you might want to use an external library like http://sharejs.org/.

It mainly depends on the type of your documents and how your users collaborate.

However, I would:

let all clients send unsaved changes to the server every once in a while (depends on how users work with the documents).
the server stores the deltas in the user’s session (even for a fat client you need something like a session)
other clients editing/viewing the same document get those temporary changes or at least a hint that there might be so.

Advantages:

no DB updates unless someone clicks ‘save’
backup for the case the client crashes (for the session-period)
your server decides how and what data to forward to which client (e.g. you can jump start the feature with just a note and later implement a more sophisticated merge and highlight)

Disadvantages:

not ‘real-time’ – e.g. you send every 30 secs, but someone types 3 sentences in that time.
more network traffic – dependent on your documents and collaboration
possibly large sessions
possibly high computation effort if many users collaborate and do many changes

Essentially, what you are asking is how to deal with shared mutable state. Saving is the easy part; but how do you deal with multiple people editing the same thing at the same time? You want all users to be viewing the same document while synchronizing simultaneous edits, all in real-time.

As you have probably gathered, it’s a hard problem! There are a few pragmatic solutions:

Modify your application requirements to not allow true simultaneous editing. Edits can be merged like with source control systems, with results broadcasted to each client. You could build this yourself but it’d be a poorer user experience.
Outsource the synchronization of state mutations to an open-source solution that integrates with your existing technology. ShareDB is the current leader in this space. It is based on Operational Transformation and used in at least one production system. This will take care of the saving issue you’re concerned with but won’t help with any of the additional UX features mandatory for any collaborative application.
Use an off-the-shelf platform such as Convergence (disclaimer: I am a founder) to handle all the difficult bits for you. You’ll also get additional tools for real-time collaboration such as cursor/mouse tracking, selections, and chat to build out a superior collaborative experience quickly. See this question for a good roundup of all the existing tools out there.

Filed under: softwareengineering - @ 01:23

Thẻ: problem-solving, real-time

How to save during real-time collaboration

I want multiple users to edit same document. Problem I’m facing is when a new user joins, he might see an outdated document. How do I make sure that new users get most recent changes?

Some solutions I thought of:

Save on every change. I don’t like this solution because it will slow things down on UI and put load on db.
When new user joins, trigger save on all other clients. After other clients saved, load document. With this there can be inconsistency still.

Any other suggestions would be helpful.

UPDATE: After looking into suggested solution, Google Realtime API, I found out that:

Users of your app must have Google Drive and give you access to their drive. This could at best present awkward UI flow or prevent users who don’t have Google Drive from using realtime feature.
All the sharing settings that are done on your side, have to be replicated for Google document.

UPDATE 2: To acoomplish the goal, I went with Google’s Firebase

Google Drive

Non-Google Drive

So according to your research, Google Drive isn’t an option. That’s fine, but it’s going to be tougher and possibly not work as well, depending on how much you put into it.

Here’s a general strategy I would use to accomplish this problem:

Have the server be the communication multiplexer. Each person talks to the server, and the server sends out that information to everyone else. This way the server always has the most up to date view of the document.
Find a third party algorithm/module for conflict resolution. Conflict resolution is tough, and is something that still isn’t perfect. Doing this alone could easily increase the scope of the project to be far too large. If you can’t use a third party algorithm, I would suggest that you only allow one user to edit an area of a time, so that the user must obtain a lock before editing an area, or you risk destroying another users work, which will get very old, very fast.
When a new user joins, give them the most recent document and automatically start streaming the commands to them. The server has the most recent view and thus can dish it out automatically.
Backup to the database at certain intervals. Decide how often you want to back up (every 5 minutes or maybe every 50 changes.) This allows you to maintain the backup you desire.

Problems: This isn’t a perfect solution, so here are some issues you might face.

Throughput of the server could bottleneck performance
Too many people reading/writing could overload the server
People may become out of sync if a message is lost, so you may want to make sure you synchronize at regular points. This means sending out the whole message again, which can be costly, but otherwise people might not have the same document and not know it.

I’d recommend 1 persistent copy of the document on the server. When a client connects to the server you issue an UPDATE command(s) to that client with all the changes.

Update WorkFlow

User causes triggering change -> Client sends UPDATE to Server -> Server sends UPDATE to Clients

Viable triggers

User clicks Save
User completes a specific task
- Finishes editing a cell
- Finishes editing a sentence/paragraph/line
User clicks Undo
User presses Return key
User types a key (save on every change)

Update Implementation

1) Have a look at Knockout.js

2) Mix that in with SignalR and you should now have the ability to send out notifications to other users working on the document. From their site:

SignalR also provides a very simple, high-level API for doing server to client RPC (call JavaScript functions in your clients’ browsers from server-side .NET code) in your ASP.NET application, as well as adding useful hooks for connection management, e.g. connect/disconnect events, grouping connections, authorization.

It’s an interesting combination of the two frameworks, and you should be able to search and gather more information to handle the particulars.

For example, this codeproject example specifically addresses Co Working UIs and Continuous Clients which seems to be exactly what you’re trying to do.

New age web applications may need to offer new age user experiences – and should handle co-working and continuous client scenarios properly. This involves ensuring that the user interface is syncing properly itself across devices and across users to ensure the state of the application and user interface is maintained “as is”.

This blog post appears to be a little bit more basic and provides the groundwork for combining the two packages.

Disclosure: I’m not affiliated with any of the above links, nor have I really dug into their content to see how sound or correct it is.

But implementing OT is a difficult task and time consuming. So you might want to use an external library like http://sharejs.org/.

It mainly depends on the type of your documents and how your users collaborate.

However, I would:

let all clients send unsaved changes to the server every once in a while (depends on how users work with the documents).
the server stores the deltas in the user’s session (even for a fat client you need something like a session)
other clients editing/viewing the same document get those temporary changes or at least a hint that there might be so.

Advantages:

no DB updates unless someone clicks ‘save’
backup for the case the client crashes (for the session-period)
your server decides how and what data to forward to which client (e.g. you can jump start the feature with just a note and later implement a more sophisticated merge and highlight)

Disadvantages:

not ‘real-time’ – e.g. you send every 30 secs, but someone types 3 sentences in that time.
more network traffic – dependent on your documents and collaboration
possibly large sessions
possibly high computation effort if many users collaborate and do many changes

As you have probably gathered, it’s a hard problem! There are a few pragmatic solutions:

Modify your application requirements to not allow true simultaneous editing. Edits can be merged like with source control systems, with results broadcasted to each client. You could build this yourself but it’d be a poorer user experience.
Outsource the synchronization of state mutations to an open-source solution that integrates with your existing technology. ShareDB is the current leader in this space. It is based on Operational Transformation and used in at least one production system. This will take care of the saving issue you’re concerned with but won’t help with any of the additional UX features mandatory for any collaborative application.
Use an off-the-shelf platform such as Convergence (disclaimer: I am a founder) to handle all the difficult bits for you. You’ll also get additional tools for real-time collaboration such as cursor/mouse tracking, selections, and chat to build out a superior collaborative experience quickly. See this question for a good roundup of all the existing tools out there.

Filed under: softwareengineering - @ 01:23

Thẻ: problem-solving, real-time

Thiết kế website giá rẻ

Danh mục

How to save during real-time collaboration

Google Drive

Non-Google Drive

Update WorkFlow

Viable triggers

Update Implementation

How to save during real-time collaboration

Google Drive

Non-Google Drive

Update WorkFlow

Viable triggers

Update Implementation