ChiliProject is not maintained anymore. Please be advised that there will be no more updates.

We do not recommend that you setup new ChiliProject instances and we urge all existing users to migrate their data to a maintained system, e.g. Redmine. We will provide a migration script later. In the meantime, you can use the instructions by Christian Daehn.

[Proposal] Replace awesome_nested_set with ancestry

Added by Holger Just at 2011-04-18 06:31 pm

Currently, ChiliProject (and Redmine) use awesome_nested_set for persisting object trees into the database. It works by implementing the nested set model plus parent pointers. While this library seems to work, it has some major downsides.

The worst here is that it's not transactionally save under the default isolation level "Read comitted" of MySQL and PostgreSQL. The issue here is that any updates of the tree such as inserting an element require multiple queries which are prone to the non-repeatable reads phenomenon. Thus, if two or more users modify the same tree at the same time, inconsistencies can occur, which has already been observed for Redmine. It can even be triggered by quickly clicking two times on the "add issue" button with a parent issue set, depending on the speed of the deployed system. Although the tree can be repaired by evaluating the parent pointers, a decent tree implementation should never allow the tree to become corrupt.

Another downsides is that each insert and delete operation requires an update of all parent elements and all elements "to the right" of the affected element. Writes are thus rather slow, increasingly so with large trees.

Thus, I propose to replace the awesome_nested_set plugin with ancestry. It represents the tree using a materialized path, which basically is a slash-separated string containing the list of parent pointers to the root. This has the following implications:

  • Ancstry's model API is about the same as acts_as_nested_set's so there shouldn't be much visible change in the models.
  • Inserts and deletes are atomic operations and thus transactionally save. The only possible issue would be that there could still be new children created as old child elements are already scheduled for recursive destroy (if used). But this an issue of all callback based operations in Rails. From what I know, recursive destroy isn't even used in ChiliProject at all.
  • Read operations on trees are slightly less efficient as ancestry uses LIKE expressions to work on the tree. Its efficiency depends on the database's ability to use its indexes for these operations. As we typically have rather small trees in ChiliProject and these trees are often read in whole, this probably isn't much of a problem at all.
  • acts_as_nested_set provides an arbitrary order of elements on one level of the tree. This order is not available with ancestry. But as it isn't used anywhere and I couldn't come up with an actual use-case for that, even after thinking hard, so I guess this is a non-issue.

Replies (10)

RE: [Proposal] Replace awesome_nested_set with ancestry - Added by Eric Davis at 2011-04-21 12:05 am

Quick post of my thoughts:

  • Is there a list of how the public APIs differ? Would we need to wrap anything? There is a bit of third party code out that based on ANS already.
  • "As we typically have rather small trees in ChiliProject and these trees are often read in whole, this probably isn't much of a problem at all." - What about issues? That can be a rather large tree.
  • If I recall ANS is also unmaintained or the developers are looking for a new maintainer for it. We should check ancestry's history too to make sure it will be around for some time.
  • I'm assuming this would break db schema and query compatibility with Redmine correct?
  • Are there any other benefits other than the "consistent tree"? (it's a big benefit, just seeing what else we would get)

I'll have to check some plugins. I think I have one or two that uses (and monkey patches) ANS so I'd like to see how they would need to be changed.

Eric Davis

RE: [Proposal] Replace awesome_nested_set with ancestry - Added by Felix Schäfer at 2011-04-21 06:47 am

Eric Davis wrote:

Quick post of my thoughts:

And mine :-)

  • Is there a list of how the public APIs differ? Would we need to wrap anything? There is a bit of third party code out that based on ANS already.

The list of API diffs would be my condition too.

  • "As we typically have rather small trees in ChiliProject and these trees are often read in whole, this probably isn't much of a problem at all." - What about issues? That can be a rather large tree.

Each issue and its subissues are single trees, not all issues are together in one big tree, so I second the opinion that we "typically have rather small trees".

  • I'm assuming this would break db schema and query compatibility with Redmine correct?

Yes, but we're getting there with journalized in 2.0 anyway, so I personally don't care much.

  • Are there any other benefits other than the "consistent tree"? (it's a big benefit, just seeing what else we would get)

From a quick look at the library and how it works, I'd say one benefit would be that you could get subtree ordering (partly) from the DB. Currently, if you fetch an issue and it's tree, if you want to get the tree structure out of the DB, you can't get them ordered in any way because you get them out of the DB in the order they were inserted into it. From what I see, ancestry can order subtrees, so you could say "I want this issue, it's tree, and please order each level by start_date". I'm not sure it can do it recursively for every level, but at least it seems to be able to do it for one subtree. (Note that this feature uses ordered hashes though, so it would work on 1.9 and up only according to the README of the lib).

RE: [Proposal] Replace awesome_nested_set with ancestry - Added by Felix Schäfer at 2011-04-21 06:49 am

Oh, and if we can tell the library to use a custom attribute for the names it gives the elements in its internal representation, you get a name-based wiki tree "for free" ;-)

RE: [Proposal] Replace awesome_nested_set with ancestry - Added by Eric Davis at 2011-04-21 08:01 pm

Felix Schäfer wrote:

  • I'm assuming this would break db schema and query compatibility with Redmine correct?

Yes, but we're getting there with journalized in 2.0 anyway, so I personally don't care much.

I have the same concern with journalized but there are a lot more benefits with journalized. :)

(Note that this feature uses ordered hashes though, so it would work on 1.9 and up only according to the README of the lib).

I thought Rails has an OrderedHash that it subs in for Hash in a few places.

Oh, and if we can tell the library to use a custom attribute for the names it gives the elements in its internal representation, you get a name-based wiki tree "for free" ;-)

Maybe, I'd just be afraid of having to sanitize the page names enough for ancestry. Might work though.

Eric Davis

RE: [Proposal] Replace awesome_nested_set with ancestry - Added by Felix Schäfer at 2011-04-22 01:28 pm

Eric Davis wrote:

Felix Schäfer wrote:

(Note that this feature uses ordered hashes though, so it would work on 1.9 and up only according to the README of the lib).

I thought Rails has an OrderedHash that it subs in for Hash in a few places.

I thought too that Rails has OrderedHashes, but it seems the plugin doesn't use them, and I'm not sure It'd be worth it to hack it in.

RE: [Proposal] Replace awesome_nested_set with ancestry - Added by Felix Schäfer at 2011-08-21 11:21 am

I've queried the maintainer of acestry to see if we could get non-primary-key ancestry keys, which would mean we could get path-based subpages almost for free. See https://github.com/stefankroes/ancestry/issues/68 for the issue.

RE: [Proposal] Replace awesome_nested_set with ancestry - Added by Eric Davis at 2011-11-15 08:09 pm

I was trying to fix one of the sources of corruption [1] in the project tree today but I kept running into issues with awesome nested set.

Are we still thinking about switching to ancestry?

Eric Davis

[1]: The tree is configured to be ordered by project name but if you edit only the name the tree is still considered valid and won't rebuild. I tried to add a validation to the tree to mark it as invalid but the rebuild still isn't working correctly (code). From what I see in the awesome nested set's tests, the :order option really isn't tested that much.

- id, name (root_id, lft, rgt)
* 1 Project A (, 1, 6)
** 2 Project A1 (1, 2, 3)
** 3 A Project is before (1, 4, 5) <<< notice how this should be sorted before id:2. "A Project" <=> "Project A1" 
* 4 Project B (, 7, 14)
** 5 Project B1 (4, 8, 11)
*** 6 Project B11 (5, 9, 10)
** 7 Project B2 (4, 12, 13)
* 8 Project C (, 15, 18)
** 9 Project C1 (8, 16, 17)

RE: [Proposal] Replace awesome_nested_set with ancestry - Added by Felix Schäfer at 2011-11-27 11:07 am

I'm all for it, though I'd be even more for it if we received an answer from the maintainer regarding my query above. I know we're not that much better, but I haven't seen much activity on the plugin since june, or we must live with having to fork/maintain it at some point or another.

RE: [Proposal] Replace awesome_nested_set with ancestry - Added by Gabriel Mazetto at 2011-12-18 03:29 am

I've been using it in a comercial project and can say, it works fine. It was the chosen one for trees.

RE: [Proposal] Replace awesome_nested_set with ancestry - Added by Ben Cooksley at 2012-02-28 10:31 am

In case it helps someone, I recently needed to rebuild the entire tree so that it was alphabetical.

The following rough patch makes rebuild! do what the name suggests - rebuild the tree.

http://paste.kde.org/430604/raw/

(1-10/10)