[CLI Project] Support of filtering of objects in Command Shell
Hi,
Now the mid term evaluations are done, I would be building the command line shell, as planned. I would like to present how I would like to tackle the requirement at hand.
As mentioned in my proposal and the most logical approach, all the common functionalities between the shell and the already built command line tools would use the same classes, the ones built for the command tools.
As this is a command shell, I wish to give it a SQL like feel. However, The SQL commands SELECT, INSERT INTO do not look nice in Mailman context, as we are dealing with objects and not tables. I would be using the object related terms like SHOW, CREATE etc for the respective actions.
However, I would like to discuss about the support for the WHERE clause by the shell.
The WHERE clause is not mentioned in my proposal, but I feel that this feature ought to be in the project.
If the commands require to support a WHERE clause, as SQL does,
For example,
show users where display_name = 'foo' and subscribed_list_ids contains = 'list@domain.org mailto:%27list@domain.org'
The steps involved in processing a command with a WHERE clause would be
1.Parse the command 2.Filter the results from the database 3.Perform the action using the command line tool methods if possible else write a new procedure.
For the Step 2, there are two approaches in hand:
1.Use the Storm library to query the database directly, as mailman core does. 2.Use the REST API
By using the first approach, a good performance improvement is achieved, by leaving the data filtering to database engine. But this limits the usability of of the CLI to the system in which the mailman database exists, as the remote connections to DB servers are usually disabled.
By using the second approach, the performance is bad, as it requires a
normal python for loop that compares the the results one by one.
However, this approach might be easier in case that the mailman
installation be managed remotely, if its OK to enable the REST API to
accept connections from remote machines.(I believe this is currently not
supported/disabled by default.)
By building the WHERE clause, its costlier to reuse the command line tools methods in many use cases, like delete, subscribe etc. Further, as I have already mentioned to Abhilash, it demands more time. I am not sure if that will fit into GSoC time frame. I am confident that I can make significant progress, but not sure if it can be 'completed'. Also, I am fine on completing it even after GSoC.
So, the final question is, Should the CLI support filtering using the WHERE clause? Or should I proceed as per my proposal, building a basic Mailman shell and leave the rest to be completed in future?
On Jun 28, 2014, at 10:55 PM, Rajeev S wrote:
By using the first approach, a good performance improvement is achieved, by leaving the data filtering to database engine. But this limits the usability of of the CLI to the system in which the mailman database exists, as the remote connections to DB servers are usually disabled.
Not only that, but there no backward compatibility guarantees about the database schema. The only guarantee is that there will be some automated scripts to do schema migration if it does change.
By using the second approach, the performance is bad, as it requires a normal python
forloop that compares the the results one by one. However, this approach might be easier in case that the mailman installation be managed remotely, if its OK to enable the REST API to accept connections from remote machines.(I believe this is currently not supported/disabled by default.)
Correct, and the core's REST server should never be exposed to the internet. That's the purpose of the REST proxy layer, probably best provided by Postorius, which mediates access to the REST API via an authorization layer.
However, it's entirely feasible that additional REST APIs be built that can provide more advanced access to the data model, such as filtering. Membership search and rosters already provide some functionality like this.
Cheers, -Barry
You don't need to put me in the addressee list if you Cc mailman-developers.
Rajeev S writes:
As this is a command shell, I wish to give it a SQL like feel.
I don't understand why you want it to feel like SQL. SQL is hardly something I would expect typical list admins to know about. Choice of some of the operators might be inspired by SQL's more-or-less natural language style.
For example,
show users where display_name = 'foo' and subscribed_list_ids contains = 'list@domain.org mailto:%27list@domain.org'
Given that we use objects, I somewhat prefer
show users with display_name = 'foo' and subscribed_lists containing 'list.domain.org' or 'list.domain.net'
Or somewhat contorted grammar:
show users with 'foo' as display_name and 'list.domain.org' or 'list.domain.net' in subscribed_lists
1.Use the Storm library to query the database directly, as mailman core does.
2.Use the REST API
By using the first approach, a good performance improvement is achieved, by leaving the data filtering to database engine. But this limits the usability of of the CLI to the system in which the mailman database exists, as the remote connections to DB servers are usually disabled.
I think this is a distinction without a difference as the REST API is usually disabled for remote clients as well.
By using the second approach, the performance is bad, as it requires a normal python
forloop that compares the the results one by one.
Extend the REST API if performance bothers you. (FVO "you" that include future GSoC students and users.) I don't think performance should be a consideration here. Focus on functionality.
So, the final question is, Should the CLI support filtering using the WHERE clause? Or should I proceed as per my proposal, building a basic Mailman shell and leave the rest to be completed in future?
I think it should support simple filtering at least.
participants (3)
-
Barry Warsaw -
Rajeev S -
Stephen J. Turnbull